python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/datou.py -j batch_current -C 2526006' -s traitement_3459 -M 0 -S 0 -U 100,80,95 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/home/admin/workarea/git/apy', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3588220 load datou : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 0 init_dummy_multi_datou is not linked in the step_by_step architecture ! WARNING : step 1294 init_dummy_multi_datou is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2526006'] with mtr_portfolio_ids : ['20043200'] and first list_photo_ids : [] new path : /proc/3588220/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score over limit max, limiting to limit_max 20 list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 20 ; length of list_pids : 20 ; length of list_args : 20 time to download the photos : 3.0441157817840576 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Feb 4 11:03:19 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-04 11:03:22.668087: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-04 11:03:22.695132: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-04 11:03:22.697295: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fa6d8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-04 11:03:22.697355: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-04 11:03:22.702023: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-04 11:03:22.955866: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x34cd6120 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-04 11:03:22.955948: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-04 11:03:22.957597: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 11:03:22.958204: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 11:03:22.961712: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 11:03:22.964898: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 11:03:22.965508: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 11:03:22.968782: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 11:03:22.969780: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 11:03:22.974257: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 11:03:22.975987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 11:03:22.976119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 11:03:22.976936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-04 11:03:22.976954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-04 11:03:22.976979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-04 11:03:22.978293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-04 11:03:23.274127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 11:03:23.274424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 11:03:23.274480: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 11:03:23.274517: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 11:03:23.274556: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 11:03:23.274587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 11:03:23.274616: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 11:03:23.274646: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 11:03:23.281260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 11:03:23.283161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 11:03:23.283232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 11:03:23.283250: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 11:03:23.283266: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 11:03:23.283283: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 11:03:23.283300: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 11:03:23.283317: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 11:03:23.283337: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 11:03:23.284627: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 11:03:23.284661: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-04 11:03:23.284671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-04 11:03:23.284679: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-04 11:03:23.286061: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-04 11:03:30.954064: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 11:03:31.148277: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 20 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 148.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 Detection mask done ! Trying to reset tf kernel 3588420 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5264 tf kernel not reseted sub process len(results) : 20 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 20 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10553 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.021284818649291992 nb_pixel_total : 1424267 time to create 1 rle with new method : 0.03791332244873047 length of segment : 1929 time for calcul the mask position with numpy : 0.004880428314208984 nb_pixel_total : 303171 time to create 1 rle with new method : 0.008718252182006836 length of segment : 939 time for calcul the mask position with numpy : 0.004523515701293945 nb_pixel_total : 314930 time to create 1 rle with new method : 0.009234428405761719 length of segment : 932 time for calcul the mask position with numpy : 0.023677349090576172 nb_pixel_total : 1361904 time to create 1 rle with new method : 0.04204392433166504 length of segment : 2019 time for calcul the mask position with numpy : 0.0015673637390136719 nb_pixel_total : 89742 time to create 1 rle with old method : 0.10311079025268555 length of segment : 332 time for calcul the mask position with numpy : 0.003492116928100586 nb_pixel_total : 319481 time to create 1 rle with new method : 0.007712602615356445 length of segment : 919 time for calcul the mask position with numpy : 0.003545045852661133 nb_pixel_total : 249790 time to create 1 rle with new method : 0.008491039276123047 length of segment : 840 time for calcul the mask position with numpy : 0.01855301856994629 nb_pixel_total : 1345084 time to create 1 rle with new method : 0.04543113708496094 length of segment : 1577 time for calcul the mask position with numpy : 0.00025463104248046875 nb_pixel_total : 16043 time to create 1 rle with old method : 0.01941847801208496 length of segment : 125 time for calcul the mask position with numpy : 0.0003826618194580078 nb_pixel_total : 20337 time to create 1 rle with old method : 0.02440357208251953 length of segment : 144 time for calcul the mask position with numpy : 0.014579296112060547 nb_pixel_total : 809778 time to create 1 rle with new method : 0.03706765174865723 length of segment : 2076 time for calcul the mask position with numpy : 0.0018711090087890625 nb_pixel_total : 68775 time to create 1 rle with old method : 0.07866549491882324 length of segment : 322 time for calcul the mask position with numpy : 0.002683401107788086 nb_pixel_total : 121044 time to create 1 rle with old method : 0.16187739372253418 length of segment : 375 time for calcul the mask position with numpy : 0.00859689712524414 nb_pixel_total : 462335 time to create 1 rle with new method : 0.01643514633178711 length of segment : 1620 time for calcul the mask position with numpy : 0.025717496871948242 nb_pixel_total : 1432114 time to create 1 rle with new method : 0.03253960609436035 length of segment : 1533 time for calcul the mask position with numpy : 0.00026035308837890625 nb_pixel_total : 10789 time to create 1 rle with old method : 0.012758970260620117 length of segment : 100 time for calcul the mask position with numpy : 0.003807544708251953 nb_pixel_total : 326019 time to create 1 rle with new method : 0.008467912673950195 length of segment : 1040 time for calcul the mask position with numpy : 0.0001246929168701172 nb_pixel_total : 3255 time to create 1 rle with old method : 0.004143953323364258 length of segment : 85 time for calcul the mask position with numpy : 0.02229452133178711 nb_pixel_total : 1285324 time to create 1 rle with new method : 0.03624916076660156 length of segment : 1934 time for calcul the mask position with numpy : 0.0010273456573486328 nb_pixel_total : 42850 time to create 1 rle with old method : 0.07295513153076172 length of segment : 245 time for calcul the mask position with numpy : 0.005538225173950195 nb_pixel_total : 301041 time to create 1 rle with new method : 0.012843132019042969 length of segment : 930 time for calcul the mask position with numpy : 0.000774383544921875 nb_pixel_total : 48060 time to create 1 rle with old method : 0.05734753608703613 length of segment : 280 time for calcul the mask position with numpy : 0.00045013427734375 nb_pixel_total : 11111 time to create 1 rle with old method : 0.01395559310913086 length of segment : 134 time for calcul the mask position with numpy : 0.0010619163513183594 nb_pixel_total : 39395 time to create 1 rle with old method : 0.046570539474487305 length of segment : 257 time for calcul the mask position with numpy : 0.00041866302490234375 nb_pixel_total : 13330 time to create 1 rle with old method : 0.015965938568115234 length of segment : 113 time for calcul the mask position with numpy : 0.013924121856689453 nb_pixel_total : 817023 time to create 1 rle with new method : 0.022009849548339844 length of segment : 1601 time for calcul the mask position with numpy : 0.0002307891845703125 nb_pixel_total : 9301 time to create 1 rle with old method : 0.011656999588012695 length of segment : 127 time for calcul the mask position with numpy : 0.0007536411285400391 nb_pixel_total : 51325 time to create 1 rle with old method : 0.05779623985290527 length of segment : 323 time for calcul the mask position with numpy : 0.00018548965454101562 nb_pixel_total : 10527 time to create 1 rle with old method : 0.012128591537475586 length of segment : 149 time for calcul the mask position with numpy : 0.003647327423095703 nb_pixel_total : 303771 time to create 1 rle with new method : 0.008132696151733398 length of segment : 898 time for calcul the mask position with numpy : 0.0005846023559570312 nb_pixel_total : 12719 time to create 1 rle with old method : 0.013873577117919922 length of segment : 242 time for calcul the mask position with numpy : 0.00787663459777832 nb_pixel_total : 409165 time to create 1 rle with new method : 0.013564109802246094 length of segment : 980 time for calcul the mask position with numpy : 0.0022716522216796875 nb_pixel_total : 185062 time to create 1 rle with new method : 0.004533529281616211 length of segment : 465 time for calcul the mask position with numpy : 0.005200624465942383 nb_pixel_total : 374068 time to create 1 rle with new method : 0.010211706161499023 length of segment : 1101 time for calcul the mask position with numpy : 0.00010609626770019531 nb_pixel_total : 3361 time to create 1 rle with old method : 0.0038704872131347656 length of segment : 151 time for calcul the mask position with numpy : 0.012881278991699219 nb_pixel_total : 1001884 time to create 1 rle with new method : 0.021070003509521484 length of segment : 1491 time for calcul the mask position with numpy : 0.014374494552612305 nb_pixel_total : 1109630 time to create 1 rle with new method : 0.0235443115234375 length of segment : 1632 time for calcul the mask position with numpy : 0.002883434295654297 nb_pixel_total : 183567 time to create 1 rle with new method : 0.007710456848144531 length of segment : 982 time for calcul the mask position with numpy : 0.005506038665771484 nb_pixel_total : 373023 time to create 1 rle with new method : 0.011192560195922852 length of segment : 1004 time for calcul the mask position with numpy : 0.019690752029418945 nb_pixel_total : 1029521 time to create 1 rle with new method : 0.0344233512878418 length of segment : 2094 time for calcul the mask position with numpy : 0.0008234977722167969 nb_pixel_total : 46678 time to create 1 rle with old method : 0.05172157287597656 length of segment : 371 time for calcul the mask position with numpy : 0.0001842975616455078 nb_pixel_total : 9530 time to create 1 rle with old method : 0.010863304138183594 length of segment : 92 time for calcul the mask position with numpy : 0.0007410049438476562 nb_pixel_total : 64851 time to create 1 rle with old method : 0.07369542121887207 length of segment : 294 time for calcul the mask position with numpy : 0.0001659393310546875 nb_pixel_total : 7076 time to create 1 rle with old method : 0.008009195327758789 length of segment : 91 time for calcul the mask position with numpy : 0.0017466545104980469 nb_pixel_total : 155489 time to create 1 rle with new method : 0.003882884979248047 length of segment : 456 time for calcul the mask position with numpy : 0.0005974769592285156 nb_pixel_total : 20493 time to create 1 rle with old method : 0.022170305252075195 length of segment : 235 time for calcul the mask position with numpy : 0.00021576881408691406 nb_pixel_total : 6431 time to create 1 rle with old method : 0.008357524871826172 length of segment : 75 time for calcul the mask position with numpy : 0.0016300678253173828 nb_pixel_total : 73427 time to create 1 rle with old method : 0.10536360740661621 length of segment : 209 time for calcul the mask position with numpy : 0.0009431838989257812 nb_pixel_total : 32900 time to create 1 rle with old method : 0.036458492279052734 length of segment : 287 time for calcul the mask position with numpy : 0.0013625621795654297 nb_pixel_total : 56149 time to create 1 rle with old method : 0.06248760223388672 length of segment : 322 time for calcul the mask position with numpy : 0.00022172927856445312 nb_pixel_total : 5422 time to create 1 rle with old method : 0.006378173828125 length of segment : 107 time for calcul the mask position with numpy : 0.014417648315429688 nb_pixel_total : 682292 time to create 1 rle with new method : 0.024349212646484375 length of segment : 1282 time for calcul the mask position with numpy : 0.00011324882507324219 nb_pixel_total : 1435 time to create 1 rle with old method : 0.0017962455749511719 length of segment : 60 time for calcul the mask position with numpy : 0.00012111663818359375 nb_pixel_total : 6450 time to create 1 rle with old method : 0.007760286331176758 length of segment : 84 time for calcul the mask position with numpy : 0.00027370452880859375 nb_pixel_total : 4707 time to create 1 rle with old method : 0.0055043697357177734 length of segment : 148 time for calcul the mask position with numpy : 0.0002818107604980469 nb_pixel_total : 12790 time to create 1 rle with old method : 0.015507936477661133 length of segment : 296 time for calcul the mask position with numpy : 0.0013115406036376953 nb_pixel_total : 74037 time to create 1 rle with old method : 0.08224201202392578 length of segment : 379 time for calcul the mask position with numpy : 0.011237859725952148 nb_pixel_total : 443421 time to create 1 rle with new method : 0.017512798309326172 length of segment : 1031 time for calcul the mask position with numpy : 0.0014710426330566406 nb_pixel_total : 73016 time to create 1 rle with old method : 0.08153080940246582 length of segment : 504 time for calcul the mask position with numpy : 0.00017952919006347656 nb_pixel_total : 3769 time to create 1 rle with old method : 0.004402637481689453 length of segment : 73 time for calcul the mask position with numpy : 0.0002474784851074219 nb_pixel_total : 4546 time to create 1 rle with old method : 0.005049943923950195 length of segment : 92 time for calcul the mask position with numpy : 0.0020775794982910156 nb_pixel_total : 111440 time to create 1 rle with old method : 0.1233987808227539 length of segment : 627 time for calcul the mask position with numpy : 0.00017261505126953125 nb_pixel_total : 8901 time to create 1 rle with old method : 0.009811878204345703 length of segment : 147 time for calcul the mask position with numpy : 0.0023262500762939453 nb_pixel_total : 23453 time to create 1 rle with old method : 0.02593827247619629 length of segment : 197 time for calcul the mask position with numpy : 0.001383066177368164 nb_pixel_total : 64812 time to create 1 rle with old method : 0.07572197914123535 length of segment : 391 time for calcul the mask position with numpy : 0.0026650428771972656 nb_pixel_total : 148774 time to create 1 rle with old method : 0.16065168380737305 length of segment : 768 time for calcul the mask position with numpy : 0.00015497207641601562 nb_pixel_total : 5349 time to create 1 rle with old method : 0.006400346755981445 length of segment : 111 time for calcul the mask position with numpy : 0.0006976127624511719 nb_pixel_total : 28755 time to create 1 rle with old method : 0.03239107131958008 length of segment : 199 time for calcul the mask position with numpy : 0.00011754035949707031 nb_pixel_total : 4207 time to create 1 rle with old method : 0.004870414733886719 length of segment : 82 time for calcul the mask position with numpy : 0.0001316070556640625 nb_pixel_total : 1613 time to create 1 rle with old method : 0.0018734931945800781 length of segment : 70 time for calcul the mask position with numpy : 0.0005342960357666016 nb_pixel_total : 37966 time to create 1 rle with old method : 0.04229450225830078 length of segment : 243 time for calcul the mask position with numpy : 0.0005786418914794922 nb_pixel_total : 31829 time to create 1 rle with old method : 0.035401344299316406 length of segment : 213 time for calcul the mask position with numpy : 0.0010204315185546875 nb_pixel_total : 32341 time to create 1 rle with old method : 0.03630208969116211 length of segment : 357 time for calcul the mask position with numpy : 0.003985404968261719 nb_pixel_total : 162850 time to create 1 rle with new method : 0.00909280776977539 length of segment : 697 time for calcul the mask position with numpy : 0.0005652904510498047 nb_pixel_total : 18252 time to create 1 rle with old method : 0.021548032760620117 length of segment : 200 time for calcul the mask position with numpy : 0.00973820686340332 nb_pixel_total : 436027 time to create 1 rle with new method : 0.02190399169921875 length of segment : 1342 time for calcul the mask position with numpy : 0.0002052783966064453 nb_pixel_total : 9154 time to create 1 rle with old method : 0.013296365737915039 length of segment : 98 time for calcul the mask position with numpy : 0.006789445877075195 nb_pixel_total : 442295 time to create 1 rle with new method : 0.01220250129699707 length of segment : 1743 time for calcul the mask position with numpy : 0.0007798671722412109 nb_pixel_total : 53957 time to create 1 rle with old method : 0.060173988342285156 length of segment : 201 time for calcul the mask position with numpy : 0.0016663074493408203 nb_pixel_total : 52965 time to create 1 rle with old method : 0.058386802673339844 length of segment : 657 time for calcul the mask position with numpy : 0.002170562744140625 nb_pixel_total : 146721 time to create 1 rle with old method : 0.1578817367553711 length of segment : 646 time for calcul the mask position with numpy : 0.003360748291015625 nb_pixel_total : 200449 time to create 1 rle with new method : 0.005768537521362305 length of segment : 698 time for calcul the mask position with numpy : 0.0027039051055908203 nb_pixel_total : 135688 time to create 1 rle with old method : 0.15279722213745117 length of segment : 887 time for calcul the mask position with numpy : 0.011512517929077148 nb_pixel_total : 658940 time to create 1 rle with new method : 0.026804447174072266 length of segment : 1464 time for calcul the mask position with numpy : 0.0039501190185546875 nb_pixel_total : 306109 time to create 1 rle with new method : 0.007793426513671875 length of segment : 1077 time for calcul the mask position with numpy : 0.0003619194030761719 nb_pixel_total : 25176 time to create 1 rle with old method : 0.02822422981262207 length of segment : 138 time for calcul the mask position with numpy : 0.00013685226440429688 nb_pixel_total : 613 time to create 1 rle with old method : 0.0011048316955566406 length of segment : 48 time for calcul the mask position with numpy : 0.0006392002105712891 nb_pixel_total : 15132 time to create 1 rle with old method : 0.025242328643798828 length of segment : 131 time for calcul the mask position with numpy : 0.0011281967163085938 nb_pixel_total : 36598 time to create 1 rle with old method : 0.0482485294342041 length of segment : 296 time for calcul the mask position with numpy : 0.0019168853759765625 nb_pixel_total : 83244 time to create 1 rle with old method : 0.09112167358398438 length of segment : 452 time for calcul the mask position with numpy : 0.010324478149414062 nb_pixel_total : 383240 time to create 1 rle with new method : 0.017850160598754883 length of segment : 1560 time for calcul the mask position with numpy : 0.0015408992767333984 nb_pixel_total : 80279 time to create 1 rle with old method : 0.08766651153564453 length of segment : 623 time for calcul the mask position with numpy : 0.001695871353149414 nb_pixel_total : 86842 time to create 1 rle with old method : 0.09493279457092285 length of segment : 386 time for calcul the mask position with numpy : 0.0007584095001220703 nb_pixel_total : 44832 time to create 1 rle with old method : 0.05116009712219238 length of segment : 171 time for calcul the mask position with numpy : 0.00019240379333496094 nb_pixel_total : 7544 time to create 1 rle with old method : 0.008450746536254883 length of segment : 91 time for calcul the mask position with numpy : 7.653236389160156e-05 nb_pixel_total : 3146 time to create 1 rle with old method : 0.003629446029663086 length of segment : 61 time for calcul the mask position with numpy : 0.0001595020294189453 nb_pixel_total : 6891 time to create 1 rle with old method : 0.008078813552856445 length of segment : 116 time for calcul the mask position with numpy : 0.0007684230804443359 nb_pixel_total : 44946 time to create 1 rle with old method : 0.050652265548706055 length of segment : 172 time for calcul the mask position with numpy : 0.00024509429931640625 nb_pixel_total : 14162 time to create 1 rle with old method : 0.015434503555297852 length of segment : 141 time for calcul the mask position with numpy : 0.0004203319549560547 nb_pixel_total : 20256 time to create 1 rle with old method : 0.023166418075561523 length of segment : 149 time for calcul the mask position with numpy : 0.008996009826660156 nb_pixel_total : 489906 time to create 1 rle with new method : 0.01964116096496582 length of segment : 2079 time for calcul the mask position with numpy : 0.0002951622009277344 nb_pixel_total : 11571 time to create 1 rle with old method : 0.013227224349975586 length of segment : 91 time for calcul the mask position with numpy : 0.0013594627380371094 nb_pixel_total : 96910 time to create 1 rle with old method : 0.10556864738464355 length of segment : 329 time for calcul the mask position with numpy : 0.015534400939941406 nb_pixel_total : 935983 time to create 1 rle with new method : 0.025463342666625977 length of segment : 1472 time for calcul the mask position with numpy : 0.0006341934204101562 nb_pixel_total : 48573 time to create 1 rle with old method : 0.054222822189331055 length of segment : 297 time for calcul the mask position with numpy : 0.002635478973388672 nb_pixel_total : 143648 time to create 1 rle with old method : 0.15861940383911133 length of segment : 633 time for calcul the mask position with numpy : 0.00019621849060058594 nb_pixel_total : 8617 time to create 1 rle with old method : 0.010899066925048828 length of segment : 189 time for calcul the mask position with numpy : 0.004599809646606445 nb_pixel_total : 312867 time to create 1 rle with new method : 0.009115219116210938 length of segment : 954 time for calcul the mask position with numpy : 5.507469177246094e-05 nb_pixel_total : 480 time to create 1 rle with old method : 0.0006537437438964844 length of segment : 44 time for calcul the mask position with numpy : 0.0006678104400634766 nb_pixel_total : 23275 time to create 1 rle with old method : 0.02684926986694336 length of segment : 270 time for calcul the mask position with numpy : 0.0003192424774169922 nb_pixel_total : 12934 time to create 1 rle with old method : 0.015082120895385742 length of segment : 188 time for calcul the mask position with numpy : 0.0002033710479736328 nb_pixel_total : 8377 time to create 1 rle with old method : 0.00996088981628418 length of segment : 70 time for calcul the mask position with numpy : 0.0037016868591308594 nb_pixel_total : 181225 time to create 1 rle with new method : 0.008116722106933594 length of segment : 478 time for calcul the mask position with numpy : 0.00896453857421875 nb_pixel_total : 490718 time to create 1 rle with new method : 0.017618894577026367 length of segment : 1053 time for calcul the mask position with numpy : 0.00019240379333496094 nb_pixel_total : 2747 time to create 1 rle with old method : 0.0033736228942871094 length of segment : 91 time for calcul the mask position with numpy : 8.988380432128906e-05 nb_pixel_total : 2815 time to create 1 rle with old method : 0.0032906532287597656 length of segment : 77 time for calcul the mask position with numpy : 0.0015697479248046875 nb_pixel_total : 75505 time to create 1 rle with old method : 0.08029031753540039 length of segment : 338 time for calcul the mask position with numpy : 0.0016276836395263672 nb_pixel_total : 58480 time to create 1 rle with old method : 0.06412148475646973 length of segment : 282 time for calcul the mask position with numpy : 0.00035190582275390625 nb_pixel_total : 8278 time to create 1 rle with old method : 0.00982666015625 length of segment : 110 time for calcul the mask position with numpy : 0.0010838508605957031 nb_pixel_total : 49164 time to create 1 rle with old method : 0.05694913864135742 length of segment : 299 time for calcul the mask position with numpy : 0.00015616416931152344 nb_pixel_total : 4287 time to create 1 rle with old method : 0.005205631256103516 length of segment : 88 time for calcul the mask position with numpy : 0.0008587837219238281 nb_pixel_total : 45572 time to create 1 rle with old method : 0.05107259750366211 length of segment : 233 time for calcul the mask position with numpy : 0.0009012222290039062 nb_pixel_total : 24520 time to create 1 rle with old method : 0.02836132049560547 length of segment : 292 time for calcul the mask position with numpy : 0.0062177181243896484 nb_pixel_total : 411270 time to create 1 rle with new method : 0.011104822158813477 length of segment : 1309 time for calcul the mask position with numpy : 0.0041658878326416016 nb_pixel_total : 201129 time to create 1 rle with new method : 0.006867408752441406 length of segment : 868 time for calcul the mask position with numpy : 0.00038909912109375 nb_pixel_total : 17023 time to create 1 rle with old method : 0.01971745491027832 length of segment : 96 time for calcul the mask position with numpy : 0.010032176971435547 nb_pixel_total : 508489 time to create 1 rle with new method : 0.019745349884033203 length of segment : 1570 time for calcul the mask position with numpy : 0.0002529621124267578 nb_pixel_total : 9886 time to create 1 rle with old method : 0.01138758659362793 length of segment : 101 time for calcul the mask position with numpy : 0.00018310546875 nb_pixel_total : 9500 time to create 1 rle with old method : 0.010405540466308594 length of segment : 139 time for calcul the mask position with numpy : 0.00016117095947265625 nb_pixel_total : 8566 time to create 1 rle with old method : 0.009894609451293945 length of segment : 93 time spent for convertir_results : 14.548319339752197 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 153 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 72311 save missing photos in datou_result : time spend for datou_step_exec : 36.6912899017334 time spend to save output : 3.9669206142425537 total time spend for step 1 : 40.65821051597595 step2:crop_condition Tue Feb 4 11:04:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 20 ! batch 1 Loaded 153 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 14 About to insert : list_path_to_insert length 14 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1332631994, 1332631884, 1332631801, 1332631780, 1332631774, 1332631738, 1332631735, 1332631732, 1332631729, 1332631726, 1332631722, 1332631714, 1332631711, 1332631706, 1332631700, 1332631685, 1332631682, 1332631643, 1332631639, 1332631635] Looping around the photos to save general results len do output : 40 /-3653338227Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779530Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779539Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779552Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779557Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779574Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779573Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779582Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653338246Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653338245Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779571Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779586Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779606Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779614Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779638Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779551Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779587Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779607Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779532Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779538Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779553Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779566Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779579Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779577Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779595Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779601Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779609Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779612Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779629Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779628Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779631Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779520Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779535Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779626Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779581Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3642779575Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631994', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631884', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631801', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631780', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631774', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631738', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631735', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631732', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631729', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631726', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631722', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631714', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631711', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631706', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631700', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631685', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631682', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631643', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631639', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631635', None, None, None, None, None, '2526006') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 140 time used for this insertion : 0.030869245529174805 save_final save missing photos in datou_result : time spend for datou_step_exec : 11.275649309158325 time spend to save output : 0.032221317291259766 total time spend for step 2 : 11.307870626449585 step3:thcl Tue Feb 4 11:04:11 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.022681713104248047 time to convert the images to numpy array : 0.030696392059326172 time to import caffe and check if the image exist : 0.015273571014404297 time to convert the images to numpy array : 0.03970503807067871 time to import caffe and check if the image exist : 0.005616903305053711 time to convert the images to numpy array : 0.05289959907531738 time to import caffe and check if the image exist : 0.04449629783630371 time to convert the images to numpy array : 0.019393205642700195 time to import caffe and check if the image exist : 0.01081085205078125 time to convert the images to numpy array : 0.056851863861083984 time to import caffe and check if the image exist : 0.0225679874420166 time to convert the images to numpy array : 0.050479888916015625 time to import caffe and check if the image exist : 0.021456003189086914 time to convert the images to numpy array : 0.051612138748168945 time to import caffe and check if the image exist : 0.04785871505737305 time to convert the images to numpy array : 0.028392553329467773 time to import caffe and check if the image exist : 0.015210390090942383 time to convert the images to numpy array : 0.06083035469055176 time to import caffe and check if the image exist : 0.0379025936126709 time to convert the images to numpy array : 0.03802204132080078 total time to convert the images to numpy array : 0.273787260055542 list photo_ids error: [] list photo_ids correct : [-3642779628, -3642779631, -3642779520, -3642779535, -3642779638, -3642779551, -3642779587, -3642779607, -3642779557, -3642779574, -3642779573, -3642779582, -3642779586, -3642779606, -3642779614, -3642779637, -3642779532, -3642779538, -3642779553, -3642779566, -3642779579, -3642779577, -3642779595, -3642779601, -3642779609, -3642779604, -3642779612, -3642779629, -3653338227, -3642779530, -3642779539, -3642779552, -3642779588, -3653338246, -3653338245, -3642779571, -3642779603, -3642779626, -3642779581, -3642779575] number of photos to traite : 40 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5485 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/:/home/admin/workarea/git/apy/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5266 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.3114204406738281 time used to do the prediction : 0.0998992919921875 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5264 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/:/home/admin/workarea/git/apy/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10553 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.39612507820129395 time used to do the prediction : 0.08477067947387695 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 5.0067901611328125e-06 save missing photos in datou_result : time spend for datou_step_exec : 7.818943977355957 time spend to save output : 0.0003592967987060547 total time spend for step 3 : 7.819303274154663 step4:merge_mask_thcl_custom Tue Feb 4 11:04:19 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 153 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1332631994, 1332631884, 1332631801, 1332631780, 1332631774, 1332631738, 1332631735, 1332631732, 1332631729, 1332631726, 1332631722, 1332631714, 1332631711, 1332631706, 1332631700, 1332631685, 1332631682, 1332631643, 1332631639, 1332631635] Looping around the photos to save general results len do output : 20 /1332631994Didn't retrieve data .Didn't retrieve data . /1332631884Didn't retrieve data .Didn't retrieve data . /1332631801Didn't retrieve data .Didn't retrieve data . /1332631780Didn't retrieve data .Didn't retrieve data . /1332631774Didn't retrieve data .Didn't retrieve data . /1332631738Didn't retrieve data .Didn't retrieve data . /1332631735Didn't retrieve data .Didn't retrieve data . /1332631732Didn't retrieve data .Didn't retrieve data . /1332631729Didn't retrieve data .Didn't retrieve data . /1332631726Didn't retrieve data .Didn't retrieve data . /1332631722Didn't retrieve data .Didn't retrieve data . /1332631714Didn't retrieve data .Didn't retrieve data . /1332631711Didn't retrieve data .Didn't retrieve data . /1332631706Didn't retrieve data .Didn't retrieve data . /1332631700Didn't retrieve data .Didn't retrieve data . /1332631685Didn't retrieve data .Didn't retrieve data . /1332631682Didn't retrieve data .Didn't retrieve data . /1332631643Didn't retrieve data .Didn't retrieve data . /1332631639Didn't retrieve data .Didn't retrieve data . /1332631635Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631994', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631884', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631801', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631780', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631774', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631738', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631735', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631732', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631729', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631726', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631722', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631714', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631711', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631706', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631700', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631685', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631682', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631643', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631639', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631635', None, None, None, None, None, '2526006') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 60 time used for this insertion : 0.014572381973266602 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.061928749084472656 time spend to save output : 0.015379190444946289 total time spend for step 4 : 0.07730793952941895 step5:rle_unique_nms_with_priority Tue Feb 4 11:04:19 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 5 nb_hashtags : 1 time to prepare the origin masks : 0.5249695777893066 time for calcul the mask position with numpy : 0.00944066047668457 nb_pixel_total : 372995 time to create 1 rle with new method : 0.055219173431396484 time for calcul the mask position with numpy : 0.007335662841796875 nb_pixel_total : 91931 time to create 1 rle with old method : 0.11069822311401367 time for calcul the mask position with numpy : 0.00586390495300293 nb_pixel_total : 467 time to create 1 rle with old method : 0.0009644031524658203 time for calcul the mask position with numpy : 0.006998777389526367 nb_pixel_total : 32387 time to create 1 rle with old method : 0.033519744873046875 time for calcul the mask position with numpy : 0.006895303726196289 nb_pixel_total : 3635 time to create 1 rle with old method : 0.00536036491394043 time for calcul the mask position with numpy : 0.022342443466186523 nb_pixel_total : 1572185 time to create 1 rle with new method : 0.043996572494506836 create new chi : 0.3186659812927246 time to delete rle : 0.01413273811340332 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6804 TO DO : save crop sub photo not yet done ! save time : 0.40569496154785156 nb_obj : 6 nb_hashtags : 2 time to prepare the origin masks : 0.6707117557525635 time for calcul the mask position with numpy : 0.012502908706665039 nb_pixel_total : 555418 time to create 1 rle with new method : 0.04491567611694336 time for calcul the mask position with numpy : 0.007084369659423828 nb_pixel_total : 89529 time to create 1 rle with old method : 0.0992884635925293 time for calcul the mask position with numpy : 0.007000923156738281 nb_pixel_total : 478 time to create 1 rle with old method : 0.0017783641815185547 time for calcul the mask position with numpy : 0.007294893264770508 nb_pixel_total : 21280 time to create 1 rle with old method : 0.030369281768798828 time for calcul the mask position with numpy : 0.0066699981689453125 nb_pixel_total : 41572 time to create 1 rle with old method : 0.05028343200683594 time for calcul the mask position with numpy : 0.00651860237121582 nb_pixel_total : 4106 time to create 1 rle with old method : 0.00691986083984375 time for calcul the mask position with numpy : 0.019655942916870117 nb_pixel_total : 1361217 time to create 1 rle with new method : 0.039947509765625 create new chi : 0.34761595726013184 time to delete rle : 0.0009889602661132812 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 9587 TO DO : save crop sub photo not yet done ! save time : 0.5610697269439697 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.3936154842376709 time for calcul the mask position with numpy : 0.012199640274047852 nb_pixel_total : 699855 time to create 1 rle with new method : 0.04710197448730469 time for calcul the mask position with numpy : 0.00693964958190918 nb_pixel_total : 11050 time to create 1 rle with old method : 0.013615131378173828 time for calcul the mask position with numpy : 0.006840705871582031 nb_pixel_total : 13799 time to create 1 rle with old method : 0.015750646591186523 time for calcul the mask position with numpy : 0.0072917938232421875 nb_pixel_total : 6813 time to create 1 rle with old method : 0.009953975677490234 time for calcul the mask position with numpy : 0.01785421371459961 nb_pixel_total : 1342083 time to create 1 rle with new method : 0.03950047492980957 create new chi : 0.18193483352661133 time to delete rle : 0.0006010532379150391 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 6286 TO DO : save crop sub photo not yet done ! save time : 0.3847775459289551 nb_obj : 6 nb_hashtags : 1 time to prepare the origin masks : 0.6118364334106445 time for calcul the mask position with numpy : 0.016001224517822266 nb_pixel_total : 1032252 time to create 1 rle with new method : 0.04329514503479004 time for calcul the mask position with numpy : 0.007200002670288086 nb_pixel_total : 4135 time to create 1 rle with old method : 0.005377769470214844 time for calcul the mask position with numpy : 0.008103609085083008 nb_pixel_total : 148880 time to create 1 rle with old method : 0.19225192070007324 time for calcul the mask position with numpy : 0.006396770477294922 nb_pixel_total : 4981 time to create 1 rle with old method : 0.0062563419342041016 time for calcul the mask position with numpy : 0.00675654411315918 nb_pixel_total : 59761 time to create 1 rle with old method : 0.07169914245605469 time for calcul the mask position with numpy : 0.008597612380981445 nb_pixel_total : 11860 time to create 1 rle with old method : 0.02126455307006836 time for calcul the mask position with numpy : 0.01561880111694336 nb_pixel_total : 811731 time to create 1 rle with new method : 0.05532431602478027 create new chi : 0.4747743606567383 time to delete rle : 0.0013132095336914062 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 9853 TO DO : save crop sub photo not yet done ! save time : 0.6440398693084717 nb_obj : 5 nb_hashtags : 1 time to prepare the origin masks : 0.6313471794128418 time for calcul the mask position with numpy : 0.013808727264404297 nb_pixel_total : 483286 time to create 1 rle with new method : 0.04558825492858887 time for calcul the mask position with numpy : 0.007059335708618164 nb_pixel_total : 54228 time to create 1 rle with old method : 0.060388803482055664 time for calcul the mask position with numpy : 0.011468648910522461 nb_pixel_total : 57 time to create 1 rle with old method : 0.00022363662719726562 time for calcul the mask position with numpy : 0.007317066192626953 nb_pixel_total : 8666 time to create 1 rle with old method : 0.01027679443359375 time for calcul the mask position with numpy : 0.0075397491455078125 nb_pixel_total : 97106 time to create 1 rle with old method : 0.1069798469543457 time for calcul the mask position with numpy : 0.021643638610839844 nb_pixel_total : 1430257 time to create 1 rle with new method : 0.04266190528869629 create new chi : 0.34242749214172363 time to delete rle : 0.0006926059722900391 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6231 TO DO : save crop sub photo not yet done ! save time : 0.37105345726013184 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.48593592643737793 time for calcul the mask position with numpy : 0.018045425415039062 nb_pixel_total : 638820 time to create 1 rle with new method : 0.058814048767089844 time for calcul the mask position with numpy : 0.0075376033782958984 nb_pixel_total : 5406 time to create 1 rle with old method : 0.009665489196777344 time for calcul the mask position with numpy : 0.008458852767944336 nb_pixel_total : 103170 time to create 1 rle with old method : 0.13819241523742676 time for calcul the mask position with numpy : 0.00669407844543457 nb_pixel_total : 42175 time to create 1 rle with old method : 0.04601335525512695 time for calcul the mask position with numpy : 0.019748449325561523 nb_pixel_total : 1284029 time to create 1 rle with new method : 0.03966188430786133 create new chi : 0.3617591857910156 time to delete rle : 0.0007441043853759766 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 6609 TO DO : save crop sub photo not yet done ! save time : 0.36797356605529785 nb_obj : 8 nb_hashtags : 3 time to prepare the origin masks : 0.8626608848571777 time for calcul the mask position with numpy : 0.014567136764526367 nb_pixel_total : 1100703 time to create 1 rle with new method : 0.05025053024291992 time for calcul the mask position with numpy : 0.006558418273925781 nb_pixel_total : 32335 time to create 1 rle with old method : 0.03569459915161133 time for calcul the mask position with numpy : 0.006200075149536133 nb_pixel_total : 1557 time to create 1 rle with old method : 0.0018036365509033203 time for calcul the mask position with numpy : 0.012000322341918945 nb_pixel_total : 814323 time to create 1 rle with new method : 0.04186892509460449 time for calcul the mask position with numpy : 0.007714509963989258 nb_pixel_total : 50730 time to create 1 rle with old method : 0.05534672737121582 time for calcul the mask position with numpy : 0.006486415863037109 nb_pixel_total : 39188 time to create 1 rle with old method : 0.04282426834106445 time for calcul the mask position with numpy : 0.0064084529876708984 nb_pixel_total : 13268 time to create 1 rle with old method : 0.01465749740600586 time for calcul the mask position with numpy : 0.006725311279296875 nb_pixel_total : 11049 time to create 1 rle with old method : 0.012479782104492188 time for calcul the mask position with numpy : 0.00613856315612793 nb_pixel_total : 10447 time to create 1 rle with old method : 0.011430740356445312 create new chi : 0.3480379581451416 time to delete rle : 0.0006282329559326172 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 5969 TO DO : save crop sub photo not yet done ! save time : 0.33644795417785645 nb_obj : 9 nb_hashtags : 1 time to prepare the origin masks : 0.89664626121521 time for calcul the mask position with numpy : 0.014670133590698242 nb_pixel_total : 773644 time to create 1 rle with new method : 0.04357600212097168 time for calcul the mask position with numpy : 0.006964921951293945 nb_pixel_total : 13505 time to create 1 rle with old method : 0.0160524845123291 time for calcul the mask position with numpy : 0.00639796257019043 nb_pixel_total : 3354 time to create 1 rle with old method : 0.003907203674316406 time for calcul the mask position with numpy : 0.00667119026184082 nb_pixel_total : 43917 time to create 1 rle with old method : 0.047911643981933594 time for calcul the mask position with numpy : 0.0060312747955322266 nb_pixel_total : 728 time to create 1 rle with old method : 0.0012204647064208984 time for calcul the mask position with numpy : 0.006209850311279297 nb_pixel_total : 14401 time to create 1 rle with old method : 0.015962600708007812 time for calcul the mask position with numpy : 0.011713027954101562 nb_pixel_total : 795646 time to create 1 rle with new method : 0.03987741470336914 time for calcul the mask position with numpy : 0.008663654327392578 nb_pixel_total : 415627 time to create 1 rle with new method : 0.041021108627319336 time for calcul the mask position with numpy : 0.0062198638916015625 nb_pixel_total : 123 time to create 1 rle with old method : 0.00028777122497558594 time for calcul the mask position with numpy : 0.006099700927734375 nb_pixel_total : 12655 time to create 1 rle with old method : 0.0139007568359375 create new chi : 0.3159828186035156 time to delete rle : 0.0007014274597167969 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 7836 TO DO : save crop sub photo not yet done ! save time : 0.45876431465148926 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.5540146827697754 time for calcul the mask position with numpy : 0.01815342903137207 nb_pixel_total : 804260 time to create 1 rle with new method : 0.04295969009399414 time for calcul the mask position with numpy : 0.013579607009887695 nb_pixel_total : 130173 time to create 1 rle with old method : 0.16785192489624023 time for calcul the mask position with numpy : 0.006373167037963867 nb_pixel_total : 449 time to create 1 rle with old method : 0.0010287761688232422 time for calcul the mask position with numpy : 0.007934808731079102 nb_pixel_total : 30827 time to create 1 rle with old method : 0.03712940216064453 time for calcul the mask position with numpy : 0.017213106155395508 nb_pixel_total : 1107891 time to create 1 rle with new method : 0.03672623634338379 create new chi : 0.3521449565887451 time to delete rle : 0.0007576942443847656 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 6430 TO DO : save crop sub photo not yet done ! save time : 0.3466370105743408 nb_obj : 5 nb_hashtags : 4 time to prepare the origin masks : 0.5353372097015381 time for calcul the mask position with numpy : 0.016202926635742188 nb_pixel_total : 922548 time to create 1 rle with new method : 0.04662466049194336 time for calcul the mask position with numpy : 0.0062062740325927734 nb_pixel_total : 3788 time to create 1 rle with old method : 0.004235982894897461 time for calcul the mask position with numpy : 0.006045103073120117 nb_pixel_total : 3712 time to create 1 rle with old method : 0.004239559173583984 time for calcul the mask position with numpy : 0.006125926971435547 nb_pixel_total : 64632 time to create 1 rle with old method : 0.0700678825378418 time for calcul the mask position with numpy : 0.0066738128662109375 nb_pixel_total : 37616 time to create 1 rle with old method : 0.046109914779663086 time for calcul the mask position with numpy : 0.020234107971191406 nb_pixel_total : 1041304 time to create 1 rle with new method : 0.03505825996398926 create new chi : 0.2723569869995117 time to delete rle : 0.0006613731384277344 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6468 TO DO : save crop sub photo not yet done ! save time : 0.4952704906463623 nb_obj : 12 nb_hashtags : 3 time to prepare the origin masks : 1.1460378170013428 time for calcul the mask position with numpy : 0.018139123916625977 nb_pixel_total : 1115513 time to create 1 rle with new method : 0.03984975814819336 time for calcul the mask position with numpy : 0.006285429000854492 nb_pixel_total : 10496 time to create 1 rle with old method : 0.011817693710327148 time for calcul the mask position with numpy : 0.005953788757324219 nb_pixel_total : 4652 time to create 1 rle with old method : 0.00534510612487793 time for calcul the mask position with numpy : 0.005820512771606445 nb_pixel_total : 6417 time to create 1 rle with old method : 0.0071370601654052734 time for calcul the mask position with numpy : 0.006129026412963867 nb_pixel_total : 73181 time to create 1 rle with old method : 0.07807278633117676 time for calcul the mask position with numpy : 0.006461620330810547 nb_pixel_total : 1429 time to create 1 rle with old method : 0.0016644001007080078 time for calcul the mask position with numpy : 0.011478424072265625 nb_pixel_total : 656418 time to create 1 rle with new method : 0.03831219673156738 time for calcul the mask position with numpy : 0.006973981857299805 nb_pixel_total : 5405 time to create 1 rle with old method : 0.006306886672973633 time for calcul the mask position with numpy : 0.006649971008300781 nb_pixel_total : 66798 time to create 1 rle with old method : 0.07076907157897949 time for calcul the mask position with numpy : 0.0062351226806640625 nb_pixel_total : 32678 time to create 1 rle with old method : 0.034842729568481445 time for calcul the mask position with numpy : 0.006566286087036133 nb_pixel_total : 73790 time to create 1 rle with old method : 0.0795753002166748 time for calcul the mask position with numpy : 0.006137371063232422 nb_pixel_total : 6370 time to create 1 rle with old method : 0.006894588470458984 time for calcul the mask position with numpy : 0.005984783172607422 nb_pixel_total : 20453 time to create 1 rle with old method : 0.02207779884338379 create new chi : 0.5049529075622559 time to delete rle : 0.0011372566223144531 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 7331 TO DO : save crop sub photo not yet done ! save time : 0.4178957939147949 nb_obj : 15 nb_hashtags : 6 time to prepare the origin masks : 1.4857721328735352 time for calcul the mask position with numpy : 0.01920175552368164 nb_pixel_total : 1326706 time to create 1 rle with new method : 0.04413008689880371 time for calcul the mask position with numpy : 0.006404876708984375 nb_pixel_total : 8888 time to create 1 rle with old method : 0.011364459991455078 time for calcul the mask position with numpy : 0.006480693817138672 nb_pixel_total : 5259 time to create 1 rle with old method : 0.0066144466400146484 time for calcul the mask position with numpy : 0.006931304931640625 nb_pixel_total : 18917 time to create 1 rle with old method : 0.021098613739013672 time for calcul the mask position with numpy : 0.006365299224853516 nb_pixel_total : 23770 time to create 1 rle with old method : 0.025919675827026367 time for calcul the mask position with numpy : 0.0063343048095703125 nb_pixel_total : 1504 time to create 1 rle with old method : 0.00177001953125 time for calcul the mask position with numpy : 0.0062444210052490234 nb_pixel_total : 4164 time to create 1 rle with old method : 0.0047855377197265625 time for calcul the mask position with numpy : 0.008934497833251953 nb_pixel_total : 5213 time to create 1 rle with old method : 0.005717754364013672 time for calcul the mask position with numpy : 0.00748133659362793 nb_pixel_total : 149045 time to create 1 rle with old method : 0.1709303855895996 time for calcul the mask position with numpy : 0.006339073181152344 nb_pixel_total : 29568 time to create 1 rle with old method : 0.03301191329956055 time for calcul the mask position with numpy : 0.0062215328216552734 nb_pixel_total : 3062 time to create 1 rle with old method : 0.0036132335662841797 time for calcul the mask position with numpy : 0.007394313812255859 nb_pixel_total : 4527 time to create 1 rle with old method : 0.008727550506591797 time for calcul the mask position with numpy : 0.008070945739746094 nb_pixel_total : 3761 time to create 1 rle with old method : 0.004373311996459961 time for calcul the mask position with numpy : 0.006779670715332031 nb_pixel_total : 8661 time to create 1 rle with old method : 0.010266542434692383 time for calcul the mask position with numpy : 0.00964212417602539 nb_pixel_total : 442763 time to create 1 rle with new method : 0.035663604736328125 time for calcul the mask position with numpy : 0.007523298263549805 nb_pixel_total : 37792 time to create 1 rle with old method : 0.0598292350769043 create new chi : 0.5776617527008057 time to delete rle : 0.0010552406311035156 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 7578 TO DO : save crop sub photo not yet done ! save time : 0.4400496482849121 nb_obj : 11 nb_hashtags : 4 time to prepare the origin masks : 1.4890475273132324 time for calcul the mask position with numpy : 0.015223264694213867 nb_pixel_total : 1074697 time to create 1 rle with new method : 0.037757158279418945 time for calcul the mask position with numpy : 0.00657963752746582 nb_pixel_total : 39182 time to create 1 rle with old method : 0.04293417930603027 time for calcul the mask position with numpy : 0.006616353988647461 nb_pixel_total : 3592 time to create 1 rle with old method : 0.004727602005004883 time for calcul the mask position with numpy : 0.007203817367553711 nb_pixel_total : 122594 time to create 1 rle with old method : 0.13673830032348633 time for calcul the mask position with numpy : 0.006517648696899414 nb_pixel_total : 2376 time to create 1 rle with old method : 0.0029172897338867188 time for calcul the mask position with numpy : 0.006584882736206055 nb_pixel_total : 5786 time to create 1 rle with old method : 0.006562709808349609 time for calcul the mask position with numpy : 0.0068073272705078125 nb_pixel_total : 24618 time to create 1 rle with old method : 0.029302120208740234 time for calcul the mask position with numpy : 0.006632089614868164 nb_pixel_total : 17762 time to create 1 rle with old method : 0.020776033401489258 time for calcul the mask position with numpy : 0.011610984802246094 nb_pixel_total : 513157 time to create 1 rle with new method : 0.04030966758728027 time for calcul the mask position with numpy : 0.007222414016723633 nb_pixel_total : 162588 time to create 1 rle with new method : 0.031873464584350586 time for calcul the mask position with numpy : 0.007404327392578125 nb_pixel_total : 52973 time to create 1 rle with old method : 0.058632612228393555 time for calcul the mask position with numpy : 0.00698089599609375 nb_pixel_total : 54275 time to create 1 rle with old method : 0.06245064735412598 create new chi : 0.5751872062683105 time to delete rle : 0.001430511474609375 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 10137 TO DO : save crop sub photo not yet done ! save time : 0.5784552097320557 nb_obj : 7 nb_hashtags : 2 time to prepare the origin masks : 0.6859695911407471 time for calcul the mask position with numpy : 0.015013694763183594 nb_pixel_total : 1150544 time to create 1 rle with new method : 0.03531289100646973 time for calcul the mask position with numpy : 0.006795406341552734 nb_pixel_total : 6174 time to create 1 rle with old method : 0.010105609893798828 time for calcul the mask position with numpy : 0.006589651107788086 nb_pixel_total : 610 time to create 1 rle with old method : 0.0010437965393066406 time for calcul the mask position with numpy : 0.0064849853515625 nb_pixel_total : 139 time to create 1 rle with old method : 0.0008792877197265625 time for calcul the mask position with numpy : 0.0070574283599853516 nb_pixel_total : 58645 time to create 1 rle with old method : 0.07211923599243164 time for calcul the mask position with numpy : 0.009721517562866211 nb_pixel_total : 521891 time to create 1 rle with new method : 0.028374195098876953 time for calcul the mask position with numpy : 0.00659489631652832 nb_pixel_total : 135552 time to create 1 rle with old method : 0.13732481002807617 time for calcul the mask position with numpy : 0.007060527801513672 nb_pixel_total : 200045 time to create 1 rle with new method : 0.027997255325317383 create new chi : 0.3823399543762207 time to delete rle : 0.0006330013275146484 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 7287 TO DO : save crop sub photo not yet done ! save time : 0.4190201759338379 nb_obj : 11 nb_hashtags : 6 time to prepare the origin masks : 1.0632669925689697 time for calcul the mask position with numpy : 0.030918598175048828 nb_pixel_total : 1403939 time to create 1 rle with new method : 0.05338597297668457 time for calcul the mask position with numpy : 0.0074405670166015625 nb_pixel_total : 563 time to create 1 rle with old method : 0.0009179115295410156 time for calcul the mask position with numpy : 0.007466316223144531 nb_pixel_total : 59397 time to create 1 rle with old method : 0.06739592552185059 time for calcul the mask position with numpy : 0.0069353580474853516 nb_pixel_total : 82 time to create 1 rle with old method : 0.00025010108947753906 time for calcul the mask position with numpy : 0.007413387298583984 nb_pixel_total : 19545 time to create 1 rle with old method : 0.021586894989013672 time for calcul the mask position with numpy : 0.00802922248840332 nb_pixel_total : 44886 time to create 1 rle with old method : 0.04938197135925293 time for calcul the mask position with numpy : 0.007352352142333984 nb_pixel_total : 7302 time to create 1 rle with old method : 0.008147716522216797 time for calcul the mask position with numpy : 0.007603645324707031 nb_pixel_total : 22180 time to create 1 rle with old method : 0.02475452423095703 time for calcul the mask position with numpy : 0.0105133056640625 nb_pixel_total : 381118 time to create 1 rle with new method : 0.053221702575683594 time for calcul the mask position with numpy : 0.007696866989135742 nb_pixel_total : 82993 time to create 1 rle with old method : 0.09143996238708496 time for calcul the mask position with numpy : 0.007041215896606445 nb_pixel_total : 36546 time to create 1 rle with old method : 0.038831472396850586 time for calcul the mask position with numpy : 0.007059574127197266 nb_pixel_total : 15049 time to create 1 rle with old method : 0.016260147094726562 create new chi : 0.5461390018463135 time to delete rle : 0.0008902549743652344 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 8170 TO DO : save crop sub photo not yet done ! save time : 0.4481184482574463 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.6038777828216553 time for calcul the mask position with numpy : 0.01882624626159668 nb_pixel_total : 1554963 time to create 1 rle with new method : 0.04372739791870117 time for calcul the mask position with numpy : 0.006623268127441406 nb_pixel_total : 15842 time to create 1 rle with old method : 0.018869638442993164 time for calcul the mask position with numpy : 0.006685972213745117 nb_pixel_total : 11547 time to create 1 rle with old method : 0.012768983840942383 time for calcul the mask position with numpy : 0.006238460540771484 nb_pixel_total : 2432 time to create 1 rle with old method : 0.004046440124511719 time for calcul the mask position with numpy : 0.009499073028564453 nb_pixel_total : 488816 time to create 1 rle with new method : 0.03423666954040527 create new chi : 0.16560983657836914 time to delete rle : 0.0005662441253662109 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6850 TO DO : save crop sub photo not yet done ! save time : 0.4017760753631592 nb_obj : 7 nb_hashtags : 2 time to prepare the origin masks : 0.6369245052337646 time for calcul the mask position with numpy : 0.013504743576049805 nb_pixel_total : 931187 time to create 1 rle with new method : 0.04595375061035156 time for calcul the mask position with numpy : 0.006181478500366211 nb_pixel_total : 478 time to create 1 rle with old method : 0.0005700588226318359 time for calcul the mask position with numpy : 0.006331920623779297 nb_pixel_total : 95903 time to create 1 rle with old method : 0.10264968872070312 time for calcul the mask position with numpy : 0.006263017654418945 nb_pixel_total : 8605 time to create 1 rle with old method : 0.00984501838684082 time for calcul the mask position with numpy : 0.006060123443603516 nb_pixel_total : 258 time to create 1 rle with old method : 0.0009195804595947266 time for calcul the mask position with numpy : 0.006293535232543945 nb_pixel_total : 55701 time to create 1 rle with old method : 0.05980372428894043 time for calcul the mask position with numpy : 0.01279449462890625 nb_pixel_total : 933030 time to create 1 rle with new method : 0.03844404220581055 time for calcul the mask position with numpy : 0.0063097476959228516 nb_pixel_total : 48438 time to create 1 rle with old method : 0.05438542366027832 create new chi : 0.38173818588256836 time to delete rle : 0.0008592605590820312 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 6791 TO DO : save crop sub photo not yet done ! save time : 0.42159414291381836 nb_obj : 7 nb_hashtags : 2 time to prepare the origin masks : 0.5614407062530518 time for calcul the mask position with numpy : 0.018015384674072266 nb_pixel_total : 1351271 time to create 1 rle with new method : 0.038985252380371094 time for calcul the mask position with numpy : 0.006474494934082031 nb_pixel_total : 12829 time to create 1 rle with old method : 0.020357131958007812 time for calcul the mask position with numpy : 0.006463766098022461 nb_pixel_total : 2728 time to create 1 rle with old method : 0.004452943801879883 time for calcul the mask position with numpy : 0.006419181823730469 nb_pixel_total : 652 time to create 1 rle with old method : 0.0017685890197753906 time for calcul the mask position with numpy : 0.010250329971313477 nb_pixel_total : 493747 time to create 1 rle with new method : 0.03598594665527344 time for calcul the mask position with numpy : 0.005932807922363281 nb_pixel_total : 8325 time to create 1 rle with old method : 0.009517192840576172 time for calcul the mask position with numpy : 0.006123065948486328 nb_pixel_total : 23243 time to create 1 rle with old method : 0.025770187377929688 time for calcul the mask position with numpy : 0.0069124698638916016 nb_pixel_total : 180805 time to create 1 rle with new method : 0.027443647384643555 create new chi : 0.2344346046447754 time to delete rle : 0.0004951953887939453 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 5652 TO DO : save crop sub photo not yet done ! save time : 0.34854578971862793 nb_obj : 8 nb_hashtags : 4 time to prepare the origin masks : 0.6715230941772461 time for calcul the mask position with numpy : 0.022962331771850586 nb_pixel_total : 1404396 time to create 1 rle with new method : 0.05125284194946289 time for calcul the mask position with numpy : 0.006586790084838867 nb_pixel_total : 49028 time to create 1 rle with old method : 0.05982828140258789 time for calcul the mask position with numpy : 0.0068819522857666016 nb_pixel_total : 45587 time to create 1 rle with old method : 0.049970388412475586 time for calcul the mask position with numpy : 0.007311582565307617 nb_pixel_total : 75309 time to create 1 rle with old method : 0.0815131664276123 time for calcul the mask position with numpy : 0.008950471878051758 nb_pixel_total : 404074 time to create 1 rle with new method : 0.042417287826538086 time for calcul the mask position with numpy : 0.006787538528442383 nb_pixel_total : 24448 time to create 1 rle with old method : 0.03573775291442871 time for calcul the mask position with numpy : 0.006807804107666016 nb_pixel_total : 8206 time to create 1 rle with old method : 0.009064435958862305 time for calcul the mask position with numpy : 0.006439924240112305 nb_pixel_total : 4272 time to create 1 rle with old method : 0.005033731460571289 time for calcul the mask position with numpy : 0.0064983367919921875 nb_pixel_total : 58280 time to create 1 rle with old method : 0.06416964530944824 create new chi : 0.48378896713256836 time to delete rle : 0.0010297298431396484 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 6758 TO DO : save crop sub photo not yet done ! save time : 0.3816039562225342 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 0.4279165267944336 time for calcul the mask position with numpy : 0.018947839736938477 nb_pixel_total : 1250044 time to create 1 rle with new method : 0.0320589542388916 time for calcul the mask position with numpy : 0.00623321533203125 nb_pixel_total : 9783 time to create 1 rle with old method : 0.010996341705322266 time for calcul the mask position with numpy : 0.006298065185546875 nb_pixel_total : 9481 time to create 1 rle with old method : 0.010904788970947266 time for calcul the mask position with numpy : 0.010684013366699219 nb_pixel_total : 587330 time to create 1 rle with new method : 0.03084588050842285 time for calcul the mask position with numpy : 0.006529569625854492 nb_pixel_total : 16960 time to create 1 rle with old method : 0.018958568572998047 time for calcul the mask position with numpy : 0.007456302642822266 nb_pixel_total : 200002 time to create 1 rle with new method : 0.03113722801208496 create new chi : 0.19455671310424805 time to delete rle : 0.0005292892456054688 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 5492 TO DO : save crop sub photo not yet done ! save time : 0.29825687408447266 map_output_result : {1332631994: (0.0, 'Should be the crop_list due to order', 0.0), 1332631884: (0.0, 'Should be the crop_list due to order', 0.0), 1332631801: (0.0, 'Should be the crop_list due to order', 0.0), 1332631780: (0.0, 'Should be the crop_list due to order', 0.0), 1332631774: (0.0, 'Should be the crop_list due to order', 0.0), 1332631738: (0.0, 'Should be the crop_list due to order', 0.0), 1332631735: (0.0, 'Should be the crop_list due to order', 0.0), 1332631732: (0.0, 'Should be the crop_list due to order', 0.0), 1332631729: (0.0, 'Should be the crop_list due to order', 0.0), 1332631726: (0.0, 'Should be the crop_list due to order', 0.0), 1332631722: (0.0, 'Should be the crop_list due to order', 0.0), 1332631714: (0.0, 'Should be the crop_list due to order', 0.0), 1332631711: (0.0, 'Should be the crop_list due to order', 0.0), 1332631706: (0.0, 'Should be the crop_list due to order', 0.0), 1332631700: (0.0, 'Should be the crop_list due to order', 0.0), 1332631685: (0.0, 'Should be the crop_list due to order', 0.0), 1332631682: (0.0, 'Should be the crop_list due to order', 0.0), 1332631643: (0.0, 'Should be the crop_list due to order', 0.0), 1332631639: (0.0, 'Should be the crop_list due to order', 0.0), 1332631635: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1332631994, 1332631884, 1332631801, 1332631780, 1332631774, 1332631738, 1332631735, 1332631732, 1332631729, 1332631726, 1332631722, 1332631714, 1332631711, 1332631706, 1332631700, 1332631685, 1332631682, 1332631643, 1332631639, 1332631635] Looping around the photos to save general results len do output : 20 /1332631994.Didn't retrieve data . /1332631884.Didn't retrieve data . /1332631801.Didn't retrieve data . /1332631780.Didn't retrieve data . /1332631774.Didn't retrieve data . /1332631738.Didn't retrieve data . /1332631735.Didn't retrieve data . /1332631732.Didn't retrieve data . /1332631729.Didn't retrieve data . /1332631726.Didn't retrieve data . /1332631722.Didn't retrieve data . /1332631714.Didn't retrieve data . /1332631711.Didn't retrieve data . /1332631706.Didn't retrieve data . /1332631700.Didn't retrieve data . /1332631685.Didn't retrieve data . /1332631682.Didn't retrieve data . /1332631643.Didn't retrieve data . /1332631639.Didn't retrieve data . /1332631635.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631994', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631884', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631801', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631780', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631774', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631738', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631735', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631732', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631729', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631726', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631722', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631714', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631711', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631706', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631700', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631685', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631682', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631643', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631639', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631635', None, None, None, None, None, '2526006') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 60 time used for this insertion : 0.014405012130737305 save_final save missing photos in datou_result : time spend for datou_step_exec : 31.438611030578613 time spend to save output : 0.01512002944946289 total time spend for step 5 : 31.453731060028076 step6:crop_condition Tue Feb 4 11:04:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 20 ! batch 1 Loaded 183 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663492_3588220 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7651309967041016 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663494_3588220 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9203908443450928 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663496_3588220 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6042752265930176 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663497_3588220 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9236578941345215 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663499_3588220 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.2907705307006836 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738663502_3588220 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.5544853210449219 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1332631994, 1332631884, 1332631801, 1332631780, 1332631774, 1332631738, 1332631735, 1332631732, 1332631729, 1332631726, 1332631722, 1332631714, 1332631711, 1332631706, 1332631700, 1332631685, 1332631682, 1332631643, 1332631639, 1332631635] Looping around the photos to save general results len do output : 18 /1334520743Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520744Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520745Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520746Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520747Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520751Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520757Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520758Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520759Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520761Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520762Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520763Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520764Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520765Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520772Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520773Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520774Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334520775Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631994', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631884', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631801', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631780', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631774', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631738', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631735', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631732', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631729', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631726', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631722', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631714', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631711', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631706', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631700', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631685', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631682', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631643', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631639', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631635', None, None, None, None, None, '2526006') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 74 time used for this insertion : 0.03300023078918457 save_final save missing photos in datou_result : time spend for datou_step_exec : 13.542824745178223 time spend to save output : 0.03374767303466797 total time spend for step 6 : 13.57657241821289 step7:ventilate_hashtags_in_portfolio Tue Feb 4 11:05:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20043200 get user id for portfolio 20043200 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043200 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','metal','papier','cartonnette','plastique','mal_croppe','flou','Teint_Dans_La_Masse','kraft','Carton_brun','autre_refus','Carton_gris')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043200 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','metal','papier','cartonnette','plastique','mal_croppe','flou','Teint_Dans_La_Masse','kraft','Carton_brun','autre_refus','Carton_gris')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043200 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','metal','papier','cartonnette','plastique','mal_croppe','flou','Teint_Dans_La_Masse','kraft','Carton_brun','autre_refus','Carton_gris')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20206471,20206472,20206473,20206474,20206475,20206476,20206477,20206478,20206479,20206480,20206481,20206482?tags=environnement,metal,papier,cartonnette,plastique,mal_croppe,flou,Teint_Dans_La_Masse,kraft,Carton_brun,autre_refus,Carton_gris Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1332631994, 1332631884, 1332631801, 1332631780, 1332631774, 1332631738, 1332631735, 1332631732, 1332631729, 1332631726, 1332631722, 1332631714, 1332631711, 1332631706, 1332631700, 1332631685, 1332631682, 1332631643, 1332631639, 1332631635] Looping around the photos to save general results len do output : 1 /20043200. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631994', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631884', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631801', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631780', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631774', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631738', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631735', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631732', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631729', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631726', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631722', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631714', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631711', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631706', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631700', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631685', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631682', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631643', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631639', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631635', None, None, None, None, None, '2526006') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.11548137664794922 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.7644364833831787 time spend to save output : 0.11579251289367676 total time spend for step 7 : 0.8802289962768555 step8:final Tue Feb 4 11:05:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1332631994: ('0.050673850254881816',), 1332631884: ('0.050673850254881816',), 1332631801: ('0.050673850254881816',), 1332631780: ('0.050673850254881816',), 1332631774: ('0.050673850254881816',), 1332631738: ('0.050673850254881816',), 1332631735: ('0.050673850254881816',), 1332631732: ('0.050673850254881816',), 1332631729: ('0.050673850254881816',), 1332631726: ('0.050673850254881816',), 1332631722: ('0.050673850254881816',), 1332631714: ('0.050673850254881816',), 1332631711: ('0.050673850254881816',), 1332631706: ('0.050673850254881816',), 1332631700: ('0.050673850254881816',), 1332631685: ('0.050673850254881816',), 1332631682: ('0.050673850254881816',), 1332631643: ('0.050673850254881816',), 1332631639: ('0.050673850254881816',), 1332631635: ('0.050673850254881816',)} new output for save of step final : {1332631994: ('0.050673850254881816',), 1332631884: ('0.050673850254881816',), 1332631801: ('0.050673850254881816',), 1332631780: ('0.050673850254881816',), 1332631774: ('0.050673850254881816',), 1332631738: ('0.050673850254881816',), 1332631735: ('0.050673850254881816',), 1332631732: ('0.050673850254881816',), 1332631729: ('0.050673850254881816',), 1332631726: ('0.050673850254881816',), 1332631722: ('0.050673850254881816',), 1332631714: ('0.050673850254881816',), 1332631711: ('0.050673850254881816',), 1332631706: ('0.050673850254881816',), 1332631700: ('0.050673850254881816',), 1332631685: ('0.050673850254881816',), 1332631682: ('0.050673850254881816',), 1332631643: ('0.050673850254881816',), 1332631639: ('0.050673850254881816',), 1332631635: ('0.050673850254881816',)} [1332631994, 1332631884, 1332631801, 1332631780, 1332631774, 1332631738, 1332631735, 1332631732, 1332631729, 1332631726, 1332631722, 1332631714, 1332631711, 1332631706, 1332631700, 1332631685, 1332631682, 1332631643, 1332631639, 1332631635] Looping around the photos to save general results len do output : 20 /1332631994.Didn't retrieve data . /1332631884.Didn't retrieve data . /1332631801.Didn't retrieve data . /1332631780.Didn't retrieve data . /1332631774.Didn't retrieve data . /1332631738.Didn't retrieve data . /1332631735.Didn't retrieve data . /1332631732.Didn't retrieve data . /1332631729.Didn't retrieve data . /1332631726.Didn't retrieve data . /1332631722.Didn't retrieve data . /1332631714.Didn't retrieve data . /1332631711.Didn't retrieve data . /1332631706.Didn't retrieve data . /1332631700.Didn't retrieve data . /1332631685.Didn't retrieve data . /1332631682.Didn't retrieve data . /1332631643.Didn't retrieve data . /1332631639.Didn't retrieve data . /1332631635.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631994', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631884', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631801', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631780', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631774', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631738', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631735', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631732', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631729', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631726', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631722', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631714', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631711', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631706', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631700', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631685', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631682', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631643', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631639', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631635', None, None, None, None, None, '2526006') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 60 time used for this insertion : 0.014199256896972656 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.12115597724914551 time spend to save output : 0.015154838562011719 total time spend for step 8 : 0.13631081581115723 step9:velours_tree Tue Feb 4 11:05:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.03323078155517578 time spend to save output : 3.647804260253906e-05 total time spend for step 9 : 0.03326725959777832 step10:send_mail_cod Tue Feb 4 11:05:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin in order to get the selector url, please entre the license of selector results_Auto_P20043200_04-02-2025_11_05_05.pdf 20206472 imagette202064721738663505 20206474 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202064741738663505 20206475 change filename to text .change filename to text .change filename to text .change filename to text .imagette202064751738663506 20206476 imagette202064761738663506 20206477 imagette202064771738663506 20206479 change filename to text .imagette202064791738663506 20206481 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202064811738663507 20206478 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202064781738663508 20206480 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202064801738663508 20206482 change filename to text .change filename to text .imagette202064821738663509 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20043200 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20206471,20206472,20206473,20206474,20206475,20206476,20206477,20206478,20206479,20206480,20206481,20206482?tags=environnement,metal,papier,cartonnette,plastique,mal_croppe,flou,Teint_Dans_La_Masse,kraft,Carton_brun,autre_refus,Carton_gris your option no_mail is active, we will not send the real mail to your client args[1332631994] : ('0.050673850254881816',) no score found for photo 1332631994 We are sending mail with results at report@fotonower.com args[1332631884] : ('0.050673850254881816',) no score found for photo 1332631884 We are sending mail with results at report@fotonower.com args[1332631801] : ('0.050673850254881816',) no score found for photo 1332631801 We are sending mail with results at report@fotonower.com args[1332631780] : ('0.050673850254881816',) no score found for photo 1332631780 We are sending mail with results at report@fotonower.com args[1332631774] : ('0.050673850254881816',) no score found for photo 1332631774 We are sending mail with results at report@fotonower.com args[1332631738] : ('0.050673850254881816',) no score found for photo 1332631738 We are sending mail with results at report@fotonower.com args[1332631735] : ('0.050673850254881816',) no score found for photo 1332631735 We are sending mail with results at report@fotonower.com args[1332631732] : ('0.050673850254881816',) no score found for photo 1332631732 We are sending mail with results at report@fotonower.com args[1332631729] : ('0.050673850254881816',) no score found for photo 1332631729 We are sending mail with results at report@fotonower.com args[1332631726] : ('0.050673850254881816',) no score found for photo 1332631726 We are sending mail with results at report@fotonower.com args[1332631722] : ('0.050673850254881816',) no score found for photo 1332631722 We are sending mail with results at report@fotonower.com args[1332631714] : ('0.050673850254881816',) no score found for photo 1332631714 We are sending mail with results at report@fotonower.com args[1332631711] : ('0.050673850254881816',) no score found for photo 1332631711 We are sending mail with results at report@fotonower.com args[1332631706] : ('0.050673850254881816',) no score found for photo 1332631706 We are sending mail with results at report@fotonower.com args[1332631700] : ('0.050673850254881816',) no score found for photo 1332631700 We are sending mail with results at report@fotonower.com args[1332631685] : ('0.050673850254881816',) no score found for photo 1332631685 We are sending mail with results at report@fotonower.com args[1332631682] : ('0.050673850254881816',) no score found for photo 1332631682 We are sending mail with results at report@fotonower.com args[1332631643] : ('0.050673850254881816',) no score found for photo 1332631643 We are sending mail with results at report@fotonower.com args[1332631639] : ('0.050673850254881816',) no score found for photo 1332631639 We are sending mail with results at report@fotonower.com args[1332631635] : ('0.050673850254881816',) no score found for photo 1332631635 We are sending mail with results at report@fotonower.com refus_total : 0.050673850254881816 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20043200 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1332629845,1332611705,1332630692,1332630718,1332612527,1332611840,1332631714,1332629719,1332631157,1332629777,1332611546,1332610733,1332631016,1332612349,1332631073,1332631575,1332611297,1332612319,1332630546,1332631431) Found this number of photos: 20 begin to download photo : 1332629845 begin to download photo : 1332611840 begin to download photo : 1332611546 begin to download photo : 1332631575 download finish for photo 1332611546 begin to download photo : 1332610733 download finish for photo 1332611840 begin to download photo : 1332631714 download finish for photo 1332629845 begin to download photo : 1332611705 download finish for photo 1332631575 begin to download photo : 1332611297 download finish for photo 1332610733 begin to download photo : 1332631016 download finish for photo 1332611705 begin to download photo : 1332630692 download finish for photo 1332611297 begin to download photo : 1332612319 download finish for photo 1332612319 begin to download photo : 1332630546 download finish for photo 1332631016 begin to download photo : 1332612349 download finish for photo 1332630692 begin to download photo : 1332630718 download finish for photo 1332631714 begin to download photo : 1332629719 download finish for photo 1332612349 begin to download photo : 1332631073 download finish for photo 1332630718 begin to download photo : 1332612527 download finish for photo 1332631073 download finish for photo 1332629719 begin to download photo : 1332631157 download finish for photo 1332630546 begin to download photo : 1332631431 download finish for photo 1332612527 download finish for photo 1332631431 download finish for photo 1332631157 begin to download photo : 1332629777 download finish for photo 1332629777 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043200_04-02-2025_11_05_05.pdf results_Auto_P20043200_04-02-2025_11_05_05.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043200_04-02-2025_11_05_05.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20043200','results_Auto_P20043200_04-02-2025_11_05_05.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043200_04-02-2025_11_05_05.pdf','pdf','','0.47','0.050673850254881816') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1332631994, 1332631884, 1332631801, 1332631780, 1332631774, 1332631738, 1332631735, 1332631732, 1332631729, 1332631726, 1332631722, 1332631714, 1332631711, 1332631706, 1332631700, 1332631685, 1332631682, 1332631643, 1332631639, 1332631635] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631994', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631884', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631801', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631780', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631774', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631738', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631735', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631732', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631729', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631726', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631722', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631714', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631711', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631706', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631700', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631685', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631682', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631643', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631639', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631635', None, None, None, None, None, '2526006') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 20 time used for this insertion : 0.01507258415222168 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.861417770385742 time spend to save output : 0.01541590690612793 total time spend for step 10 : 6.87683367729187 step11:split_time_score Tue Feb 4 11:05:12 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('11', 211),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 29012025 20043200 Nombre de photos uploadées : 211 / 23040 (0%) 29012025 20043200 Nombre de photos taguées (types de déchets): 0 / 211 (0%) 29012025 20043200 Nombre de photos taguées (volume) : 0 / 211 (0%) elapsed_time : load_data_split_time_score 2.384185791015625e-06 elapsed_time : order_list_meta_photo_and_scores 6.198883056640625e-06 ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.00581812858581543 elapsed_time : insert_dashboard_record_day_entry 0.023070335388183594 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.05964569986773992 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20038647_29-01-2025_09_33_49.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20038647 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20038647 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20038303 order by id desc limit 1 Qualite : 0.05453990257520215 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20036989_29-01-2025_08_12_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20036989 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20036989 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20037835 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20037561 order by id desc limit 1 Qualite : 0.029203380749725394 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20037563_29-01-2025_08_42_25.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20037563 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20037563 AND mptpi.`type`=3726 To do Qualite : 0.035880779652306634 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20038304_29-01-2025_09_22_44.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20038304 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20038304 AND mptpi.`type`=3726 To do Qualite : 0.04420808919501898 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043199_04-02-2025_11_04_26.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20043199 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043199 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20040004 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20040005 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042811 order by id desc limit 1 Qualite : 0.04179473612560559 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20042812_29-01-2025_13_32_57.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042812 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20042812 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042942 order by id desc limit 1 Qualite : 0.050673850254881816 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043200_04-02-2025_11_05_05.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20043200 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043200 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042552 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042169 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042170 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20042172 order by id desc limit 1 Qualite : 0.03250609913107722 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043201_29-01-2025_14_11_08.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20043201 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043201 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20043550 order by id desc limit 1 Qualite : 0.009347489222823262 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20043551_29-01-2025_14_41_40.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20043551 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20043551 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20043552 order by id desc limit 1 Qualite : 0.040548452513978116 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20044107_29-01-2025_15_32_02.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20044107 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20044107 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20044108 order by id desc limit 1 Qualite : 0.04088001272734784 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20044298_29-01-2025_15_27_01.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20044298 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20044298 AND mptpi.`type`=3726 To do Qualite : 0.025866735673049823 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20044671_29-01-2025_16_01_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20044671 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20044671 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20045174 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20045175 order by id desc limit 1 Qualite : 0.036707148122498 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20045657_29-01-2025_19_01_15.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20045657 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20045657 AND mptpi.`type`=3726 To do Qualite : 0.04449978208679873 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20045658_29-01-2025_16_51_25.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20045658 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20045658 AND mptpi.`type`=3726 To do Qualite : 0.06389528464627381 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20045933_29-01-2025_17_33_28.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20045933 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20045933 AND mptpi.`type`=3726 To do Qualite : 0.05614592496812481 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20046299_29-01-2025_18_02_54.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20046299 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20046299 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20047762 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20047763 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20047764 order by id desc limit 1 Qualite : 0.0513536384207915 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20047765_29-01-2025_19_33_11.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20047765 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20047765 AND mptpi.`type`=3726 To do Qualite : 0.030150108126073005 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20048159_29-01-2025_20_01_14.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20048159 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20048159 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20048160 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20048632 order by id desc limit 1 Qualite : 0.02217739158266012 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20048633_29-01-2025_20_21_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20048633 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20048633 AND mptpi.`type`=3726 To do Qualite : 0.04372363561039585 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20048832_29-01-2025_20_27_01.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20048832 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20048832 AND mptpi.`type`=3726 To do Qualite : 0.007334799965609816 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20049145_29-01-2025_21_00_41.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20049145 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20049145 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20105092 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066611 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066612 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066613 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066615 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066616 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066617 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20105093 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066619 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066620 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20066621 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20105118 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'29012025': {'nb_upload': 211, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1332631994, 1332631884, 1332631801, 1332631780, 1332631774, 1332631738, 1332631735, 1332631732, 1332631729, 1332631726, 1332631722, 1332631714, 1332631711, 1332631706, 1332631700, 1332631685, 1332631682, 1332631643, 1332631639, 1332631635] Looping around the photos to save general results len do output : 1 /20043200Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631994', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631884', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631801', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631780', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631774', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631738', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631735', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631732', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631729', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631726', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631722', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631714', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631711', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631706', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631700', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631685', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631682', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631643', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631639', None, None, None, None, None, '2526006') ('3459', None, None, None, None, None, None, None, '2526006') ('3459', '20043200', '1332631635', None, None, None, None, None, '2526006') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.01728534698486328 save_final save missing photos in datou_result : time spend for datou_step_exec : 26.288460969924927 time spend to save output : 0.01751232147216797 total time spend for step 11 : 26.305973291397095 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 77.46user 26.05system 2:23.87elapsed 71%CPU (0avgtext+0avgdata 3067108maxresident)k 1024inputs+55808outputs (4major+2307166minor)pagefaults 0swaps