python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 20 -a ' -a 4189' -s datou_current_4189 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 4194119 load datou : 4189 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 list_input_json: {} Current got : datou_id : 4189, datou_cur_ids : ['2545260'] with mtr_portfolio_ids : ['20203696'] and first list_photo_ids : [] new path : /proc/4194119/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : split_time_score over limit max, limiting to limit_max 100 list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB time to download the photos : 0.030215024948120117 About to test input to load Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 1 step1:split_time_score Tue Feb 4 14:30:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec begin split time score 2022-04-13 10:29:59 0 TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3379, 'mtr_user_id': 31, 'name': 'learn_classif_flux_maj_generique_effnet_v2_s_02062022', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'aluminium,ela,film_pedb,flux_dev,jrm,pcm,pcnc,pehd_pp,pet_clair,refus,tapis_vide', 'svm_portfolios_learning': '5515864,5515840,5515844,5515850,6244400,6237996,6237998,5515847,5515841,5515868,5515866', 'photo_hashtag_type': 4374, 'photo_desc_type': 5680, 'type_classification': 'tf_classification2', 'hashtag_id_list': '493546845,492741797,2107760237,2107760238,495916461,560181804,1284539308,2107760239,2107755846,538914404,2107748999'}] thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 358), ('14', 165)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {1: 499} L04022025 20203696 Nombre de photos uploadées : 523 / 23040 (2%) 04022025 20203696 Nombre de photos taguées (types de déchets): 499 / 523 (95%) 04022025 20203696 Nombre de photos taguées (volume) : 499 / 523 (95%) elapsed_time : load_data_split_time_score 4.5299530029296875e-06 elapsed_time : order_list_meta_photo_and_scores 0.0002574920654296875 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL???????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.06630420684814453 L elapsed_time : insert_dashboard_record_day_entry 0.024500370025634766 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL Creating list_photo_by_hashtags Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None elapsed_time : list_photo_by_hashtags 0.05166339874267578 Creating list_photo_total elapsed_time : select_descriptors 21.57310652732849 04022025 20203696 Nombre de photos avec descriptors (type 5680) : 391 / 415 (94%) Missing descriptors for photos 0 and 1334501667 L0:00:00|ON:Missing descriptors for photos 1334501667 and 1334501671 LMissing descriptors for photos 1334501671 and 1334501680 LLMissing descriptors for photos 1334501688 and 1334501896 LMissing descriptors for photos 1334501896 and 1334502020 LMissing descriptors for photos 1334502020 and 1334502147 LMissing descriptors for photos 1334502147 and 1334502272 LMissing descriptors for photos 1334502272 and 1334502391 LMissing descriptors for photos 1334502391 and 1334502516 LMissing descriptors for photos 1334502516 and 1334502801 LMissing descriptors for photos 1334502801 and 1334502805 LMissing descriptors for photos 1334502805 and 1334502811 LMissing descriptors for photos 1334502811 and 1334502813 LMissing descriptors for photos 1334502813 and 1334502815 LMissing descriptors for photos 1334502815 and 1334502816 LMissing descriptors for photos 1334502816 and 1334503152 LMissing descriptors for photos 1334503152 and 1334503155 LMissing descriptors for photos 1334503155 and 1334503175 LMissing descriptors for photos 1334503175 and 1334503176 LMissing descriptors for photos 1334503176 and 1334503191 LLLMissing descriptors for photos 1334503444 and 1334503446 LLMissing descriptors for photos 1334503456 and 1334503905 LMissing descriptors for photos 1334503905 and 1334504119 LMissing descriptors for photos 1334504119 and 1334504270 LMissing descriptors for photos 1334504270 and 1334504352 LMissing descriptors for photos 1334504352 and 1334504398 LMissing descriptors for photos 1334504398 and 1334504488 LMissing descriptors for photos 1334504488 and 1334504894 LMissing descriptors for photos 1334504894 and 1334505039 LMissing descriptors for photos 1334505039 and 1334505131 LMissing descriptors for photos 1334505131 and 1334505368 LMissing descriptors for photos 1334505368 and 1334505605 LMissing descriptors for photos 1334505605 and 1334505782 LMissing descriptors for photos 1334505782 and 1334506060 LMissing descriptors for photos 1334506060 and 1334506069 LMissing descriptors for photos 1334506069 and 1334506094 LMissing descriptors for photos 1334506094 and 1334506128 LMissing descriptors for photos 1334506128 and 1334506200 LMissing descriptors for photos 1334506200 and 1334506250 LMissing descriptors for photos 1334506250 and 1334506607 LMissing descriptors for photos 1334506607 and 1334506672 LMissing descriptors for photos 1334506672 and 1334506680 LMissing descriptors for photos 1334506680 and 1334506696 LMissing descriptors for photos 1334506696 and 1334506756 LMissing descriptors for photos 1334506756 and 1334506780 LMissing descriptors for photos 1334506780 and 1334507067 LMissing descriptors for photos 1334507067 and 1334507073 LMissing descriptors for photos 1334507073 and 1334507076 LMissing descriptors for photos 1334507076 and 1334507080 LMissing descriptors for photos 1334507080 and 1334507087 LMissing descriptors for photos 1334507087 and 1334507092 LMissing descriptors for photos 1334507092 and 1334507799 LMissing descriptors for photos 1334507799 and 1334507919 LMissing descriptors for photos 1334507919 and 1334507997 LMissing descriptors for photos 1334507997 and 1334508004 LMissing descriptors for photos 1334508004 and 1334508234 LMissing descriptors for photos 1334508234 and 1334508239 LMissing descriptors for photos 1334508239 and 1334508306 LMissing descriptors for photos 1334508306 and 1334508355 LLL0:22:01.359062|OFF:0:00:00|ON:LL0:00:58.721639|OFF:0:00:00|ON:LMissing descriptors for photos 1334517329 and 1334517334 LLL0:33:00.247193|OFF:LL0:00:00|ON:LLLLLLLLLLLLLLLLLLLLLLMissing descriptors for photos 1334560003 and 1334560132 LMissing descriptors for photos 1334560132 and 1334560137 LMissing descriptors for photos 1334560137 and 1334560139 LMissing descriptors for photos 1334560139 and 1334560140 LMissing descriptors for photos 1334560140 and 1334560144 LMissing descriptors for photos 1334560144 and 1334560148 LMissing descriptors for photos 1334560148 and 1334560198 LMissing descriptors for photos 1334560198 and 1334560203 LLLMissing descriptors for photos 1334560221 and 1334560225 LMissing descriptors for photos 1334560225 and 1334560229 LMissing descriptors for photos 1334560229 and 1334560342 LMissing descriptors for photos 1334560342 and 1334560344 LMissing descriptors for photos 1334560344 and 1334560348 LMissing descriptors for photos 1334560348 and 1334560353 LMissing descriptors for photos 1334560353 and 1334560359 LMissing descriptors for photos 1334560359 and 1334560363 LMissing descriptors for photos 1334560363 and 1334560385 LLLMissing descriptors for photos 1334560392 and 1334560397 LLLMissing descriptors for photos 1334560407 and 1334560483 LLLMissing descriptors for photos 1334560495 and 1334560499 LMissing descriptors for photos 1334560499 and 1334560504 LMissing descriptors for photos 1334560504 and 1334560519 LMissing descriptors for photos 1334560519 and 1334560553 LMissing descriptors for photos 1334560553 and 1334560554 LMissing descriptors for photos 1334560554 and 1334560557 LMissing descriptors for photos 1334560557 and 1334560562 LMissing descriptors for photos 1334560562 and 1334560569 LMissing descriptors for photos 1334560569 and 1334560631 LMissing descriptors for photos 1334560631 and 1334560633 LMissing descriptors for photos 1334560633 and 1334560635 LMissing descriptors for photos 1334560635 and 1334560636 LMissing descriptors for photos 1334560636 and 1334560638 LMissing descriptors for photos 1334560638 and 1334560709 LMissing descriptors for photos 1334560709 and 1334560765 LMissing descriptors for photos 1334560765 and 1334560813 LMissing descriptors for photos 1334560813 and 1334560867 LMissing descriptors for photos 1334560867 and 1334560868 LLLLLLLLLLL3:13:59.618095|OFF:L0:00:59.739309|ON:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLERROR : Hum hum, what can we do for different size of descriptors (ignore the difference ) : 1280 vs 0 photo_id : 1334564190 photo_id_prec : 1334564123 LMissing descriptors for photos 1334564190 and 1334564200 LMissing descriptors for photos 1334564200 and 1334564239 LMissing descriptors for photos 1334564239 and 1334564243 LMissing descriptors for photos 1334564243 and 1334564244 LMissing descriptors for photos 1334564244 and 1334564283 LMissing descriptors for photos 1334564283 and 1334564429 LMissing descriptors for photos 1334564429 and 1334564446 LMissing descriptors for photos 1334564446 and 1334564457 LMissing descriptors for photos 1334564457 and 1334564479 LMissing descriptors for photos 1334564479 and 1334564512 LMissing descriptors for photos 1334564512 and 1334564516 LMissing descriptors for photos 1334564516 and 1334564557 LMissing descriptors for photos 1334564557 and 1334564573 LMissing descriptors for photos 1334564573 and 1334564656 LMissing descriptors for photos 1334564656 and 1334564661 LMissing descriptors for photos 1334564661 and 1334564677 LMissing descriptors for photos 1334564677 and 1334564688 LMissing descriptors for photos 1334564688 and 1334564856 LMissing descriptors for photos 1334564856 and 1334564861 LMissing descriptors for photos 1334564861 and 1334564875 LMissing descriptors for photos 1334564875 and 1334564995 LMissing descriptors for photos 1334564995 and 1334565052 LMissing descriptors for photos 1334565052 and 1334565065 L 04022025 Removing 5 photos because of the 'same image' condition Total on : 14999.945989 list_time_on Total off : 59.739309 list_time_off dist_desc begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 523 time used for this insertion : 0.09242081642150879 photos_removed : len 5 elapsed_time : remove_photo_duplicate 0.16374993324279785 To do, maybe not at the correct place ! .L.L.L..L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L..L..L.L..L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L..................................................................L.......L.........................................................................................................................L.L............................................................................L.L........L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.Lforce hashtag to JRM elapsed_time : CREATE_PORT_BATCH_BY_HOUR 0.018515825271606445 NUMBER BATCH : 6 list_ponderation used : [1e-05, 1e-05, 1e-05, 1e-05, 1e-05] , list_hashtag_class_create_as_list : ['jrm'] result_one_balle_Type_JRM:{'day': '04022025', 'map_nb_amount': {0: 3, 1: 22, 2: 0, 3: 1, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 840.10897397995, 'nb_balles_papier': 0.00027000000000000006, 'begin_time_port': 'image_04022025_10_00_02_675205m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.00027000000000000006 result_one_balle_Type_JRM:{'day': '04022025', 'map_nb_amount': {0: 15, 1: 68, 2: 2, 3: 2, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 840.3962059020996, 'nb_balles_papier': 0.0008800000000000017, 'begin_time_port': 'image_04022025_10_15_02_503190m0.jpg 1e-05 for time 1, id_amount 1 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0008800000000000017 result_one_balle_Type_JRM:{'day': '04022025', 'map_nb_amount': {0: 9, 1: 76, 2: 1, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 840.075168132782, 'nb_balles_papier': 0.0008700000000000016, 'begin_time_port': 'image_04022025_10_30_02_629530m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0008700000000000016 LLLLLLLLLLLLLLLLLLLLLLLLLresult_one_balle_Type_JRM:{'day': '04022025', 'map_nb_amount': {0: 13, 1: 71, 2: 0, 3: 1, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 839.6714918613434, 'nb_balles_papier': 0.0008700000000000016, 'begin_time_port': 'image_04022025_10_45_02_445794m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0008700000000000016 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLresult_one_balle_Type_JRM:{'day': '04022025', 'map_nb_amount': {0: 5, 1: 39, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 839.5746469497681, 'nb_balles_papier': 0.0004400000000000005, 'begin_time_port': 'image_04022025_14_00_02_870831_0000.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0004400000000000005 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info LLresult_one_balle_Type_JRM:{'day': '04022025', 'map_nb_amount': {0: 5, 1: 45, 2: 1, 3: 1, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 719.1234579086304, 'nb_balles_papier': 0.0005300000000000007, 'begin_time_port': 'image_04022025_14_15_03_248893_0000.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0005300000000000007 We have rejected 0 photos because of the batch_size condition ! NUMBER BATCH list_of_portfolios_to_create : 6 list_same_port_ids : [20204899] find same portfolio which already exist 20204899 , we will use it list_same_port_ids : [20206950] find same portfolio which already exist 20206950 , we will use it list_same_port_ids : [20206952] find same portfolio which already exist 20206952 , we will use it list_same_port_ids : [] https://marlene.fotonower.com/api/v1/secured/portfolio/new?name=JRM_diff_batch__04022025_10_45_02_445794&access_token=b05576c56a0e42ad0cb9b46155f68f82 list_same_port_ids : [] https://marlene.fotonower.com/api/v1/secured/portfolio/new?name=JRM_diff_batch__04022025_14_00_02_870831&access_token=b05576c56a0e42ad0cb9b46155f68f82 list_same_port_ids : [] https://marlene.fotonower.com/api/v1/secured/portfolio/new?name=JRM_diff_batch__04022025_14_15_03_248893&access_token=b05576c56a0e42ad0cb9b46155f68f82 Qualite : 0.1394988903120916 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20204899 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20206950 AND mptpi.`type`=3726 To do Qualite : 0.061665249093740264 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20206952 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20211488 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20211490 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20211494 AND mptpi.`type`=3726 To do elapsed_time : count_nb_balles_and_create_portfolio 6.957463026046753 # DISPLAY ALL COLLECTED DATA : {'04022025': {'nb_upload': 523, 'nb_taggue_class': 499, 'nb_taggue_densite': 499, 'nb_descriptors': 391}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1334565065, 1334565052, 1334564995, 1334564875, 1334564861, 1334564856, 1334564688, 1334564677, 1334564661, 1334564656, 1334564573, 1334564557, 1334564516, 1334564512, 1334564479, 1334564457, 1334564446, 1334564429, 1334564283, 1334564244, 1334564243, 1334564239, 1334564200, 1334564190, 1334564123, 1334564120, 1334564104, 1334564095, 1334564087, 1334564081, 1334564063, 1334564060, 1334564056, 1334564052, 1334563955, 1334563806, 1334563650, 1334563637, 1334563596, 1334563592, 1334563585, 1334563564, 1334563314, 1334563282, 1334563274, 1334563270, 1334563266, 1334563251, 1334563050, 1334563043, 1334563037, 1334563018, 1334562914, 1334562868, 1334562533, 1334562364, 1334562196, 1334562191, 1334562156, 1334562133, 1334561971, 1334561870, 1334561865, 1334561848, 1334561845, 1334561842, 1334561779, 1334561776, 1334561771, 1334561766, 1334561699, 1334561694, 1334561640, 1334561631, 1334561628, 1334561622, 1334561620, 1334561617, 1334561565, 1334561563, 1334561559, 1334561552, 1334561548, 1334561540, 1334561524, 1334561521, 1334561518, 1334561516, 1334561510, 1334561505, 1334561414, 1334561398, 1334561394, 1334561391, 1334561387, 1334561381, 1334561277, 1334561269, 1334561238, 1334561237] Looping around the photos to save general results len do output : 1 /20203696Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334565065', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334565052', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564995', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564875', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564861', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564856', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564688', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564677', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564661', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564656', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564573', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564557', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564516', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564512', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564479', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564457', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564446', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564429', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564283', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564244', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564243', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564239', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564200', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564190', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564123', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564120', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564104', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564095', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564087', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564081', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564063', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564060', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564056', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334564052', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563955', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563806', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563650', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563637', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563596', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563592', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563585', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563564', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563314', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563282', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563274', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563270', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563266', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563251', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563050', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563043', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563037', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334563018', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334562914', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334562868', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334562533', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334562364', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334562196', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334562191', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334562156', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334562133', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561971', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561870', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561865', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561848', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561845', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561842', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561779', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561776', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561771', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561766', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561699', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561694', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561640', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561631', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561628', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561622', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561620', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561617', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561565', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561563', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561559', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561552', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561548', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561540', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561524', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561521', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561518', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561516', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561510', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561505', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561414', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561398', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561394', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561391', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561387', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561381', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561277', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561269', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561238', None, None, None, None, None, '2545260') ('4189', None, None, None, None, None, None, None, '2545260') ('4189', '20203696', '1334561237', None, None, None, None, None, '2545260') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 101 time used for this insertion : 0.02521681785583496 save_final save missing photos in datou_result : time spend for datou_step_exec : 28.98612141609192 time spend to save output : 0.02579212188720703 total time spend for step 1 : 29.011913537979126 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 2 len(input) + len(total_photo_id_missing) : 1 set_done_treatment 2.70user 0.59system 0:31.68elapsed 10%CPU (0avgtext+0avgdata 105776maxresident)k 192inputs+3288outputs (0major+61152minor)pagefaults 0swaps