python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 20 -a ' -a 4189' -s datou_current_4189 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1775329 load datou : 4189 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 list_input_json: {} Current got : datou_id : 4189, datou_cur_ids : ['4346817'] with mtr_portfolio_ids : ['24834339'] and first list_photo_ids : [] new path : /proc/1775329/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : split_time_score over limit max, limiting to limit_max 100 list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB time to download the photos : 0.09414291381835938 About to test input to load Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 1 step1:split_time_score Thu Feb 19 02:30:30 2026 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec begin split time score 2022-04-13 10:29:59 0 TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3379, 'mtr_user_id': 31, 'name': 'learn_classif_flux_maj_generique_effnet_v2_s_02062022', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'aluminium,ela,film_pedb,flux_dev,jrm,pcm,pcnc,pehd_pp,pet_clair,refus,tapis_vide', 'svm_portfolios_learning': '5515864,5515840,5515844,5515850,6244400,6237996,6237998,5515847,5515841,5515868,5515866', 'photo_hashtag_type': 4374, 'photo_desc_type': 5680, 'type_classification': 'tf_classification2', 'hashtag_id_list': '493546845,492741797,2107760237,2107760238,495916461,560181804,1284539308,2107760239,2107755846,538914404,2107748999'}] thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 516), ('14', 359)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {1: 856} L11072025 24834339 Nombre de photos uploadées : 875 / 23040 (3%) 11072025 24834339 Nombre de photos taguées (types de déchets): 856 / 875 (97%) 11072025 24834339 Nombre de photos taguées (volume) : 856 / 875 (97%) elapsed_time : load_data_split_time_score 2.384185791015625e-06 elapsed_time : order_list_meta_photo_and_scores 0.0003268718719482422 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL??????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.03651547431945801 LCatched exception ! Connect or reconnect ! Catched exception ! Connect or reconnect ! elapsed_time : insert_dashboard_record_day_entry 0.24232935905456543 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL Creating list_photo_by_hashtags Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None elapsed_time : list_photo_by_hashtags 0.06511092185974121 Creating list_photo_total elapsed_time : select_descriptors 56.989803075790405 11072025 24834339 Nombre de photos avec descriptors (type 5680) : 675 / 694 (97%) Missing descriptors for photos 0 and 1371565506 L0:00:00|ON:Missing descriptors for photos 1371565506 and 1371565589 LMissing descriptors for photos 1371565589 and 1371565674 LMissing descriptors for photos 1371565674 and 1371565678 LMissing descriptors for photos 1371565678 and 1371565767 LMissing descriptors for photos 1371565767 and 1371565801 LMissing descriptors for photos 1371565801 and 1371565913 LMissing descriptors for photos 1371565913 and 1371565919 LMissing descriptors for photos 1371565919 and 1371565942 LMissing descriptors for photos 1371565942 and 1371566100 LMissing descriptors for photos 1371566100 and 1371566105 LMissing descriptors for photos 1371566105 and 1371566118 LMissing descriptors for photos 1371566118 and 1371566200 LMissing descriptors for photos 1371566200 and 1371566311 LMissing descriptors for photos 1371566311 and 1371566326 LMissing descriptors for photos 1371566326 and 1371566421 LMissing descriptors for photos 1371566421 and 1371566537 LMissing descriptors for photos 1371566537 and 1371566589 LMissing descriptors for photos 1371566589 and 1371566606 LMissing descriptors for photos 1371566606 and 1371566667 LMissing descriptors for photos 1371566667 and 1371566813 LMissing descriptors for photos 1371566813 and 1371566894 LMissing descriptors for photos 1371566894 and 1371566896 LMissing descriptors for photos 1371566896 and 1371566978 LMissing descriptors for photos 1371566978 and 1371567143 LMissing descriptors for photos 1371567143 and 1371567226 LMissing descriptors for photos 1371567226 and 1371567228 LMissing descriptors for photos 1371567228 and 1371567298 LMissing descriptors for photos 1371567298 and 1371567333 LMissing descriptors for photos 1371567333 and 1371567510 LMissing descriptors for photos 1371567510 and 1371567565 LMissing descriptors for photos 1371567565 and 1371567626 LMissing descriptors for photos 1371567626 and 1371567629 LMissing descriptors for photos 1371567629 and 1371567729 LMissing descriptors for photos 1371567729 and 1371567801 LMissing descriptors for photos 1371567801 and 1371567919 LMissing descriptors for photos 1371567919 and 1371567972 LMissing descriptors for photos 1371567972 and 1371568127 LMissing descriptors for photos 1371568127 and 1371568190 LMissing descriptors for photos 1371568190 and 1371568198 LMissing descriptors for photos 1371568198 and 1371568311 LMissing descriptors for photos 1371568311 and 1371568376 LMissing descriptors for photos 1371568376 and 1371568478 LMissing descriptors for photos 1371568478 and 1371568532 LMissing descriptors for photos 1371568532 and 1371568549 LMissing descriptors for photos 1371568549 and 1371568707 LMissing descriptors for photos 1371568707 and 1371568708 LMissing descriptors for photos 1371568708 and 1371568850 LMissing descriptors for photos 1371568850 and 1371568891 LMissing descriptors for photos 1371568891 and 1371568899 LMissing descriptors for photos 1371568899 and 1371568948 LMissing descriptors for photos 1371568948 and 1371569109 LMissing descriptors for photos 1371569109 and 1371569215 LMissing descriptors for photos 1371569215 and 1371569259 LMissing descriptors for photos 1371569259 and 1371569261 LMissing descriptors for photos 1371569261 and 1371569330 LMissing descriptors for photos 1371569330 and 1371569575 LMissing descriptors for photos 1371569575 and 1371569646 LMissing descriptors for photos 1371569646 and 1371569647 LMissing descriptors for photos 1371569647 and 1371569796 LMissing descriptors for photos 1371569796 and 1371569852 LMissing descriptors for photos 1371569852 and 1371570026 LMissing descriptors for photos 1371570026 and 1371570088 LMissing descriptors for photos 1371570088 and 1371570216 LMissing descriptors for photos 1371570216 and 1371570277 LMissing descriptors for photos 1371570277 and 1371570369 LMissing descriptors for photos 1371570369 and 1371570443 LMissing descriptors for photos 1371570443 and 1371570506 LMissing descriptors for photos 1371570506 and 1371570699 LMissing descriptors for photos 1371570699 and 1371570823 LMissing descriptors for photos 1371570823 and 1371570956 LMissing descriptors for photos 1371570956 and 1371571046 LMissing descriptors for photos 1371571046 and 1371571203 LMissing descriptors for photos 1371571203 and 1371571253 LMissing descriptors for photos 1371571253 and 1371571333 LMissing descriptors for photos 1371571333 and 1371571422 LMissing descriptors for photos 1371571422 and 1371571620 LMissing descriptors for photos 1371571620 and 1371571670 LMissing descriptors for photos 1371571670 and 1371571736 LMissing descriptors for photos 1371571736 and 1371571857 LMissing descriptors for photos 1371571857 and 1371572002 LMissing descriptors for photos 1371572002 and 1371572061 LMissing descriptors for photos 1371572061 and 1371572062 LMissing descriptors for photos 1371572062 and 1371572160 LMissing descriptors for photos 1371572160 and 1371572426 LMissing descriptors for photos 1371572426 and 1371572512 LMissing descriptors for photos 1371572512 and 1371572522 LMissing descriptors for photos 1371572522 and 1371572598 LMissing descriptors for photos 1371572598 and 1371572657 LMissing descriptors for photos 1371572657 and 1371572984 LMissing descriptors for photos 1371572984 and 1371572994 LMissing descriptors for photos 1371572994 and 1371573062 LMissing descriptors for photos 1371573062 and 1371573092 LMissing descriptors for photos 1371573092 and 1371573427 LMissing descriptors for photos 1371573427 and 1371573436 LMissing descriptors for photos 1371573436 and 1371573482 LMissing descriptors for photos 1371573482 and 1371573514 LMissing descriptors for photos 1371573514 and 1371573557 LMissing descriptors for photos 1371573557 and 1371574037 LMissing descriptors for photos 1371574037 and 1371574104 LMissing descriptors for photos 1371574104 and 1371574172 LMissing descriptors for photos 1371574172 and 1371574457 LMissing descriptors for photos 1371574457 and 1371574531 LMissing descriptors for photos 1371574531 and 1371580330 LMissing descriptors for photos 1371580330 and 1371581809 LMissing descriptors for photos 1371581809 and 1371582193 LMissing descriptors for photos 1371582193 and 1371582243 LMissing descriptors for photos 1371582243 and 1371582854 LMissing descriptors for photos 1371582854 and 1371582876 LMissing descriptors for photos 1371582876 and 1371586404 LMissing descriptors for photos 1371586404 and 1371586605 LMissing descriptors for photos 1371586605 and 1371586640 LMissing descriptors for photos 1371586640 and 1371586818 LMissing descriptors for photos 1371586818 and 1371587156 LMissing descriptors for photos 1371587156 and 1371587411 LMissing descriptors for photos 1371587411 and 1371589356 LMissing descriptors for photos 1371589356 and 1371589729 LMissing descriptors for photos 1371589729 and 1371590113 LMissing descriptors for photos 1371590113 and 1371590130 LMissing descriptors for photos 1371590130 and 1371591039 LMissing descriptors for photos 1371591039 and 1371591259 LMissing descriptors for photos 1371591259 and 1371591984 LMissing descriptors for photos 1371591984 and 1371592126 LMissing descriptors for photos 1371592126 and 1371592250 LMissing descriptors for photos 1371592250 and 1371592265 LMissing descriptors for photos 1371592265 and 1371592270 LMissing descriptors for photos 1371592270 and 1371592282 LMissing descriptors for photos 1371592282 and 1371592596 LMissing descriptors for photos 1371592596 and 1371592684 LMissing descriptors for photos 1371592684 and 1371592703 LMissing descriptors for photos 1371592703 and 1371592720 LMissing descriptors for photos 1371592720 and 1371592745 LMissing descriptors for photos 1371592745 and 1371592752 LMissing descriptors for photos 1371592752 and 1371592998 LMissing descriptors for photos 1371592998 and 1371593057 LMissing descriptors for photos 1371593057 and 1371593063 LMissing descriptors for photos 1371593063 and 1371593117 LMissing descriptors for photos 1371593117 and 1371593124 LMissing descriptors for photos 1371593124 and 1371593129 LMissing descriptors for photos 1371593129 and 1371593134 LMissing descriptors for photos 1371593134 and 1371593142 LMissing descriptors for photos 1371593142 and 1371593210 LMissing descriptors for photos 1371593210 and 1371593282 LMissing descriptors for photos 1371593282 and 1371593311 LMissing descriptors for photos 1371593311 and 1371593316 LMissing descriptors for photos 1371593316 and 1371596137 LMissing descriptors for photos 1371596137 and 1371596142 LMissing descriptors for photos 1371596142 and 1371596150 LMissing descriptors for photos 1371596150 and 1371596155 LMissing descriptors for photos 1371596155 and 1371596163 LMissing descriptors for photos 1371596163 and 1371596172 LLMissing descriptors for photos 1371599207 and 1371599301 LLMissing descriptors for photos 1371602299 and 1371602392 LLLL0:02:00.109261|OFF:LLLLLLLLLLL0:00:59.791506|ON:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL0:07:02.273634|OFF:LLL0:00:57.674446|ON:LL0:00:00|OFF:L0:00:00|ON:LLLLLLLLLLLLLLLLLLLLLLLLLL0:03:59.857633|OFF:L0:00:00|ON:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL0:07:00.319858|OFF:L0:00:00|ON:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLMissing descriptors for photos 1380087790 and 1380087792 LMissing descriptors for photos 1380087792 and 1380087838 LMissing descriptors for photos 1380087838 and 1380087839 LMissing descriptors for photos 1380087839 and 1380087840 LMissing descriptors for photos 1380087840 and 1380087841 LMissing descriptors for photos 1380087841 and 1380087842 LMissing descriptors for photos 1380087842 and 1380087843 LMissing descriptors for photos 1380087843 and 1380088030 LMissing descriptors for photos 1380088030 and 1380088032 LMissing descriptors for photos 1380088032 and 1380088034 LMissing descriptors for photos 1380088034 and 1380088036 LMissing descriptors for photos 1380088036 and 1380088038 LMissing descriptors for photos 1380088038 and 1380088040 LMissing descriptors for photos 1380088040 and 1380088153 LMissing descriptors for photos 1380088153 and 1380088155 LMissing descriptors for photos 1380088155 and 1380088158 LMissing descriptors for photos 1380088158 and 1380088161 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL0:16:59.867414|OFF:L0:00:00|ON:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL0:06:00.086874|OFF:L0:00:00|ON:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL0:07:00.008069|OFF:L0:00:00|ON:LLLLLLLLLLLLLLLLL0:02:01.649881|OFF:LLLLL0:00:58.759758|ON:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL3:08:59.672834|OFF:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL0:12:00.445501|ON:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL0:08:59.954231|OFF:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL0:01:59.383478|ON:LLLL0:01:00.604535|OFF:L0:00:00|ON:LLL0:00:00|OFF:LLLLLLLLLLL0:00:59.886715|ON:LLL0:00:59.812041|OFF:L0:00:00|ON:LLL0:00:00|OFF:LLLLL0:00:59.907965|ON:Missing descriptors for photos 1380605273 and 1380605678 LLLLLL0:00:59.850411|OFF:L0:00:00|ON:LLLL0:00:00|OFF:L0:00:00|ON:LLLLLLLLLMissing descriptors for photos 1380606159 and 1380606163 LLLMissing descriptors for photos 1380606176 and 1380606182 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLMissing descriptors for photos 1380607307 and 1380607462 LMissing descriptors for photos 1380607462 and 1380607573 LMissing descriptors for photos 1380607573 and 1380607684 LMissing descriptors for photos 1380607684 and 1380607740 LMissing descriptors for photos 1380607740 and 1380607742 LLLL0:11:00.365838|OFF:L0:00:00|ON:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL0:09:59.389966|OFF:L0:00:00|ON:LLLLL0:01:00.622115|OFF:L0:00:00|ON:LLLL0:00:59.674078|OFF:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL 11072025 Removing 154 photos because of the 'same image' condition Total on : 16564.118672999997 list_time_on Total off : 1135.8493690000003 list_time_off dist_desc begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 875 time used for this insertion : 0.27902936935424805 photos_removed : len 154 elapsed_time : remove_photo_duplicate 0.3569817543029785 To do, maybe not at the correct place ! .L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L..L.L....L.L....L.L.L.L.L.L..L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.Lforce hashtag to JRM elapsed_time : CREATE_PORT_BATCH_BY_HOUR 0.030075550079345703 NUMBER BATCH : 8 list_ponderation used : [1e-05, 1e-05, 1e-05, 1e-05, 1e-05] , list_hashtag_class_create_as_list : ['jrm'] LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLresult_one_balle_Type_JRM:{'day': '11072025', 'map_nb_amount': {0: 12, 1: 61, 2: 2, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 839.9960470199585, 'nb_balles_papier': 0.0007700000000000014, 'begin_time_port': 'image_11072025_10_00_02_565621_0001.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0007700000000000014 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLresult_one_balle_Type_JRM:{'day': '11072025', 'map_nb_amount': {0: 7, 1: 79, 2: 0, 3: 2, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 840.4797701835632, 'nb_balles_papier': 0.0008800000000000017, 'begin_time_port': 'image_11072025_10_15_02_272101_0000.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0008800000000000017 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLresult_one_balle_Type_JRM:{'day': '11072025', 'map_nb_amount': {0: 12, 1: 62, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 840.1088969707489, 'nb_balles_papier': 0.0007400000000000013, 'begin_time_port': 'image_11072025_10_30_02_652741_0000.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0007400000000000013 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLresult_one_balle_Type_JRM:{'day': '11072025', 'map_nb_amount': {0: 13, 1: 69, 2: 0, 3: 1, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 839.6772019863129, 'nb_balles_papier': 0.0008300000000000015, 'begin_time_port': 'image_11072025_10_45_02_546247_0000.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0008300000000000015 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLresult_one_balle_Type_JRM:{'day': '11072025', 'map_nb_amount': {0: 6, 1: 18, 2: 1, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 239.7317841053009, 'nb_balles_papier': 0.00025, 'begin_time_port': 'image_11072025_14_00_02_905005_0000.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.00025 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLresult_one_balle_Type_JRM:{'day': '11072025', 'map_nb_amount': {0: 10, 1: 42, 2: 1, 3: 1, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 719.942244052887, 'nb_balles_papier': 0.0005400000000000008, 'begin_time_port': 'image_11072025_14_16_03_082290_0003.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0005400000000000008 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLresult_one_balle_Type_JRM:{'day': '11072025', 'map_nb_amount': {0: 1, 1: 62, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 839.9995510578156, 'nb_balles_papier': 0.000630000000000001, 'begin_time_port': 'image_11072025_14_30_02_723290_0000.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.000630000000000001 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLresult_one_balle_Type_JRM:{'day': '11072025', 'map_nb_amount': {0: 8, 1: 45, 2: 4, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 540.1672921180725, 'nb_balles_papier': 0.0005700000000000008, 'begin_time_port': 'image_11072025_14_45_02_692293_0000.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0005700000000000008 We have rejected 0 photos because of the batch_size condition ! NUMBER BATCH list_of_portfolios_to_create : 8 list_same_port_ids : [] list_same_port_ids : [] list_same_port_ids : [] list_same_port_ids : [] list_same_port_ids : [] list_same_port_ids : [] list_same_port_ids : [] list_same_port_ids : [] # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=30446324 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=30446326 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=30446327 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=30446335 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=30446347 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=30446349 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=30446350 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=30446352 AND mptpi.`type`=3726 To do elapsed_time : count_nb_balles_and_create_portfolio 32.90629482269287 # DISPLAY ALL COLLECTED DATA : {'11072025': {'nb_upload': 875, 'nb_taggue_class': 856, 'nb_taggue_densite': 856, 'nb_descriptors': 675}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1380610610, 1380610606, 1380610604, 1380610601, 1380610599, 1380610595, 1380610583, 1380610579, 1380610577, 1380610496, 1380610492, 1380610471, 1380610425, 1380610386, 1380610349, 1380610214, 1380610162, 1380610099, 1380610048, 1380609979, 1380609923, 1380609536, 1380609490, 1380609474, 1380609436, 1380609371, 1380609335, 1380609201, 1380609191, 1380609185, 1380609173, 1380609135, 1380609095, 1380608996, 1380608991, 1380608986, 1380608979, 1380608974, 1380608970, 1380608937, 1380608930, 1380608912, 1380608905, 1380608883, 1380608880, 1380608816, 1380608811, 1380608804, 1380608799, 1380608796, 1380608793, 1380608681, 1380608655, 1380608651, 1380608647, 1380608614, 1380608597, 1380608403, 1380608373, 1380608339, 1380608324, 1380608323, 1380608321, 1380608273, 1380608270, 1380608269, 1380608268, 1380608267, 1380608261, 1380608242, 1380608241, 1380608239, 1380608235, 1380608232, 1380608225, 1380608156, 1380608152, 1380608150, 1380608146, 1380608141, 1380608139, 1380608097, 1380608095, 1380608094, 1380608093, 1380608092, 1380608089, 1380608064, 1380608063, 1380608054, 1380608052, 1380608049, 1380608041, 1380608036, 1380607987, 1380607935, 1380607931, 1380607928, 1380607918, 1380607743] Looping around the photos to save general results len do output : 1 /24834339Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610610', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610606', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610604', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610601', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610599', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610595', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610583', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610579', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610577', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610496', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610492', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610471', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610425', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610386', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610349', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610214', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610162', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610099', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380610048', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609979', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609923', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609536', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609490', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609474', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609436', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609371', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609335', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609201', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609191', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609185', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609173', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609135', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380609095', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608996', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608991', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608986', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608979', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608974', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608970', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608937', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608930', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608912', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608905', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608883', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608880', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608816', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608811', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608804', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608799', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608796', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608793', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608681', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608655', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608651', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608647', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608614', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608597', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608403', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608373', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608339', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608324', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608323', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608321', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608273', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608270', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608269', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608268', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608267', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608261', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608242', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608241', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608239', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608235', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608232', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608225', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608156', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608152', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608150', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608146', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608141', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608139', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608097', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608095', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608094', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608093', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608092', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608089', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608064', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608063', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608054', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608052', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608049', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608041', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380608036', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380607987', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380607935', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380607931', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380607928', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380607918', None, None, None, None, None, '4346817') ('4189', None, None, None, None, None, None, None, '4346817') ('4189', '24834339', '1380607743', None, None, None, None, None, '4346817') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 101 time used for this insertion : 0.0490720272064209 save_final save missing photos in datou_result : time spend for datou_step_exec : 91.01836013793945 time spend to save output : 0.04971718788146973 total time spend for step 1 : 91.06807732582092 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 2 len(input) + len(total_photo_id_missing) : 1 set_done_treatment 3.18user 0.81system 1:35.68elapsed 4%CPU (0avgtext+0avgdata 117800maxresident)k 0inputs+5600outputs (15major+54856minor)pagefaults 0swaps