python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 20 -a ' -a 4189' -s datou_current_4189 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2240724 load datou : 4189 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4189, datou_cur_ids : ['2539399'] with mtr_portfolio_ids : ['20108238'] and first list_photo_ids : [] new path : /proc/2240724/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : split_time_score over limit max, limiting to limit_max 100 list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB time to download the photos : 0.16269969940185547 About to test input to load Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 1 step1:split_time_score Mon Feb 3 10:55:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec begin split time score 2022-04-13 10:29:59 0 TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3379, 'mtr_user_id': 31, 'name': 'learn_classif_flux_maj_generique_effnet_v2_s_02062022', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'aluminium,ela,film_pedb,flux_dev,jrm,pcm,pcnc,pehd_pp,pet_clair,refus,tapis_vide', 'svm_portfolios_learning': '5515864,5515840,5515844,5515850,6244400,6237996,6237998,5515847,5515841,5515868,5515866', 'photo_hashtag_type': 4374, 'photo_desc_type': 5680, 'type_classification': 'tf_classification2', 'hashtag_id_list': '493546845,492741797,2107760237,2107760238,495916461,560181804,1284539308,2107760239,2107755846,538914404,2107748999'}] thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 358), ('14', 356)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {1: 714} 31012025 20108238 Nombre de photos uploadées : 714 / 23040 (3%) 31012025 20108238 Nombre de photos taguées (types de déchets): 714 / 714 (100%) 31012025 20108238 Nombre de photos taguées (volume) : 714 / 714 (100%) elapsed_time : load_data_split_time_score 2.6226043701171875e-06 elapsed_time : order_list_meta_photo_and_scores 0.0002872943878173828 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL elapsed_time : fill_and_build_computed_from_old_data 0.06685638427734375 elapsed_time : insert_dashboard_record_day_entry 0.02397942543029785 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL Creating list_photo_by_hashtags elapsed_time : list_photo_by_hashtags 0.05212068557739258 Creating list_photo_total elapsed_time : select_descriptors 31.549270391464233 31012025 20108238 Nombre de photos avec descriptors (type 5680) : 350 / 350 (100%) ERROR : Hum hum, what can we do for different size of descriptors (ignore the difference ) : 0 vs 1280 photo_id : 1333268195 photo_id_prec : 0 0:00:00|ON:LLMissing descriptors for photos 1333269106 and 1333269120 LLL0:06:00.859119|OFF:LLLLLL0:00:59.840354|ON:LL0:00:00|OFF:LLLLL0:00:00|ON:LL0:00:59.948829|OFF:LLLLLLL0:00:00|ON:LL0:00:59.863465|OFF:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL0:04:00.409722|ON:LMissing descriptors for photos 1333270282 and 1333270289 LMissing descriptors for photos 1333270289 and 1333270294 LMissing descriptors for photos 1333270294 and 1333270298 LMissing descriptors for photos 1333270298 and 1333270330 LMissing descriptors for photos 1333270330 and 1333270367 LLL0:05:59.917688|OFF:LL0:00:00|ON:LL0:05:59.374166|OFF:0:00:00|ON:LL0:06:00.711129|OFF:LLLLLLLLLLLLLLLLLLLL0:01:59.453031|ON:LL0:07:00.229590|OFF:0:00:00|ON:LL0:14:00.144221|OFF:0:00:00|ON:LL0:03:00.099537|OFF:0:00:00|ON:LMissing descriptors for photos 1333325825 and 1333325850 LMissing descriptors for photos 1333325850 and 1333325882 LMissing descriptors for photos 1333325882 and 1333325896 LMissing descriptors for photos 1333325896 and 1333325934 LMissing descriptors for photos 1333325934 and 1333325951 LMissing descriptors for photos 1333325951 and 1333325953 LMissing descriptors for photos 1333325953 and 1333325959 LMissing descriptors for photos 1333325959 and 1333325964 LMissing descriptors for photos 1333325964 and 1333325970 LMissing descriptors for photos 1333325970 and 1333326124 LMissing descriptors for photos 1333326124 and 1333326537 LMissing descriptors for photos 1333326537 and 1333326564 LMissing descriptors for photos 1333326564 and 1333326593 LMissing descriptors for photos 1333326593 and 1333326600 LMissing descriptors for photos 1333326600 and 1333326662 LMissing descriptors for photos 1333326662 and 1333326665 LMissing descriptors for photos 1333326665 and 1333326883 LMissing descriptors for photos 1333326883 and 1333327010 LMissing descriptors for photos 1333327010 and 1333327144 LMissing descriptors for photos 1333327144 and 1333327287 LMissing descriptors for photos 1333327287 and 1333327496 LMissing descriptors for photos 1333327496 and 1333327667 LMissing descriptors for photos 1333327667 and 1333328298 LMissing descriptors for photos 1333328298 and 1333328312 LMissing descriptors for photos 1333328312 and 1333328384 LMissing descriptors for photos 1333328384 and 1333328393 LMissing descriptors for photos 1333328393 and 1333328416 LMissing descriptors for photos 1333328416 and 1333328421 LMissing descriptors for photos 1333328421 and 1333328526 LMissing descriptors for photos 1333328526 and 1333328588 LMissing descriptors for photos 1333328588 and 1333328602 LMissing descriptors for photos 1333328602 and 1333328623 LMissing descriptors for photos 1333328623 and 1333328659 LMissing descriptors for photos 1333328659 and 1333328666 LMissing descriptors for photos 1333328666 and 1333328781 LMissing descriptors for photos 1333328781 and 1333328808 LMissing descriptors for photos 1333328808 and 1333328982 LMissing descriptors for photos 1333328982 and 1333329094 LMissing descriptors for photos 1333329094 and 1333329213 LLMissing descriptors for photos 1333329281 and 1333329286 LMissing descriptors for photos 1333329286 and 1333329293 LMissing descriptors for photos 1333329293 and 1333329299 LMissing descriptors for photos 1333329299 and 1333329305 LMissing descriptors for photos 1333329305 and 1333329310 LMissing descriptors for photos 1333329310 and 1333329441 LMissing descriptors for photos 1333329441 and 1333329449 LMissing descriptors for photos 1333329449 and 1333329495 LMissing descriptors for photos 1333329495 and 1333329503 LMissing descriptors for photos 1333329503 and 1333329517 LMissing descriptors for photos 1333329517 and 1333329544 LMissing descriptors for photos 1333329544 and 1333329651 LMissing descriptors for photos 1333329651 and 1333329657 LMissing descriptors for photos 1333329657 and 1333329664 LMissing descriptors for photos 1333329664 and 1333329674 LMissing descriptors for photos 1333329674 and 1333329678 LMissing descriptors for photos 1333329678 and 1333329682 LMissing descriptors for photos 1333329682 and 1333329714 LMissing descriptors for photos 1333329714 and 1333329721 LMissing descriptors for photos 1333329721 and 1333329728 LMissing descriptors for photos 1333329728 and 1333329731 LMissing descriptors for photos 1333329731 and 1333329734 LMissing descriptors for photos 1333329734 and 1333329736 LMissing descriptors for photos 1333329736 and 1333329762 LMissing descriptors for photos 1333329762 and 1333329788 LMissing descriptors for photos 1333329788 and 1333329926 LMissing descriptors for photos 1333329926 and 1333330053 LMissing descriptors for photos 1333330053 and 1333330075 LMissing descriptors for photos 1333330075 and 1333330078 LMissing descriptors for photos 1333330078 and 1333330291 LMissing descriptors for photos 1333330291 and 1333330299 LMissing descriptors for photos 1333330299 and 1333330307 LMissing descriptors for photos 1333330307 and 1333330398 LMissing descriptors for photos 1333330398 and 1333330452 LMissing descriptors for photos 1333330452 and 1333330454 LMissing descriptors for photos 1333330454 and 1333330481 LMissing descriptors for photos 1333330481 and 1333330492 LMissing descriptors for photos 1333330492 and 1333330497 LMissing descriptors for photos 1333330497 and 1333330500 LMissing descriptors for photos 1333330500 and 1333330554 LMissing descriptors for photos 1333330554 and 1333330557 LMissing descriptors for photos 1333330557 and 1333330663 LMissing descriptors for photos 1333330663 and 1333330669 LMissing descriptors for photos 1333330669 and 1333330742 LMissing descriptors for photos 1333330742 and 1333330787 LMissing descriptors for photos 1333330787 and 1333330802 LMissing descriptors for photos 1333330802 and 1333330806 LMissing descriptors for photos 1333330806 and 1333331141 LMissing descriptors for photos 1333331141 and 1333331178 LMissing descriptors for photos 1333331178 and 1333331188 LMissing descriptors for photos 1333331188 and 1333331191 LMissing descriptors for photos 1333331191 and 1333331192 LMissing descriptors for photos 1333331192 and 1333331194 LMissing descriptors for photos 1333331194 and 1333331228 LMissing descriptors for photos 1333331228 and 1333331238 LMissing descriptors for photos 1333331238 and 1333331243 LMissing descriptors for photos 1333331243 and 1333331275 LMissing descriptors for photos 1333331275 and 1333331412 LMissing descriptors for photos 1333331412 and 1333331531 LMissing descriptors for photos 1333331531 and 1333331650 LMissing descriptors for photos 1333331650 and 1333331707 LMissing descriptors for photos 1333331707 and 1333331713 LMissing descriptors for photos 1333331713 and 1333331722 LMissing descriptors for photos 1333331722 and 1333331794 LMissing descriptors for photos 1333331794 and 1333331796 LMissing descriptors for photos 1333331796 and 1333331821 LMissing descriptors for photos 1333331821 and 1333331828 LMissing descriptors for photos 1333331828 and 1333331839 LMissing descriptors for photos 1333331839 and 1333331845 LMissing descriptors for photos 1333331845 and 1333331850 LMissing descriptors for photos 1333331850 and 1333331866 LMissing descriptors for photos 1333331866 and 1333331876 LMissing descriptors for photos 1333331876 and 1333331883 LMissing descriptors for photos 1333331883 and 1333331894 LMissing descriptors for photos 1333331894 and 1333331899 LMissing descriptors for photos 1333331899 and 1333331904 LMissing descriptors for photos 1333331904 and 1333331922 LMissing descriptors for photos 1333331922 and 1333331930 LMissing descriptors for photos 1333331930 and 1333331936 LMissing descriptors for photos 1333331936 and 1333331941 LMissing descriptors for photos 1333331941 and 1333331958 LMissing descriptors for photos 1333331958 and 1333331962 LMissing descriptors for photos 1333331962 and 1333332026 LMissing descriptors for photos 1333332026 and 1333332037 LMissing descriptors for photos 1333332037 and 1333332046 LMissing descriptors for photos 1333332046 and 1333332174 LMissing descriptors for photos 1333332174 and 1333332244 LMissing descriptors for photos 1333332244 and 1333332247 LMissing descriptors for photos 1333332247 and 1333332536 LMissing descriptors for photos 1333332536 and 1333332656 LMissing descriptors for photos 1333332656 and 1333332671 LMissing descriptors for photos 1333332671 and 1333332676 LMissing descriptors for photos 1333332676 and 1333332681 LMissing descriptors for photos 1333332681 and 1333332732 LMissing descriptors for photos 1333332732 and 1333332773 LMissing descriptors for photos 1333332773 and 1333332780 LMissing descriptors for photos 1333332780 and 1333332787 LMissing descriptors for photos 1333332787 and 1333332792 LMissing descriptors for photos 1333332792 and 1333332798 LMissing descriptors for photos 1333332798 and 1333332802 LMissing descriptors for photos 1333332802 and 1333332832 LMissing descriptors for photos 1333332832 and 1333332839 LMissing descriptors for photos 1333332839 and 1333332844 LMissing descriptors for photos 1333332844 and 1333332875 LMissing descriptors for photos 1333332875 and 1333332883 LMissing descriptors for photos 1333332883 and 1333332886 LMissing descriptors for photos 1333332886 and 1333332957 LMissing descriptors for photos 1333332957 and 1333332967 LMissing descriptors for photos 1333332967 and 1333332972 LMissing descriptors for photos 1333332972 and 1333332977 LMissing descriptors for photos 1333332977 and 1333332980 LMissing descriptors for photos 1333332980 and 1333332983 LMissing descriptors for photos 1333332983 and 1333333012 LMissing descriptors for photos 1333333012 and 1333333018 LMissing descriptors for photos 1333333018 and 1333333130 LMissing descriptors for photos 1333333130 and 1333333142 LMissing descriptors for photos 1333333142 and 1333333145 LMissing descriptors for photos 1333333145 and 1333333149 LMissing descriptors for photos 1333333149 and 1333333206 LMissing descriptors for photos 1333333206 and 1333333244 LMissing descriptors for photos 1333333244 and 1333333250 LMissing descriptors for photos 1333333250 and 1333333253 LMissing descriptors for photos 1333333253 and 1333333254 LMissing descriptors for photos 1333333254 and 1333333255 LMissing descriptors for photos 1333333255 and 1333333281 LMissing descriptors for photos 1333333281 and 1333333296 LMissing descriptors for photos 1333333296 and 1333333301 LMissing descriptors for photos 1333333301 and 1333333309 LMissing descriptors for photos 1333333309 and 1333333320 LMissing descriptors for photos 1333333320 and 1333333325 LMissing descriptors for photos 1333333325 and 1333333350 LMissing descriptors for photos 1333333350 and 1333333359 LMissing descriptors for photos 1333333359 and 1333333370 LMissing descriptors for photos 1333333370 and 1333333374 LLLMissing descriptors for photos 1333333438 and 1333333617 LMissing descriptors for photos 1333333617 and 1333333620 LMissing descriptors for photos 1333333620 and 1333333623 LMissing descriptors for photos 1333333623 and 1333333625 LMissing descriptors for photos 1333333625 and 1333333629 LMissing descriptors for photos 1333333629 and 1333333633 LMissing descriptors for photos 1333333633 and 1333333713 LMissing descriptors for photos 1333333713 and 1333333717 LMissing descriptors for photos 1333333717 and 1333333719 LMissing descriptors for photos 1333333719 and 1333333744 LMissing descriptors for photos 1333333744 and 1333333998 LMissing descriptors for photos 1333333998 and 1333334004 LMissing descriptors for photos 1333334004 and 1333334192 LMissing descriptors for photos 1333334192 and 1333334215 LMissing descriptors for photos 1333334215 and 1333334249 LMissing descriptors for photos 1333334249 and 1333334263 LMissing descriptors for photos 1333334263 and 1333334268 LMissing descriptors for photos 1333334268 and 1333334274 LMissing descriptors for photos 1333334274 and 1333334302 LMissing descriptors for photos 1333334302 and 1333334306 LMissing descriptors for photos 1333334306 and 1333334312 LMissing descriptors for photos 1333334312 and 1333334315 LMissing descriptors for photos 1333334315 and 1333334318 LMissing descriptors for photos 1333334318 and 1333334323 LMissing descriptors for photos 1333334323 and 1333334365 LMissing descriptors for photos 1333334365 and 1333334369 LMissing descriptors for photos 1333334369 and 1333334373 LMissing descriptors for photos 1333334373 and 1333334386 LMissing descriptors for photos 1333334386 and 1333334399 LMissing descriptors for photos 1333334399 and 1333334430 LMissing descriptors for photos 1333334430 and 1333334666 LMissing descriptors for photos 1333334666 and 1333334735 LMissing descriptors for photos 1333334735 and 1333334857 LMissing descriptors for photos 1333334857 and 1333334858 LMissing descriptors for photos 1333334858 and 1333334887 LMissing descriptors for photos 1333334887 and 1333334932 LMissing descriptors for photos 1333334932 and 1333335051 LMissing descriptors for photos 1333335051 and 1333335055 LMissing descriptors for photos 1333335055 and 1333335059 LMissing descriptors for photos 1333335059 and 1333335132 LMissing descriptors for photos 1333335132 and 1333335145 LMissing descriptors for photos 1333335145 and 1333335160 LLLMissing descriptors for photos 1333335226 and 1333335232 LMissing descriptors for photos 1333335232 and 1333335235 LMissing descriptors for photos 1333335235 and 1333335237 LMissing descriptors for photos 1333335237 and 1333335250 LMissing descriptors for photos 1333335250 and 1333335288 LMissing descriptors for photos 1333335288 and 1333335294 LMissing descriptors for photos 1333335294 and 1333335345 LMissing descriptors for photos 1333335345 and 1333335349 LMissing descriptors for photos 1333335349 and 1333335353 LMissing descriptors for photos 1333335353 and 1333335357 LMissing descriptors for photos 1333335357 and 1333335501 LMissing descriptors for photos 1333335501 and 1333335507 LMissing descriptors for photos 1333335507 and 1333335520 LMissing descriptors for photos 1333335520 and 1333335556 LMissing descriptors for photos 1333335556 and 1333335622 LMissing descriptors for photos 1333335622 and 1333335648 LMissing descriptors for photos 1333335648 and 1333335721 LMissing descriptors for photos 1333335721 and 1333335725 LMissing descriptors for photos 1333335725 and 1333335728 LMissing descriptors for photos 1333335728 and 1333335733 LMissing descriptors for photos 1333335733 and 1333335736 LMissing descriptors for photos 1333335736 and 1333335741 LMissing descriptors for photos 1333335741 and 1333335781 LMissing descriptors for photos 1333335781 and 1333335787 LMissing descriptors for photos 1333335787 and 1333335791 LMissing descriptors for photos 1333335791 and 1333335796 LMissing descriptors for photos 1333335796 and 1333335800 LMissing descriptors for photos 1333335800 and 1333335816 LMissing descriptors for photos 1333335816 and 1333335861 LMissing descriptors for photos 1333335861 and 1333335863 LMissing descriptors for photos 1333335863 and 1333335899 LMissing descriptors for photos 1333335899 and 1333335914 LMissing descriptors for photos 1333335914 and 1333335918 LMissing descriptors for photos 1333335918 and 1333335952 LMissing descriptors for photos 1333335952 and 1333336108 LMissing descriptors for photos 1333336108 and 1333336110 LMissing descriptors for photos 1333336110 and 1333336112 LMissing descriptors for photos 1333336112 and 1333336116 LMissing descriptors for photos 1333336116 and 1333336117 LMissing descriptors for photos 1333336117 and 1333336123 LLLMissing descriptors for photos 1333336160 and 1333336162 LMissing descriptors for photos 1333336162 and 1333336166 LMissing descriptors for photos 1333336166 and 1333336170 LMissing descriptors for photos 1333336170 and 1333336177 LMissing descriptors for photos 1333336177 and 1333336482 LMissing descriptors for photos 1333336482 and 1333336524 LMissing descriptors for photos 1333336524 and 1333336557 LMissing descriptors for photos 1333336557 and 1333336564 LMissing descriptors for photos 1333336564 and 1333336566 LMissing descriptors for photos 1333336566 and 1333336586 LMissing descriptors for photos 1333336586 and 1333336726 LMissing descriptors for photos 1333336726 and 1333336729 LMissing descriptors for photos 1333336729 and 1333336735 LMissing descriptors for photos 1333336735 and 1333336739 LMissing descriptors for photos 1333336739 and 1333336743 LMissing descriptors for photos 1333336743 and 1333336795 LMissing descriptors for photos 1333336795 and 1333336862 LMissing descriptors for photos 1333336862 and 1333336864 LMissing descriptors for photos 1333336864 and 1333336868 LMissing descriptors for photos 1333336868 and 1333336871 LMissing descriptors for photos 1333336871 and 1333336877 LMissing descriptors for photos 1333336877 and 1333336884 LMissing descriptors for photos 1333336884 and 1333336965 LMissing descriptors for photos 1333336965 and 1333336968 LMissing descriptors for photos 1333336968 and 1333336971 LMissing descriptors for photos 1333336971 and 1333336987 LMissing descriptors for photos 1333336987 and 1333336989 LMissing descriptors for photos 1333336989 and 1333337037 LMissing descriptors for photos 1333337037 and 1333337043 LMissing descriptors for photos 1333337043 and 1333337047 LMissing descriptors for photos 1333337047 and 1333337048 LMissing descriptors for photos 1333337048 and 1333337050 LMissing descriptors for photos 1333337050 and 1333337055 LMissing descriptors for photos 1333337055 and 1333337243 LMissing descriptors for photos 1333337243 and 1333337283 LMissing descriptors for photos 1333337283 and 1333337301 LMissing descriptors for photos 1333337301 and 1333337341 LMissing descriptors for photos 1333337341 and 1333337343 LMissing descriptors for photos 1333337343 and 1333337345 LMissing descriptors for photos 1333337345 and 1333337373 LMissing descriptors for photos 1333337373 and 1333337518 LMissing descriptors for photos 1333337518 and 1333337889 LMissing descriptors for photos 1333337889 and 1333338225 LMissing descriptors for photos 1333338225 and 1333338328 LMissing descriptors for photos 1333338328 and 1333338334 LMissing descriptors for photos 1333338334 and 1333338373 LMissing descriptors for photos 1333338373 and 1333338377 LMissing descriptors for photos 1333338377 and 1333338378 LMissing descriptors for photos 1333338378 and 1333338380 LMissing descriptors for photos 1333338380 and 1333338546 LMissing descriptors for photos 1333338546 and 1333338677 LMissing descriptors for photos 1333338677 and 1333338730 LMissing descriptors for photos 1333338730 and 1333338735 LMissing descriptors for photos 1333338735 and 1333338739 LMissing descriptors for photos 1333338739 and 1333338741 LMissing descriptors for photos 1333338741 and 1333338743 LMissing descriptors for photos 1333338743 and 1333338747 LMissing descriptors for photos 1333338747 and 1333339118 LMissing descriptors for photos 1333339118 and 1333339146 LMissing descriptors for photos 1333339146 and 1333339172 LMissing descriptors for photos 1333339172 and 1333339183 LMissing descriptors for photos 1333339183 and 1333339305 LMissing descriptors for photos 1333339305 and 1333339871 LMissing descriptors for photos 1333339871 and 1333339918 LMissing descriptors for photos 1333339918 and 1333339919 LMissing descriptors for photos 1333339919 and 1333339920 LMissing descriptors for photos 1333339920 and 1333339926 LMissing descriptors for photos 1333339926 and 1333339965 LMissing descriptors for photos 1333339965 and 1333340001 LMissing descriptors for photos 1333340001 and 1333340004 LMissing descriptors for photos 1333340004 and 1333340017 LMissing descriptors for photos 1333340017 and 1333340021 LMissing descriptors for photos 1333340021 and 1333340028 LMissing descriptors for photos 1333340028 and 1333340047 LMissing descriptors for photos 1333340047 and 1333340749 LMissing descriptors for photos 1333340749 and 1333340791 LMissing descriptors for photos 1333340791 and 1333340808 LMissing descriptors for photos 1333340808 and 1333340815 LMissing descriptors for photos 1333340815 and 1333340825 LMissing descriptors for photos 1333340825 and 1333340833 LMissing descriptors for photos 1333340833 and 1333340861 LMissing descriptors for photos 1333340861 and 1333340873 LMissing descriptors for photos 1333340873 and 1333340944 LMissing descriptors for photos 1333340944 and 1333340975 LMissing descriptors for photos 1333340975 and 1333340995 LMissing descriptors for photos 1333340995 and 1333341039 LMissing descriptors for photos 1333341039 and 1333341061 LMissing descriptors for photos 1333341061 and 1333341062 LMissing descriptors for photos 1333341062 and 1333341064 LMissing descriptors for photos 1333341064 and 1333341069 LMissing descriptors for photos 1333341069 and 1333341077 LMissing descriptors for photos 1333341077 and 1334144284 L 31012025 Removing 56 photos because of the 'same image' condition Total on : 3001.147744 list_time_on Total off : 419.703107 list_time_off dist_desc begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 714 time used for this insertion : 0.23681092262268066 photos_removed : len 56 elapsed_time : remove_photo_duplicate 0.2932462692260742 To do, maybe not at the correct place ! ...............................L.....L.L..L.L.L.L..L.L.L.L..L.L.L.L.L..L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L..L.L.L.L.L.L............................L.L...................................L...................................L.L.L.L.L.L.L.L.L.L.L............................................L....................................................................................L...................L.............L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L..L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.Lforce hashtag to JRM elapsed_time : CREATE_PORT_BATCH_BY_HOUR 0.024964094161987305 NUMBER BATCH : 7 list_ponderation used : [1e-05, 1e-05, 1e-05, 1e-05, 1e-05] , list_hashtag_class_create_as_list : ['jrm'] result_one_balle_Type_JRM:{'day': '31012025', 'map_nb_amount': {0: 2, 1: 37, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 780.9214890003204, 'nb_balles_papier': 0.00039000000000000037, 'begin_time_port': 'image_31012025_10_00_02_134348m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.00039000000000000037 result_one_balle_Type_JRM:{'day': '31012025', 'map_nb_amount': {0: 5, 1: 81, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 839.7308139801025, 'nb_balles_papier': 0.0008700000000000016, 'begin_time_port': 'image_31012025_10_15_02_750949m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0008700000000000016 result_one_balle_Type_JRM:{'day': '31012025', 'map_nb_amount': {0: 7, 1: 70, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 840.2614378929138, 'nb_balles_papier': 0.0007800000000000014, 'begin_time_port': 'image_31012025_10_30_02_305385m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0007800000000000014 result_one_balle_Type_JRM:{'day': '31012025', 'map_nb_amount': {0: 9, 1: 75, 2: 0, 3: 1, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 840.1620619297028, 'nb_balles_papier': 0.0008600000000000016, 'begin_time_port': 'image_31012025_10_45_02_216856m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0008600000000000016 We have rejected 4 photos because of the batch_size condition ! NUMBER BATCH list_of_portfolios_to_create : 4 list_same_port_ids : [20109260] find same portfolio which already exist 20109260 , we will use it list_same_port_ids : [20110569] find same portfolio which already exist 20110569 , we will use it list_same_port_ids : [20110572] find same portfolio which already exist 20110572 , we will use it list_same_port_ids : [20116061] find same portfolio which already exist 20116061 , we will use it Qualite : 0.11227155161993274 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20109260 AND mptpi.`type`=3726 To do Qualite : 0.15618838470056065 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20110569 AND mptpi.`type`=3726 To do Qualite : 0.12951691009203764 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20110572 AND mptpi.`type`=3726 To do Qualite : 0.09763347828179277 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20116061 AND mptpi.`type`=3726 To do elapsed_time : count_nb_balles_and_create_portfolio 0.5116696357727051 # DISPLAY ALL COLLECTED DATA : {'31012025': {'nb_upload': 714, 'nb_taggue_class': 714, 'nb_taggue_densite': 714, 'nb_descriptors': 350}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1334144284, 1333341077, 1333341069, 1333341064, 1333341062, 1333341061, 1333341039, 1333340995, 1333340975, 1333340944, 1333340873, 1333340861, 1333340833, 1333340825, 1333340815, 1333340808, 1333340791, 1333340749, 1333340047, 1333340028, 1333340021, 1333340017, 1333340004, 1333340001, 1333339965, 1333339926, 1333339920, 1333339919, 1333339918, 1333339871, 1333339305, 1333339183, 1333339172, 1333339146, 1333339118, 1333338747, 1333338743, 1333338741, 1333338739, 1333338735, 1333338730, 1333338677, 1333338546, 1333338380, 1333338378, 1333338377, 1333338373, 1333338334, 1333338328, 1333338225, 1333337889, 1333337518, 1333337373, 1333337345, 1333337343, 1333337341, 1333337301, 1333337283, 1333337243, 1333337055, 1333337050, 1333337048, 1333337047, 1333337043, 1333337037, 1333336989, 1333336987, 1333336971, 1333336968, 1333336965, 1333336884, 1333336877, 1333336871, 1333336868, 1333336864, 1333336862, 1333336795, 1333336743, 1333336739, 1333336735, 1333336729, 1333336726, 1333336586, 1333336566, 1333336564, 1333336557, 1333336524, 1333336482, 1333336177, 1333336170, 1333336166, 1333336162, 1333336160, 1333336159, 1333336123, 1333336117, 1333336116, 1333336112, 1333336110, 1333336108] Looping around the photos to save general results len do output : 1 /20108238Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1334144284', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333341077', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333341069', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333341064', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333341062', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333341061', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333341039', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340995', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340975', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340944', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340873', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340861', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340833', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340825', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340815', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340808', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340791', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340749', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340047', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340028', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340021', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340017', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340004', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333340001', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333339965', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333339926', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333339920', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333339919', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333339918', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333339871', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333339305', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333339183', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333339172', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333339146', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333339118', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338747', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338743', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338741', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338739', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338735', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338730', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338677', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338546', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338380', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338378', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338377', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338373', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338334', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338328', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333338225', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337889', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337518', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337373', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337345', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337343', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337341', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337301', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337283', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337243', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337055', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337050', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337048', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337047', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337043', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333337037', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336989', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336987', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336971', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336968', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336965', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336884', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336877', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336871', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336868', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336864', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336862', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336795', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336743', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336739', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336735', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336729', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336726', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336586', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336566', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336564', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336557', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336524', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336482', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336177', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336170', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336166', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336162', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336160', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336159', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336123', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336117', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336116', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336112', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336110', None, None, None, None, None, '2539399') ('4189', None, None, None, None, None, None, None, '2539399') ('4189', '20108238', '1333336108', None, None, None, None, None, '2539399') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 101 time used for this insertion : 0.02855372428894043 save_final save missing photos in datou_result : time spend for datou_step_exec : 32.75303053855896 time spend to save output : 0.029584884643554688 total time spend for step 1 : 32.782615423202515 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 2 len(input) + len(total_photo_id_missing) : 1 set_done_treatment 2.19user 0.95system 0:36.68elapsed 8%CPU (0avgtext+0avgdata 105504maxresident)k 352inputs+2952outputs (1major+50274minor)pagefaults 0swaps