python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/datou.py -j batch_current -C 2539597' -s traitement_sts -M 0 -S 0 -U 100,80,95 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/home/admin/workarea/git/apy', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2346308 load datou : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 0 init_dummy_multi_datou is not linked in the step_by_step architecture ! WARNING : step 1294 init_dummy_multi_datou is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 list_input_json: {} Current got : datou_id : 4323, datou_cur_ids : ['2539597'] with mtr_portfolio_ids : ['20127079'] and first list_photo_ids : [] new path : /proc/2346308/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : split_time_score over limit max, limiting to limit_max 100 list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB time to download the photos : 0.32607269287109375 About to test input to load Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 1 step1:split_time_score Mon Feb 3 11:31:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec begin split time score 2022-04-13 10:29:59 0 TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3847, 'mtr_user_id': 31, 'name': 'learn_MM_generique_050224', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'aluminium,ela,emr,film_pedb,flux_dev,jrm,pcm,pcnc,pehd_pp,pet_clair,refus,tapis_vide', 'svm_portfolios_learning': '13096157,13096155,13096163,13096159,13301956,13095886,13096162,13096160,13358264,13096158,5515868,13276803', 'photo_hashtag_type': 4932, 'photo_desc_type': 6032, 'type_classification': 'tf_classification2', 'hashtag_id_list': '493546845,492741797,616987804,2107760237,2107760238,495916461,560181804,1284539308,2107760239,2107755846,538914404,2107748999'}] thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('00', 210),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {1: 210} 01022025 20127079 Nombre de photos uploadées : 210 / 23040 (0%) 01022025 20127079 Nombre de photos taguées (types de déchets): 210 / 210 (100%) 01022025 20127079 Nombre de photos taguées (volume) : 210 / 210 (100%) elapsed_time : load_data_split_time_score 2.384185791015625e-06 elapsed_time : order_list_meta_photo_and_scores 0.00010657310485839844 elapsed_time : fill_and_build_computed_from_old_data 0.011867761611938477 elapsed_time : insert_dashboard_record_day_entry 0.022356748580932617 Creating list_photo_total elapsed_time : select_descriptors 6.019174575805664 01022025 20127079 Nombre de photos avec descriptors (type 6032) : 63 / 63 (100%) Missing descriptors for photos 0 and 1333500995 0:00:00|ON:Missing descriptors for photos 1333500995 and 1333500982 Missing descriptors for photos 1333500982 and 1333500941 Missing descriptors for photos 1333500941 and 1333500936 Missing descriptors for photos 1333500936 and 1333500838 Missing descriptors for photos 1333500838 and 1333500816 Missing descriptors for photos 1333500816 and 1333501137 Missing descriptors for photos 1333501137 and 1333501135 Missing descriptors for photos 1333501135 and 1333501093 Missing descriptors for photos 1333501093 and 1333501087 Missing descriptors for photos 1333501087 and 1333501082 Missing descriptors for photos 1333501082 and 1333501075 Missing descriptors for photos 1333501075 and 1333501180 Missing descriptors for photos 1333501180 and 1333501175 Missing descriptors for photos 1333501175 and 1333501171 Missing descriptors for photos 1333501171 and 1333501166 Missing descriptors for photos 1333501166 and 1333501159 Missing descriptors for photos 1333501159 and 1333501156 Missing descriptors for photos 1333501156 and 1333502427 Missing descriptors for photos 1333502427 and 1333502423 Missing descriptors for photos 1333502423 and 1333502418 Missing descriptors for photos 1333502418 and 1333502416 Missing descriptors for photos 1333502416 and 1333502414 Missing descriptors for photos 1333502414 and 1333502412 Missing descriptors for photos 1333502412 and 1333502469 Missing descriptors for photos 1333502469 and 1333502466 Missing descriptors for photos 1333502466 and 1333502464 Missing descriptors for photos 1333502464 and 1333502462 Missing descriptors for photos 1333502462 and 1333502458 Missing descriptors for photos 1333502458 and 1333502451 Missing descriptors for photos 1333503209 and 1333503206 Missing descriptors for photos 1333503206 and 1333503204 Missing descriptors for photos 1333503204 and 1333503198 Missing descriptors for photos 1333503198 and 1333503169 Missing descriptors for photos 1333503169 and 1333504164 Missing descriptors for photos 1333504164 and 1333504159 Missing descriptors for photos 1333504159 and 1333504154 Missing descriptors for photos 1333504154 and 1333504095 Missing descriptors for photos 1333504095 and 1333503887 Missing descriptors for photos 1333503887 and 1333503685 Missing descriptors for photos 1333503685 and 1333504552 Missing descriptors for photos 1333504552 and 1333504549 Missing descriptors for photos 1333504549 and 1333504513 Missing descriptors for photos 1333504513 and 1333504508 Missing descriptors for photos 1333504508 and 1333504263 Missing descriptors for photos 1333504263 and 1333504228 Missing descriptors for photos 1333504228 and 1333504684 Missing descriptors for photos 1333504684 and 1333504663 Missing descriptors for photos 1333504663 and 1333504661 Missing descriptors for photos 1333504661 and 1333504660 Missing descriptors for photos 1333504660 and 1333504658 Missing descriptors for photos 1333504658 and 1333504652 Missing descriptors for photos 1333504652 and 1333505161 Missing descriptors for photos 1333505161 and 1333504899 Missing descriptors for photos 1333504899 and 1333504894 Missing descriptors for photos 1333504894 and 1333504891 Missing descriptors for photos 1333504891 and 1333504849 Missing descriptors for photos 1333504849 and 1333504835 Missing descriptors for photos 1333504835 and 1333505466 Missing descriptors for photos 1333505466 and 1333505460 Missing descriptors for photos 1333505460 and 1333505332 Missing descriptors for photos 1333505332 and 1333505329 Missing descriptors for photos 1333505329 and 1333505327 Missing descriptors for photos 1333505327 and 1333505283 Missing descriptors for photos 1333505283 and 1333505597 Missing descriptors for photos 1333505597 and 1333505594 Missing descriptors for photos 1333505594 and 1333505591 Missing descriptors for photos 1333505591 and 1333505590 Missing descriptors for photos 1333505590 and 1333505588 Missing descriptors for photos 1333505588 and 1333505637 Missing descriptors for photos 1333505637 and 1333505633 Missing descriptors for photos 1333505633 and 1333505585 Missing descriptors for photos 1333505585 and 1333505627 Missing descriptors for photos 1333505627 and 1333505622 Missing descriptors for photos 1333505622 and 1333505616 Missing descriptors for photos 1333505616 and 1333505612 Missing descriptors for photos 1333505612 and 1333505740 Missing descriptors for photos 1333505740 and 1333505724 Missing descriptors for photos 1333505724 and 1333505663 Missing descriptors for photos 1333505663 and 1333505662 Missing descriptors for photos 1333505662 and 1333505660 Missing descriptors for photos 1333505660 and 1333505657 Missing descriptors for photos 1333505657 and 1333506071 Missing descriptors for photos 1333506071 and 1333506044 Missing descriptors for photos 1333506044 and 1333506040 Missing descriptors for photos 1333506040 and 1333506032 Missing descriptors for photos 1333506032 and 1333506023 Missing descriptors for photos 1333506023 and 1333505875 Missing descriptors for photos 1333505875 and 1333506121 Missing descriptors for photos 1333506121 and 1333506119 Missing descriptors for photos 1333506119 and 1333506117 Missing descriptors for photos 1333506117 and 1333506116 Missing descriptors for photos 1333506116 and 1333506113 Missing descriptors for photos 1333506113 and 1333506111 Missing descriptors for photos 1333506111 and 1333506468 Missing descriptors for photos 1333506468 and 1333506296 Missing descriptors for photos 1333506296 and 1333506244 Missing descriptors for photos 1333506244 and 1333506242 Missing descriptors for photos 1333506242 and 1333506240 Missing descriptors for photos 1333506240 and 1333506207 Missing descriptors for photos 1333506207 and 1333506899 Missing descriptors for photos 1333506899 and 1333506566 Missing descriptors for photos 1333506566 and 1333506553 Missing descriptors for photos 1333506553 and 1333506531 Missing descriptors for photos 1333506531 and 1333506519 Missing descriptors for photos 1333506519 and 1333506515 Missing descriptors for photos 1333506515 and 1333506510 Missing descriptors for photos 1333506510 and 1333506888 Missing descriptors for photos 1333506888 and 1333506853 Missing descriptors for photos 1333506853 and 1333506831 Missing descriptors for photos 1333506831 and 1333506829 Missing descriptors for photos 1333506829 and 1333506761 0:21:29.999989|OFF:0:00:04.998985|ON:0:01:49.001019|OFF:0:00:05.999998|ON:0:01:54.000002|OFF:0:00:04.998990|ON:0:00:49.001007|OFF:0:00:06.000001|ON:0:00:11.998991|OFF:0:00:30|ON:0:00:12.001005|OFF:0:00:12.000001|ON:0:00:48.000005|OFF:0:00:10.998986|ON:Missing descriptors for photos 1333510216 and 1333510212 Missing descriptors for photos 1333510212 and 1333510211 Missing descriptors for photos 1333510211 and 1333510265 Missing descriptors for photos 1333510265 and 1333510263 Missing descriptors for photos 1333510263 and 1333510260 Missing descriptors for photos 1333510260 and 1333510256 Missing descriptors for photos 1333510256 and 1333510252 Missing descriptors for photos 1333510252 and 1333510247 Missing descriptors for photos 1333510247 and 1333510645 Missing descriptors for photos 1333510645 and 1333510534 Missing descriptors for photos 1333510534 and 1333510447 Missing descriptors for photos 1333510447 and 1333510354 Missing descriptors for photos 1333510354 and 1333510350 Missing descriptors for photos 1333510350 and 1333510346 Missing descriptors for photos 1333510346 and 1333510970 Missing descriptors for photos 1333511164 and 1333511158 Missing descriptors for photos 1333511158 and 1333511150 Missing descriptors for photos 1333511150 and 1333511146 Missing descriptors for photos 1333511146 and 1333511141 Missing descriptors for photos 1333511141 and 1333511220 Missing descriptors for photos 1333511220 and 1333511213 Missing descriptors for photos 1333511213 and 1333511200 Missing descriptors for photos 1333511200 and 1333511137 Missing descriptors for photos 1333511137 and 1333511187 Missing descriptors for photos 1333511187 and 1333511185 Missing descriptors for photos 1333511185 and 1333511182 Missing descriptors for photos 1333511182 and 1333511484 Missing descriptors for photos 1333511484 and 1333511465 Missing descriptors for photos 1333511465 and 1333511420 Missing descriptors for photos 1333511420 and 1333511401 Missing descriptors for photos 1333511401 and 1333511309 Missing descriptors for photos 1333511309 and 1333511261 01022025 Removing 9 photos because of the 'same image' condition Total on : 1634.0020180000001 list_time_on Total off : 74.996961 list_time_off dist_desc begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 210 time used for this insertion : 0.06156325340270996 photos_removed : len 9 elapsed_time : remove_photo_duplicate 0.11635088920593262 Creating list_photo_total XXXXXXXXXtime_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average time_diff is bigger than the limit of interval, we ignore the result of this image in moving average elapsed_time : count_sum_diff_and_build_graph 0.017441749572753906 Total photos : 210 .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .Change port : 0 hashtag : 495916461 photo_id =1333503024 : jrm .......can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .Change port : 7 hashtag : 2107760237 photo_id =1333507015 : film_pedb ....can't find max_score_info ...........can't find max_score_info ............can't find max_score_info .....can't find max_score_info ...can't find max_score_info ...can't find max_score_info .can't find max_score_info ....can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .Change port : 36 hashtag : 616987804 photo_id =1333510968 : emr ...........can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info .can't find max_score_info Total photos : 210 Number of lists : 4 counter photos in port : 54 hashtag : aluminium(493546845) : 0 photos in 0 portfolios ! hashtag : ela(492741797) : 0 photos in 0 portfolios ! hashtag : emr(616987804) : 11 photos in 1 portfolios ! hashtag : film_pedb(2107760237) : 36 photos in 1 portfolios ! hashtag : flux_dev(2107760238) : 0 photos in 0 portfolios ! hashtag : jrm(495916461) : 7 photos in 1 portfolios ! hashtag : pcm(560181804) : 0 photos in 0 portfolios ! hashtag : pcnc(1284539308) : 0 photos in 0 portfolios ! hashtag : pehd_pp(2107760239) : 0 photos in 0 portfolios ! hashtag : pet_clair(2107755846) : 0 photos in 0 portfolios ! hashtag : refus(538914404) : 0 photos in 0 portfolios ! hashtag : tapis_vide(2107748999) : 0 photos in 1 portfolios ! elapsed_time : group_photo_by_moyenne_exp 0.006234645843505859 elapsed_time : compute_and_correct_tag_with_moyenne_mobile 2.6226043701171875e-06 today str has not a value , we define it as the date of the first image todaystr_first : 01022025 attention , prev_timestamp is 0 , we do nothing ******************Count Time bigger than 30s : 18 #Number Photos for regression : {'01022025': {493546845: {2107751013: 0, 2107751014: 0, 2107751015: 0, 2107751016: 0, 2107751017: 0}, 492741797: {2107751013: 0, 2107751014: 0, 2107751015: 0, 2107751016: 0, 2107751017: 0}, 616987804: {2107751013: 0, 2107751014: 13.00100588798523, 2107751015: 70.99898409843445, 2107751016: 0, 2107751017: 0}, 2107760237: {2107751013: 0, 2107751014: 43.001009941101074, 2107751015: 204.00213479995728, 2107751016: 160.99897718429565, 2107751017: 0}, 2107760238: {2107751013: 0, 2107751014: 0, 2107751015: 0, 2107751016: 0, 2107751017: 0}, 495916461: {2107751013: 0, 2107751014: 36.000004053115845, 2107751015: 7.00101113319397, 2107751016: 10.998989820480347, 2107751017: 0}, 560181804: {2107751013: 0, 2107751014: 0, 2107751015: 43.00102186203003, 2107751016: 0, 2107751017: 0}, 1284539308: {2107751013: 0, 2107751014: 0, 2107751015: 0, 2107751016: 0, 2107751017: 0}, 2107760239: {2107751013: 0, 2107751014: 5.9999401569366455, 2107751015: 0, 2107751016: 0, 2107751017: 0}, 2107755846: {2107751013: 0, 2107751014: 0, 2107751015: 0, 2107751016: 0, 2107751017: 0}, 538914404: {2107751013: 0, 2107751014: 0, 2107751015: 0, 2107751016: 0, 2107751017: 0}, 2107748999: {2107751013: 0, 2107751014: 0, 2107751015: 1486.9979419708252, 2107751016: 0, 2107751017: 0}}} 01022025|aluminium, 05102018_papier_non_papier_dense:0 01022025|aluminium, 05102018_papier_non_papier_peu_dense:0 01022025|aluminium, 05102018_papier_non_papier_presque_vide:0 01022025|aluminium, 05102018_papier_non_papier_tres_dense:0 01022025|aluminium, 05102018_papier_non_papier_tres_peu_dense:0 01022025|ela, 05102018_papier_non_papier_dense:0 01022025|ela, 05102018_papier_non_papier_peu_dense:0 01022025|ela, 05102018_papier_non_papier_presque_vide:0 01022025|ela, 05102018_papier_non_papier_tres_dense:0 01022025|ela, 05102018_papier_non_papier_tres_peu_dense:0 01022025|emr, 05102018_papier_non_papier_dense:0 01022025|emr, 05102018_papier_non_papier_peu_dense:13.00100588798523 01022025|emr, 05102018_papier_non_papier_presque_vide:70.99898409843445 01022025|emr, 05102018_papier_non_papier_tres_dense:0 01022025|emr, 05102018_papier_non_papier_tres_peu_dense:0 01022025|film_pedb, 05102018_papier_non_papier_dense:0 01022025|film_pedb, 05102018_papier_non_papier_peu_dense:43.001009941101074 01022025|film_pedb, 05102018_papier_non_papier_presque_vide:204.00213479995728 01022025|film_pedb, 05102018_papier_non_papier_tres_dense:160.99897718429565 01022025|film_pedb, 05102018_papier_non_papier_tres_peu_dense:0 01022025|flux_dev, 05102018_papier_non_papier_dense:0 01022025|flux_dev, 05102018_papier_non_papier_peu_dense:0 01022025|flux_dev, 05102018_papier_non_papier_presque_vide:0 01022025|flux_dev, 05102018_papier_non_papier_tres_dense:0 01022025|flux_dev, 05102018_papier_non_papier_tres_peu_dense:0 01022025|jrm, 05102018_papier_non_papier_dense:0 01022025|jrm, 05102018_papier_non_papier_peu_dense:36.000004053115845 01022025|jrm, 05102018_papier_non_papier_presque_vide:7.00101113319397 01022025|jrm, 05102018_papier_non_papier_tres_dense:10.998989820480347 01022025|jrm, 05102018_papier_non_papier_tres_peu_dense:0 01022025|pcm, 05102018_papier_non_papier_dense:0 01022025|pcm, 05102018_papier_non_papier_peu_dense:0 01022025|pcm, 05102018_papier_non_papier_presque_vide:43.00102186203003 01022025|pcm, 05102018_papier_non_papier_tres_dense:0 01022025|pcm, 05102018_papier_non_papier_tres_peu_dense:0 01022025|pcnc, 05102018_papier_non_papier_dense:0 01022025|pcnc, 05102018_papier_non_papier_peu_dense:0 01022025|pcnc, 05102018_papier_non_papier_presque_vide:0 01022025|pcnc, 05102018_papier_non_papier_tres_dense:0 01022025|pcnc, 05102018_papier_non_papier_tres_peu_dense:0 01022025|pehd_pp, 05102018_papier_non_papier_dense:0 01022025|pehd_pp, 05102018_papier_non_papier_peu_dense:5.9999401569366455 01022025|pehd_pp, 05102018_papier_non_papier_presque_vide:0 01022025|pehd_pp, 05102018_papier_non_papier_tres_dense:0 01022025|pehd_pp, 05102018_papier_non_papier_tres_peu_dense:0 01022025|pet_clair, 05102018_papier_non_papier_dense:0 01022025|pet_clair, 05102018_papier_non_papier_peu_dense:0 01022025|pet_clair, 05102018_papier_non_papier_presque_vide:0 01022025|pet_clair, 05102018_papier_non_papier_tres_dense:0 01022025|pet_clair, 05102018_papier_non_papier_tres_peu_dense:0 01022025|refus, 05102018_papier_non_papier_dense:0 01022025|refus, 05102018_papier_non_papier_peu_dense:0 01022025|refus, 05102018_papier_non_papier_presque_vide:0 01022025|refus, 05102018_papier_non_papier_tres_dense:0 01022025|refus, 05102018_papier_non_papier_tres_peu_dense:0 01022025|tapis_vide, 05102018_papier_non_papier_dense:0 01022025|tapis_vide, 05102018_papier_non_papier_peu_dense:0 01022025|tapis_vide, 05102018_papier_non_papier_presque_vide:1486.9979419708252 01022025|tapis_vide, 05102018_papier_non_papier_tres_dense:0 01022025|tapis_vide, 05102018_papier_non_papier_tres_peu_dense:0 #Number Photos for regression amount gros magasin papier (time_diff then nb_photo) : We have not displayed the number of photos removed for one material since Rungis_Papier wasn't in the thcl used ! 01022025_time_diff_distrib Number amount portfolio for this type of dechet : aluminium 0 Number amount portfolio for this type of dechet : ela 0 Number amount portfolio for this type of dechet : emr 10 https://marlene.fotonower.com/api/v1/secured/portfolio/new?name=01022025_emr_05102018_papier_non_papier_peu_dense&access_token=0fc1cdda0f63f39f777d9cb33b1aa204 Created to study and clean : 20173918 with name like 01022025_emr_05102018_papier_non_papier_peu_dense https://marlene.fotonower.com/api/v1/secured/portfolio/new?name=01022025_emr_05102018_papier_non_papier_presque_vide&access_token=0fc1cdda0f63f39f777d9cb33b1aa204 Created to study and clean : 20173919 with name like 01022025_emr_05102018_papier_non_papier_presque_vide Number amount portfolio for this type of dechet : film_pedb 35 https://marlene.fotonower.com/api/v1/secured/portfolio/new?name=01022025_film_pedb_05102018_papier_non_papier_peu_dense&access_token=0fc1cdda0f63f39f777d9cb33b1aa204 Created to study and clean : 20173920 with name like 01022025_film_pedb_05102018_papier_non_papier_peu_dense https://marlene.fotonower.com/api/v1/secured/portfolio/new?name=01022025_film_pedb_05102018_papier_non_papier_presque_vide&access_token=0fc1cdda0f63f39f777d9cb33b1aa204 Created to study and clean : 20173921 with name like 01022025_film_pedb_05102018_papier_non_papier_presque_vide https://marlene.fotonower.com/api/v1/secured/portfolio/new?name=01022025_film_pedb_05102018_papier_non_papier_tres_dense&access_token=0fc1cdda0f63f39f777d9cb33b1aa204 Created to study and clean : 20173922 with name like 01022025_film_pedb_05102018_papier_non_papier_tres_dense Number amount portfolio for this type of dechet : flux_dev 0 Number amount portfolio for this type of dechet : jrm 5 Number amount portfolio for this type of dechet : pcm 3 Number amount portfolio for this type of dechet : pcnc 0 Number amount portfolio for this type of dechet : pehd_pp 1 https://marlene.fotonower.com/api/v1/secured/portfolio/new?name=01022025_pehd_pp_05102018_papier_non_papier_peu_dense&access_token=0fc1cdda0f63f39f777d9cb33b1aa204 Created to study and clean : 20173923 with name like 01022025_pehd_pp_05102018_papier_non_papier_peu_dense Number amount portfolio for this type of dechet : pet_clair 0 Number amount portfolio for this type of dechet : refus 0 Number amount portfolio for this type of dechet : tapis_vide 146 https://marlene.fotonower.com/api/v1/secured/portfolio/new?name=01022025_tapis_vide_05102018_papier_non_papier_presque_vide&access_token=0fc1cdda0f63f39f777d9cb33b1aa204 Created to study and clean : 20173924 with name like 01022025_tapis_vide_05102018_papier_non_papier_presque_vide NUMBER BATCH : 4 list_ponderation used : [0.001, 0.001, 0.001, 0.001, 0.001, 0.001, 0.001, 0.001, 0.001, 0.001, 0.001] , list_hashtag_class_create_as_list : ['pcnc', 'gm', 'emr', 'ela', 'pet_clair', 'film_pedb', 'pehd_pp', 'flux_dev_rigide', 'aluminium', 'tapis_vide', 'refus', 'gros_cartons', 'gm', 'flux_dev_rigide', 'gros_cartons'] We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! hashtag non trouvé ! result_one_balle_Type_jrm:{'day': '01022025', 'map_nb_amount': {0: 2, 1: 3, 2: 2, 3: 0, 4: 0}, 'map_time_amount': {0: 36.000004053115845, 1: 43.00101113319397, 2: 10.998989820480347, 3: 0, 4: 0}, 'duration': 60.00000190734863, 'nb_balles_papier': 0.09000000500679017, 'begin_time_port': 'image_01022025_00_05_22_6189.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.09000000500679017 hashtag non trouvé ! hashtag non trouvé ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! result_one_balle_Type_film_pedb:{'day': '01022025', 'map_nb_amount': {0: 4, 1: 16, 2: 16, 3: 0, 4: 0}, 'map_time_amount': {0: 49.00095009803772, 1: 204.00213479995728, 2: 160.99897718429565, 3: 0, 4: 0}, 'duration': 426.0010690689087, 'nb_balles_papier': 0.4140020620822907, 'begin_time_port': 'image_01022025_00_21_22_5182.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.4140020620822907 hashtag non trouvé ! We filter photos on hashtag condition ! hashtag non trouvé ! hashtag non trouvé ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! result_one_balle_Type_emr:{'day': '01022025', 'map_nb_amount': {0: 2, 1: 9, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 13.00100588798523, 1: 78.00000596046448, 2: 0, 3: 0, 4: 0}, 'duration': 83.99998998641968, 'nb_balles_papier': 0.0910010118484497, 'begin_time_port': 'image_01022025_00_31_28_6198.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0910010118484497 We filter photos on hashtag condition ! hashtag non trouvé ! We filter photos on hashtag condition ! hashtag non trouvé ! hashtag non trouvé ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We have rejected 0 photos because of the batch_size condition ! NUMBER BATCH list_of_portfolios_to_create : 3 type_dechet jrm not in list to create port, so we just list it with its name (VR TODO 3-6-19 : we need to change this) list_same_port_ids : [] https://marlene.fotonower.com/api/v1/secured/portfolio/new?name=film_pedb_diff_batch__01022025_00_21_22_005182&access_token=0fc1cdda0f63f39f777d9cb33b1aa204 list_same_port_ids : [] https://marlene.fotonower.com/api/v1/secured/portfolio/new?name=emr_diff_batch__01022025_00_31_28_006198&access_token=0fc1cdda0f63f39f777d9cb33b1aa204 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11978 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11987 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11986 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11982 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11982 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11985 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11985 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11979 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11979 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11990 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11990 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11981 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11980 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11980 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11989 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11981 doesn't seem to be define in the database( WARNING : type of input 3 of step 11980 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11978 doesn't seem to be define in the database( WARNING : type of input 2 of step 11982 doesn't seem to be define in the database( WARNING : output 1 of step 11978 have datatype=2 whereas input 1 of step 11985 have datatype=7 WARNING : type of output 2 of step 11985 doesn't seem to be define in the database( WARNING : type of input 1 of step 11979 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11981 have datatype=10 whereas input 3 of step 11988 have datatype=6 WARNING : type of input 2 of step 11990 doesn't seem to be define in the database( WARNING : output 1 of step 11979 have datatype=7 whereas input 2 of step 11990 have datatype=None WARNING : type of output 3 of step 11990 doesn't seem to be define in the database( WARNING : type of input 1 of step 11981 doesn't seem to be define in the database( WARNING : output 0 of step 11981 have datatype=10 whereas input 0 of step 11991 have datatype=18 WARNING : type of input 5 of step 11988 doesn't seem to be define in the database( WARNING : output 0 of step 11991 have datatype=11 whereas input 5 of step 11988 have datatype=None WARNING : type of output 1 of step 11987 doesn't seem to be define in the database( WARNING : type of input 3 of step 11982 doesn't seem to be define in the database( WARNING : type of output 1 of step 11986 doesn't seem to be define in the database( WARNING : type of input 3 of step 11982 doesn't seem to be define in the database( WARNING : output 0 of step 11985 have datatype=1 whereas input 0 of step 11979 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20173925 AND mptpi.`type`=4461 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20173926 AND mptpi.`type`=4207 To do elapsed_time : count_nb_balles_and_create_portfolio 10.033345222473145 # DISPLAY ALL COLLECTED DATA : {'01022025': {'nb_upload': 210, 'nb_taggue_class': 210, 'nb_taggue_densite': 210, 'nb_descriptors': 63, 'number_port': 4, 'count_photo_in_port': 54, 'nb_port_per_class': {'aluminium': {'nb_photos': 0, 'nb_portfolios': 0}, 'ela': {'nb_photos': 0, 'nb_portfolios': 0}, 'emr': {'nb_photos': 11, 'nb_portfolios': 1}, 'film_pedb': {'nb_photos': 36, 'nb_portfolios': 1}, 'flux_dev': {'nb_photos': 0, 'nb_portfolios': 0}, 'jrm': {'nb_photos': 7, 'nb_portfolios': 1}, 'pcm': {'nb_photos': 0, 'nb_portfolios': 0}, 'pcnc': {'nb_photos': 0, 'nb_portfolios': 0}, 'pehd_pp': {'nb_photos': 0, 'nb_portfolios': 0}, 'pet_clair': {'nb_photos': 0, 'nb_portfolios': 0}, 'refus': {'nb_photos': 0, 'nb_portfolios': 0}, 'tapis_vide': {'nb_photos': 0, 'nb_portfolios': 1}}}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1333511484, 1333511465, 1333511420, 1333511401, 1333511309, 1333511261, 1333511220, 1333511213, 1333511200, 1333511187, 1333511185, 1333511182, 1333511164, 1333511158, 1333511150, 1333511146, 1333511141, 1333511137, 1333511013, 1333511009, 1333511004, 1333511000, 1333510996, 1333510992, 1333510970, 1333510968, 1333510935, 1333510900, 1333510896, 1333510868, 1333510645, 1333510534, 1333510447, 1333510354, 1333510350, 1333510346, 1333510265, 1333510263, 1333510260, 1333510256, 1333510252, 1333510247, 1333510226, 1333510223, 1333510218, 1333510216, 1333510212, 1333510211, 1333510183, 1333510181, 1333510176, 1333510171, 1333510046, 1333510006, 1333509895, 1333509891, 1333509886, 1333509884, 1333509882, 1333509881, 1333509733, 1333509699, 1333509655, 1333509654, 1333509631, 1333509518, 1333508969, 1333508744, 1333508720, 1333508689, 1333508586, 1333508583, 1333508519, 1333508516, 1333508504, 1333508501, 1333508439, 1333508218, 1333507061, 1333507057, 1333507053, 1333507051, 1333507049, 1333507048, 1333507015, 1333507013, 1333507012, 1333507011, 1333507010, 1333507007, 1333506899, 1333506888, 1333506853, 1333506831, 1333506829, 1333506761, 1333506566, 1333506553, 1333506531, 1333506519] Looping around the photos to save general results len do output : 1 /20127079Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511484', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511465', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511420', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511401', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511309', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511261', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511220', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511213', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511200', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511187', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511185', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511182', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511164', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511158', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511150', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511146', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511141', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511137', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511013', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511009', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511004', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333511000', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510996', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510992', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510970', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510968', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510935', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510900', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510896', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510868', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510645', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510534', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510447', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510354', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510350', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510346', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510265', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510263', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510260', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510256', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510252', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510247', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510226', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510223', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510218', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510216', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510212', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510211', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510183', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510181', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510176', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510171', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510046', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333510006', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333509895', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333509891', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333509886', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333509884', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333509882', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333509881', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333509733', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333509699', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333509655', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333509654', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333509631', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333509518', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333508969', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333508744', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333508720', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333508689', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333508586', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333508583', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333508519', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333508516', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333508504', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333508501', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333508439', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333508218', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333507061', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333507057', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333507053', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333507051', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333507049', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333507048', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333507015', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333507013', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333507012', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333507011', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333507010', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333507007', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333506899', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333506888', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333506853', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333506831', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333506829', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333506761', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333506566', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333506553', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333506531', None, None, None, None, None, '2539597') ('4323', None, None, None, None, None, None, None, '2539597') ('4323', '20127079', '1333506519', None, None, None, None, None, '2539597') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 101 time used for this insertion : 0.04790353775024414 save_final save missing photos in datou_result : time spend for datou_step_exec : 16.36275362968445 time spend to save output : 0.04863548278808594 total time spend for step 1 : 16.411389112472534 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 2 len(input) + len(total_photo_id_missing) : 1 set_done_treatment 1.29user 0.56system 0:20.66elapsed 9%CPU (0avgtext+0avgdata 102560maxresident)k 192inputs+664outputs (8major+49804minor)pagefaults 0swaps