WARNING : output 0 of step 11084 have datatype=16 whereas input 0 of step 11098 have datatype=1 WARNING : output 1 of step 11084 have datatype=2 whereas input 1 of step 11098 have datatype=7 WARNING : output 0 of step 11086 have datatype=6 whereas input 2 of step 11098 have datatype=5 WARNING : type of output 3 of step 11098 doesn't seem to be define in the database( WARNING : type of input 1 of step 11091 doesn't seem to be define in the database( DataTypes for each output/input checked ! {"input_values":[]} Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 4230, 'hashtag_proportion': 'papier,carton,metal,pet_clair,autre,pehd,pet_fonce', 'hashtag_parmi': 'entrant', 'hashtag_weights': {'autre': 3.0, 'barquette_avec_film': 3.0, 'carton': 3.0, 'metal': 3.0, 'papier': 3.0, 'pehd': 3.0, 'pet_fonce': 3.0, 'pet_clair': 1.0, 'flou': 1.0, 'mal_croppe': 1.0}} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier place_id : 99 month : 08 mtr_id : dashboard_places : SyctomXV date : 2023-08 year : 2023 raspi : 0 select count(distinct mtr_photo_id) from MTRUser.mtr_portfolio_photos where mtr_portfolio_id in (select id from MTRUser.mtr_portfolios where id in (select mtr_portfolio_id from MTRPhoto.dashboard_results where dashboard_run_id in(select last_run_id from MTRPhoto.dashboard_entry_day where dashboard_place_id in (select id from MTRPhoto.dashboard_places where name = 'SyctomXV' and date like '%2023-08%') and created_at like '%2023-08%'))); nb_day : (1, 31) after unwanted_material_data We return tuple() but get_coverage_data_from_list_of_date return unexpectedly data from stuff in error with raspi=0 so we force here ! after coverage_data SELECT DATE(date_debut) as date, sum(qualite*nombre_balle)/sum(nombre_balle), count(distinct mtr_photo_id) FROM MTRPhoto.dashboard_results dr, MTRUser.mtr_portfolio_photos mpp, MTRPhoto.dashboard_entry_day ded where mpp.mtr_portfolio_id=dr.mtr_portfolio_id and ded.dashboard_place_id = 99 and month(ded.date) = 08 and year(ded.date) = 2023 and dr.dashboard_run_id=ded.last_run_id and dr.qualite>=0 GROUP BY ded.date order by dr.date_debut; select avg(cps.score) as score, SUBSTRING_INDEX(SUBSTRING_INDEX(mp.name,'__', -1), '_', 1) as sbstr, SUBSTRING_INDEX(SUBSTRING_INDEX(mp.name,'_', -2), '_', 1) as sbstr2, SUBSTRING_INDEX(mp.name,'_', -1) as sbstr3 FROM MTRUser.mtr_portfolio_photos mpp, MTRUser.mtr_portfolios mp,MTRPhoto.class_photo_score cps,MTRBack.hashtags h,MTRPhoto.dashboard_results dr,MTRPhoto.dashboard_entry_day ded, MTRPhoto.dashboard_places dp WHERE mp.id=mpp.mtr_portfolio_id and cps.photo_id=mpp.mtr_photo_id and h.hashtag_id=cps.hashtag_id and dr.mtr_portfolio_id = mpp.mtr_portfolio_id and dr.dashboard_run_id = ded.last_run_id and ded.dashboard_place_id = dp.id and h.hashtag in ('blur', 'mean_blur', 'clear') and dp.id = 99 and mp.name like '%082023%' and cps.thcl = 1533 GROUP BY sbstr ; after number_of_batch date_start : 2023-08-01 : dt_date_just_month_year : 2023-08-01 00:00:00 : VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! Error parsing crontab ! [Errno 2] No such file or directory: '' You better check your LOGRASPI env !!! after pl.get_datou_sts_from_crontab : verbose : True no sts found, try to find from database SELECT dri.id FROM MTRPhoto.dashboard_run_ids dri, MTRPhoto.dashboard_entry_day ded, MTRPhoto.dashboard_places dp WHERE dp.name= "SyctomXV" AND ded.dashboard_place_id=dp.id AND dri.dashboard_entry_day=ded.id AND dri.id=ded.last_run_id AND ded.date >= "2023-08-01" AND ded.date <= "2023-08-31" apple3 {'test': {'mat': 'test', 'pht': 0, 'datou_carac_id': 370, 'unwanted_material': [], 'hashtag_majoritaire_from_carac': 538914404}, 'entrant': {'mat': 'entrant', 'pht': 4230, 'datou_carac_id': 4142, 'unwanted_material': [], 'hashtag_majoritaire_from_carac': 'entrant'}} [] after get_hostname_from_raspi after impurety_average_per_hashtag {'number_of_analysed_photos': ((0,),), 'unwanted_material_monthly_percentage': 0, 'material_repartition': (), 'material_dict': {}, 'coverage_data': (), 'impurety_average_data': (), 'impurety_average_data_per_hashtag': {}, 'quality_array': [], 'number_of_batch': ((0,),), 'datou_id_sts': 4095, 'map_hashtag_number_mat': {'hash_to_mat_nb': [], 'main_stat': {'nb_photos': 0}}, 'hostname': (), 'impurety_average_per_hashtag': []} TODO : Insert select and so on SELECT id, name, misc_info, datou_id_for_right FROM MTRPhoto.dashboard_places WHERE name="SyctomXV"; SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4095 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4095 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4095 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4095 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! {"input_values":[]} We expected an STS datou with a camera_place_name, so there is a PB, datou_id: 4095 ! {'id': 4095, 'name': 'split_time_score_cod_ParisXV', 'limit_max': 150, 'nb_steps': 1, 'estimated_time': 86400, 'list_steps': [{'id': 10869, 'type': 'split_time_score_with_photo', 'param': '0', 'json_param_as_string': "{'cod': 1, 'camera_place_name': 'SyctomXV', 'ETA': 86400, 'param_moyenne_mobile': 5, 'thcl': 3379, 'list_task_complete': [''], 'list_tasks': 'LOAD,ORDER,CONVENTION,SPLIT_BY_DARK,SPLIT_TIME, CREATE_PORT_LAUNCH_CARAC_FILL_DASHBOARD', 'forced_hashtag': 'entrant', 'list_hashtag_class_create': 'entrant', 'ponderation': '0.001,0.001,0.001,0.001,0.001', 'host': 'www.fotonower.com', 'datou_proportion_papier': {'test': 370, 'entrant': 4142}, 'input_default_values': []}", 'json_param': {'cod': 1, 'camera_place_name': 'SyctomXV', 'ETA': 86400, 'param_moyenne_mobile': 5, 'thcl': 3379, 'list_task_complete': [''], 'list_tasks': 'LOAD,ORDER,CONVENTION,SPLIT_BY_DARK,SPLIT_TIME, CREATE_PORT_LAUNCH_CARAC_FILL_DASHBOARD', 'forced_hashtag': 'entrant', 'list_hashtag_class_create': 'entrant', 'ponderation': '0.001,0.001,0.001,0.001,0.001', 'host': 'www.fotonower.com', 'datou_proportion_papier': {'test': 370, 'entrant': 4142}, 'input_default_values': []}, 'nb_input': 2, 'nb_output': 1, 'prod': False, 'is_local': False, 'is_datou_depend': False, 'is_photo_id_local': False, 'list_data': []}], 'depends_step': []} check if it is the case where we have multiple sts in one datou yes, we found a datou with multiple sts, we will choose the last one by default SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=370 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=370 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 370 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=370 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! DataTypes for each output/input checked ! {"input_values":[]} Unexpected error treated as WARNING for now expected given param in final step {'min_score': 0.7, 'name_pipeline': 'proportion_papier_tapis', 'hashtag_type': 705, 'hashtag_proportion': 492668766, 'hashtag_parmi': 538914404, 'hashtag_to_consider': 'papier_only', 'hashtag_background': 493012381} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier ventilate_hashtags_in_portfolio step not found ! SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4142 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4142 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4142 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4142 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11084 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11087 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11087 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11086 argmax have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 11098 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11090 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11090 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11094 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11094 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11091 ventilate_hashtags_in_portfolio is not consistent : 3 used against 1 in the step definition ! WARNING : number of outputs for step 11091 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11092 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11092 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11097 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11084 doesn't seem to be define in the database( WARNING : type of input 2 of step 11087 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11091 have datatype=10 whereas input 2 of step 11093 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11094 doesn't seem to be define in the database( WARNING : output 1 of step 11090 have datatype=7 whereas input 2 of step 11094 have datatype=None WARNING : type of output 3 of step 11094 doesn't seem to be define in the database( WARNING : type of input 1 of step 11091 doesn't seem to be define in the database( WARNING : type of output 1 of step 11091 doesn't seem to be define in the database( WARNING : type of input 3 of step 11092 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11091 have datatype=10 whereas input 0 of step 11095 have datatype=18 WARNING : type of input 5 of step 11093 doesn't seem to be define in the database( WARNING : output 0 of step 11095 have datatype=11 whereas input 5 of step 11093 have datatype=None WARNING : type of input 2 of step 11091 doesn't seem to be define in the database( WARNING : output 0 of step 11096 have datatype=5 whereas input 2 of step 11091 have datatype=None WARNING : output 0 of step 11093 have datatype=10 whereas input 0 of step 11097 have datatype=18 WARNING : type of output 2 of step 11098 doesn't seem to be define in the database( WARNING : type of input 1 of step 11090 doesn't seem to be define in the database( WARNING : output 0 of step 11084 have datatype=16 whereas input 0 of step 11098 have datatype=1 WARNING : output 1 of step 11084 have datatype=2 whereas input 1 of step 11098 have datatype=7 WARNING : output 0 of step 11086 have datatype=6 whereas input 2 of step 11098 have datatype=5 WARNING : type of output 3 of step 11098 doesn't seem to be define in the database( WARNING : type of input 1 of step 11091 doesn't seem to be define in the database( DataTypes for each output/input checked ! {"input_values":[]} Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 4230, 'hashtag_proportion': 'papier,carton,metal,pet_clair,autre,pehd,pet_fonce', 'hashtag_parmi': 'entrant', 'hashtag_weights': {'autre': 3.0, 'barquette_avec_film': 3.0, 'carton': 3.0, 'metal': 3.0, 'papier': 3.0, 'pehd': 3.0, 'pet_fonce': 3.0, 'pet_clair': 1.0, 'flou': 1.0, 'mal_croppe': 1.0}} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier {'number_of_analysed_photos': ((0,),), 'unwanted_material_monthly_percentage': 0, 'material_repartition': (), 'material_dict': {}, 'coverage_data': (), 'impurety_average_data': (), 'impurety_average_data_per_hashtag': {}, 'quality_array': [], 'number_of_batch': ((0,),), 'datou_id_sts': 4095, 'map_hashtag_number_mat': {'hash_to_mat_nb': [], 'main_stat': {'nb_photos': 0}}, 'hostname': (), 'impurety_average_per_hashtag': []} filepath : /data_2/data_log/job/memo/SyctomXV/sla_mensuel_SyctomXV_mois_08_annee_2023.pdf Before plot_photos_and_unwanted_material_summary # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! Before organize_and_plot_unwanted_material_stats organize_and_plot_unwanted_material_stats returned null before organize_and_plot_impurety_average_data WARNING : no impurety_average_data, we do not create this page, by the way, c'est pas normal ! before organize_and_plot_quality_photo_data finish generating filepath : /data_2/data_log/job/memo/SyctomXV/sla_mensuel_SyctomXV_mois_08_annee_2023.pdf le fichier se trouve ici : /data_2/data_log/job/memo/SyctomXV 2.50user 2.72system 0:05.61elapsed 93%CPU (0avgtext+0avgdata 191200maxresident)k 105200inputs+136outputs (448major+65798minor)pagefaults 0swaps