python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120 import MySQLdb succeeded Import error (python version) python version = 3 warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data list_job_run_as_list : ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel'] python version used : 3 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')] #&_# BEGIN OF TEST : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/mask_test.py Test mask-detection python version used : 3 ############################### TEST memory used ################################ free memory at begining : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5290 run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 0 ; length of list_pids : 0 ; length of list_args : 0 time to download the photos : 0.009079694747924805 About to test input to load empty input or other problem ret : 0 no datou_current to update free memory after detection : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5290 ############################### TEST detect object ################################ run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 0 ; length of list_pids : 0 ; length of list_args : 0 time to download the photos : 0.005881309509277344 About to test input to load empty input or other problem ret : 0 no datou_current to update ERROR : cannot unpack non-iterable NoneType object reconnect to base ! warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data #&_# TEST FAILED #&_# : tests/mask_test #&_# Error : invalid literal for int() with base 10: 'cannot unpack non-iterable NoneType object' /home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/migration_mbp_22_09_25_6b97d44a6efa1f1691f192e531c2c17a72e983b6 SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/migration_mbp_22_09_25_6b97d44a6efa1f1691f192e531c2c17a72e983b6','{"mask_detection": "fail"}','0','http://marlene.fotonower-preprod.com/job/2025/September/24092025/python_test3/','mask_detection','unknown'); #&_# END OF TEST #&_# : tests/mask_test #&_# #&_# BEGIN OF TEST : tests/datou_test #&_# /home/admin/workarea/git/Velours/python/tests/datou_test.py Datou All Test python version used : 3 ############################### TEST sam ################################ TEST SAM Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4573 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4573 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4573 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : sam list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.view_photos_with_urls_250912 ph WHERE photo_id IN (1189321094) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 1189321094 download finish for photo 1189321094 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.17151904106140137 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! WARNING : we have an input that is not a photo, we should get rid of it Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:sam Wed Sep 24 17:33:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1758728011_1852108_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094} map_photo_id_path_extension : {1189321094: {'path': 'temp/1758728011_1852108_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}} map_subphoto_mainphoto : {} Beginning of datou step sam ! pht : 4677 Inside sam : nb paths : 1 (640, 960, 3) time for calcul the mask position with numpy : 0.0022547245025634766 nb_pixel_total : 16184 time to create 1 rle with old method : 0.01718926429748535 time for calcul the mask position with numpy : 0.0014140605926513672 nb_pixel_total : 13972 time to create 1 rle with old method : 0.01570749282836914 time for calcul the mask position with numpy : 0.0024690628051757812 nb_pixel_total : 83592 time to create 1 rle with old method : 0.08987307548522949 time for calcul the mask position with numpy : 0.0014317035675048828 nb_pixel_total : 14654 time to create 1 rle with old method : 0.016569852828979492 time for calcul the mask position with numpy : 0.0014812946319580078 nb_pixel_total : 5614 time to create 1 rle with old method : 0.006391286849975586 time for calcul the mask position with numpy : 0.0015118122100830078 nb_pixel_total : 16441 time to create 1 rle with old method : 0.018268108367919922 time for calcul the mask position with numpy : 0.0015025138854980469 nb_pixel_total : 8611 time to create 1 rle with old method : 0.009392738342285156 time for calcul the mask position with numpy : 0.0014214515686035156 nb_pixel_total : 3774 time to create 1 rle with old method : 0.004133462905883789 time for calcul the mask position with numpy : 0.0013511180877685547 nb_pixel_total : 6636 time to create 1 rle with old method : 0.0075647830963134766 time for calcul the mask position with numpy : 0.0013539791107177734 nb_pixel_total : 10831 time to create 1 rle with old method : 0.012184619903564453 time for calcul the mask position with numpy : 0.0014493465423583984 nb_pixel_total : 2949 time to create 1 rle with old method : 0.0034890174865722656 time for calcul the mask position with numpy : 0.0015721321105957031 nb_pixel_total : 38696 time to create 1 rle with old method : 0.04189658164978027 time for calcul the mask position with numpy : 0.0014660358428955078 nb_pixel_total : 9508 time to create 1 rle with old method : 0.010566234588623047 time for calcul the mask position with numpy : 0.0015811920166015625 nb_pixel_total : 29440 time to create 1 rle with old method : 0.031852006912231445 time for calcul the mask position with numpy : 0.0013897418975830078 nb_pixel_total : 3300 time to create 1 rle with old method : 0.0037009716033935547 time for calcul the mask position with numpy : 0.0013244152069091797 nb_pixel_total : 4234 time to create 1 rle with old method : 0.0046617984771728516 time for calcul the mask position with numpy : 0.001325368881225586 nb_pixel_total : 2453 time to create 1 rle with old method : 0.002840757369995117 time for calcul the mask position with numpy : 0.0013127326965332031 nb_pixel_total : 4169 time to create 1 rle with old method : 0.004750251770019531 time for calcul the mask position with numpy : 0.0014472007751464844 nb_pixel_total : 3924 time to create 1 rle with old method : 0.004498481750488281 time for calcul the mask position with numpy : 0.0013651847839355469 nb_pixel_total : 1227 time to create 1 rle with old method : 0.0014939308166503906 time for calcul the mask position with numpy : 0.0014688968658447266 nb_pixel_total : 7640 time to create 1 rle with old method : 0.008499622344970703 time for calcul the mask position with numpy : 0.0014433860778808594 nb_pixel_total : 2780 time to create 1 rle with old method : 0.0032804012298583984 time for calcul the mask position with numpy : 0.0014395713806152344 nb_pixel_total : 2078 time to create 1 rle with old method : 0.002302408218383789 time for calcul the mask position with numpy : 0.0014767646789550781 nb_pixel_total : 13109 time to create 1 rle with old method : 0.016519546508789062 time for calcul the mask position with numpy : 0.0013883113861083984 nb_pixel_total : 4248 time to create 1 rle with old method : 0.005060911178588867 time for calcul the mask position with numpy : 0.001377105712890625 nb_pixel_total : 584 time to create 1 rle with old method : 0.0007011890411376953 time for calcul the mask position with numpy : 0.0013744831085205078 nb_pixel_total : 876 time to create 1 rle with old method : 0.0010993480682373047 time for calcul the mask position with numpy : 0.0014200210571289062 nb_pixel_total : 1648 time to create 1 rle with old method : 0.002073526382446289 time for calcul the mask position with numpy : 0.0013952255249023438 nb_pixel_total : 5479 time to create 1 rle with old method : 0.006201505661010742 time for calcul the mask position with numpy : 0.001432657241821289 nb_pixel_total : 9836 time to create 1 rle with old method : 0.011245250701904297 time for calcul the mask position with numpy : 0.0013427734375 nb_pixel_total : 2801 time to create 1 rle with old method : 0.003257274627685547 time for calcul the mask position with numpy : 0.0013728141784667969 nb_pixel_total : 11912 time to create 1 rle with old method : 0.01327371597290039 time for calcul the mask position with numpy : 0.0013909339904785156 nb_pixel_total : 3535 time to create 1 rle with old method : 0.004098653793334961 time for calcul the mask position with numpy : 0.0013742446899414062 nb_pixel_total : 8606 time to create 1 rle with old method : 0.009769439697265625 time for calcul the mask position with numpy : 0.0014050006866455078 nb_pixel_total : 5322 time to create 1 rle with old method : 0.006146669387817383 time for calcul the mask position with numpy : 0.0014133453369140625 nb_pixel_total : 595 time to create 1 rle with old method : 0.0007152557373046875 time for calcul the mask position with numpy : 0.0014226436614990234 nb_pixel_total : 2448 time to create 1 rle with old method : 0.002824068069458008 time for calcul the mask position with numpy : 0.0014324188232421875 nb_pixel_total : 971 time to create 1 rle with old method : 0.0011675357818603516 time for calcul the mask position with numpy : 0.0013916492462158203 nb_pixel_total : 10614 time to create 1 rle with old method : 0.011947393417358398 time for calcul the mask position with numpy : 0.0013892650604248047 nb_pixel_total : 814 time to create 1 rle with old method : 0.0010256767272949219 time for calcul the mask position with numpy : 0.0014231204986572266 nb_pixel_total : 13051 time to create 1 rle with old method : 0.014833211898803711 time for calcul the mask position with numpy : 0.0013263225555419922 nb_pixel_total : 342 time to create 1 rle with old method : 0.00041604042053222656 time for calcul the mask position with numpy : 0.0013613700866699219 nb_pixel_total : 1253 time to create 1 rle with old method : 0.0014576911926269531 time for calcul the mask position with numpy : 0.0014925003051757812 nb_pixel_total : 4173 time to create 1 rle with old method : 0.0051975250244140625 time for calcul the mask position with numpy : 0.0013823509216308594 nb_pixel_total : 3951 time to create 1 rle with old method : 0.004584312438964844 time for calcul the mask position with numpy : 0.001367807388305664 nb_pixel_total : 1354 time to create 1 rle with old method : 0.0016453266143798828 time for calcul the mask position with numpy : 0.0014278888702392578 nb_pixel_total : 1024 time to create 1 rle with old method : 0.0011298656463623047 time for calcul the mask position with numpy : 0.001592874526977539 nb_pixel_total : 39183 time to create 1 rle with old method : 0.04279899597167969 time for calcul the mask position with numpy : 0.0014362335205078125 nb_pixel_total : 868 time to create 1 rle with old method : 0.0011632442474365234 time for calcul the mask position with numpy : 0.0013089179992675781 nb_pixel_total : 2260 time to create 1 rle with old method : 0.002590656280517578 time for calcul the mask position with numpy : 0.001348257064819336 nb_pixel_total : 337 time to create 1 rle with old method : 0.0004837512969970703 time for calcul the mask position with numpy : 0.0013718605041503906 nb_pixel_total : 875 time to create 1 rle with old method : 0.0010609626770019531 time for calcul the mask position with numpy : 0.0013241767883300781 nb_pixel_total : 2323 time to create 1 rle with old method : 0.00269317626953125 time for calcul the mask position with numpy : 0.0014290809631347656 nb_pixel_total : 588 time to create 1 rle with old method : 0.0007536411285400391 time for calcul the mask position with numpy : 0.0013613700866699219 nb_pixel_total : 13448 time to create 1 rle with old method : 0.014626741409301758 time for calcul the mask position with numpy : 0.0013020038604736328 nb_pixel_total : 2407 time to create 1 rle with old method : 0.0029196739196777344 time for calcul the mask position with numpy : 0.001394033432006836 nb_pixel_total : 1164 time to create 1 rle with old method : 0.0014748573303222656 time for calcul the mask position with numpy : 0.0013706684112548828 nb_pixel_total : 1203 time to create 1 rle with old method : 0.0013675689697265625 time for calcul the mask position with numpy : 0.001401662826538086 nb_pixel_total : 2377 time to create 1 rle with old method : 0.002752542495727539 time for calcul the mask position with numpy : 0.0013911724090576172 nb_pixel_total : 693 time to create 1 rle with old method : 0.0008752346038818359 time for calcul the mask position with numpy : 0.0013184547424316406 nb_pixel_total : 1637 time to create 1 rle with old method : 0.0019779205322265625 time for calcul the mask position with numpy : 0.001321554183959961 nb_pixel_total : 577 time to create 1 rle with old method : 0.0006783008575439453 time for calcul the mask position with numpy : 0.001386880874633789 nb_pixel_total : 1707 time to create 1 rle with old method : 0.0019931793212890625 time for calcul the mask position with numpy : 0.0014495849609375 nb_pixel_total : 2776 time to create 1 rle with old method : 0.003320455551147461 time for calcul the mask position with numpy : 0.001592397689819336 nb_pixel_total : 27544 time to create 1 rle with old method : 0.02961111068725586 time for calcul the mask position with numpy : 0.0013384819030761719 nb_pixel_total : 1055 time to create 1 rle with old method : 0.0013530254364013672 time for calcul the mask position with numpy : 0.0013194084167480469 nb_pixel_total : 1075 time to create 1 rle with old method : 0.001280069351196289 time for calcul the mask position with numpy : 0.0014061927795410156 nb_pixel_total : 3093 time to create 1 rle with old method : 0.0035483837127685547 time for calcul the mask position with numpy : 0.001325845718383789 nb_pixel_total : 891 time to create 1 rle with old method : 0.001130819320678711 time for calcul the mask position with numpy : 0.0014758110046386719 nb_pixel_total : 16678 time to create 1 rle with old method : 0.019190073013305664 time for calcul the mask position with numpy : 0.0013914108276367188 nb_pixel_total : 1791 time to create 1 rle with old method : 0.001972675323486328 time for calcul the mask position with numpy : 0.0014162063598632812 nb_pixel_total : 1014 time to create 1 rle with old method : 0.001222848892211914 time for calcul the mask position with numpy : 0.001310110092163086 nb_pixel_total : 263 time to create 1 rle with old method : 0.00036215782165527344 time for calcul the mask position with numpy : 0.001413583755493164 nb_pixel_total : 8452 time to create 1 rle with old method : 0.009121179580688477 time for calcul the mask position with numpy : 0.0013203620910644531 nb_pixel_total : 1515 time to create 1 rle with old method : 0.001672983169555664 time for calcul the mask position with numpy : 0.0012981891632080078 nb_pixel_total : 267 time to create 1 rle with old method : 0.0003249645233154297 time for calcul the mask position with numpy : 0.0013461112976074219 nb_pixel_total : 9079 time to create 1 rle with old method : 0.01010441780090332 time for calcul the mask position with numpy : 0.0013110637664794922 nb_pixel_total : 719 time to create 1 rle with old method : 0.0009520053863525391 time for calcul the mask position with numpy : 0.0015103816986083984 nb_pixel_total : 18467 time to create 1 rle with old method : 0.02066206932067871 time for calcul the mask position with numpy : 0.0014617443084716797 nb_pixel_total : 4198 time to create 1 rle with old method : 0.004887104034423828 time for calcul the mask position with numpy : 0.0013439655303955078 nb_pixel_total : 3223 time to create 1 rle with old method : 0.0037839412689208984 time for calcul the mask position with numpy : 0.0014390945434570312 nb_pixel_total : 615 time to create 1 rle with old method : 0.000797271728515625 time for calcul the mask position with numpy : 0.0014262199401855469 nb_pixel_total : 2244 time to create 1 rle with old method : 0.002851247787475586 time for calcul the mask position with numpy : 0.0014064311981201172 nb_pixel_total : 1121 time to create 1 rle with old method : 0.0013577938079833984 time for calcul the mask position with numpy : 0.0014290809631347656 nb_pixel_total : 1420 time to create 1 rle with old method : 0.0017404556274414062 time for calcul the mask position with numpy : 0.0014200210571289062 nb_pixel_total : 248 time to create 1 rle with old method : 0.000347137451171875 time for calcul the mask position with numpy : 0.0014157295227050781 nb_pixel_total : 221 time to create 1 rle with old method : 0.0003204345703125 time for calcul the mask position with numpy : 0.0014808177947998047 nb_pixel_total : 11064 time to create 1 rle with old method : 0.012415170669555664 time for calcul the mask position with numpy : 0.0014178752899169922 nb_pixel_total : 1500 time to create 1 rle with old method : 0.001791238784790039 time for calcul the mask position with numpy : 0.0014209747314453125 nb_pixel_total : 735 time to create 1 rle with old method : 0.0009648799896240234 time for calcul the mask position with numpy : 0.0013966560363769531 nb_pixel_total : 928 time to create 1 rle with old method : 0.001276254653930664 time for calcul the mask position with numpy : 0.001314401626586914 nb_pixel_total : 1523 time to create 1 rle with old method : 0.0018253326416015625 time for calcul the mask position with numpy : 0.001394033432006836 nb_pixel_total : 7504 time to create 1 rle with old method : 0.00844430923461914 time for calcul the mask position with numpy : 0.001360177993774414 nb_pixel_total : 299 time to create 1 rle with old method : 0.00044846534729003906 time for calcul the mask position with numpy : 0.0013430118560791016 nb_pixel_total : 889 time to create 1 rle with old method : 0.0010380744934082031 time for calcul the mask position with numpy : 0.0014331340789794922 nb_pixel_total : 885 time to create 1 rle with old method : 0.001093149185180664 time for calcul the mask position with numpy : 0.001355886459350586 nb_pixel_total : 947 time to create 1 rle with old method : 0.0012171268463134766 time for calcul the mask position with numpy : 0.0013747215270996094 nb_pixel_total : 1322 time to create 1 rle with old method : 0.0015921592712402344 time for calcul the mask position with numpy : 0.0014069080352783203 nb_pixel_total : 1615 time to create 1 rle with old method : 0.0018887519836425781 time for calcul the mask position with numpy : 0.001369476318359375 nb_pixel_total : 3252 time to create 1 rle with old method : 0.003711700439453125 time for calcul the mask position with numpy : 0.001417398452758789 nb_pixel_total : 1204 time to create 1 rle with old method : 0.0014505386352539062 time for calcul the mask position with numpy : 0.0013637542724609375 nb_pixel_total : 1434 time to create 1 rle with old method : 0.001683950424194336 time for calcul the mask position with numpy : 0.001291036605834961 nb_pixel_total : 875 time to create 1 rle with old method : 0.0010387897491455078 time for calcul the mask position with numpy : 0.0012907981872558594 nb_pixel_total : 436 time to create 1 rle with old method : 0.0005142688751220703 insert ignore into MTRPhoto.crop_hashtag_ids (photo_id, hashtag_id, `type`,x0,x1,y0,y1,score) VALUES (%s,%s,%s,%s,%s,%s,%s,%s) insert ignore into MTRPhoto.crop_hashtag_ids (photo_id, hashtag_id, `type`,x0,x1,y0,y1,score) VALUES (%s,%s,%s,%s,%s,%s,%s,%s) batch 1 Loaded 104 chid ids of type : 4677 Number RLEs to save : 9636 INSERT IGNORE INTO MTRPhoto.crop_segments (`crop_hashtag_id`, `x0`, `y0`, `length`) VALUES (%s, %s, %s , %s) first line : ('3973477186', '928', '287', '9') ... last line : ('3973477289', '500', '59', '5') INSERT IGNORE INTO MTRPhoto.crop_sum_segments (`crop_hashtag_id`, `sum_segments`) VALUES (%s, %s) TO DO : save crop sub photo not yet done ! After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : True verbose : True saveOutput not yet implemented for datou_step.type : sam we use saveGeneral [1189321094] map_info['map_portfolio_photo'] : {} final : True mtd_id 4573 list_pids : [1189321094] Looping around the photos to save general results len do output : 1 /1189321094Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4573', None, None, None, None, None, None, None, None) ('4573', None, '1189321094', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4573', None, '1189321094', 'None', None, None, None, None, None)] time used for this insertion : 0.015774965286254883 save_final save missing photos in datou_result : time spend for datou_step_exec : 12.784839391708374 time spend to save output : 0.01604437828063965 total time spend for step 1 : 12.800883769989014 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'1189321094': [[, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], 'temp/1758728011_1852108_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png']} nb_objects detect : 104 ############################### TEST frcnn ################################ test frcnn Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4184 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4184 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4184 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : frcnn list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.view_photos_with_urls_250912 ph WHERE photo_id IN (917754606) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917754606 download finish for photo 917754606 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.16548395156860352 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:frcnn Wed Sep 24 17:33:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1758728024_1852108_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606} map_photo_id_path_extension : {917754606: {'path': 'temp/1758728024_1852108_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Faster rcnn ! classes : ['background', 'plaque'] pht : 4370 caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'} To loadFromThcl() model_param file didn't exist model_name : detection_plaque_valcor_010622 model_type : caffe_faster_rcnn list file need : ['caffemodel', 'test.prototxt'] file exist in s3 : ['caffemodel', 'test.prototxt'] file manque in s3 : [] local folder : /data/models_weight/detection_plaque_valcor_010622 /data/models_weight/detection_plaque_valcor_010622/caffemodel size_local : 349723073 size in s3 : 349723073 create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:56 caffemodel already exist and didn't need to update /data/models_weight/detection_plaque_valcor_010622/test.prototxt size_local : 7163 size in s3 : 7163 create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:55 test.prototxt already exist and didn't need to update prototxt : /data/models_weight/detection_plaque_valcor_010622/test.prototxt caffemodel : /data/models_weight/detection_plaque_valcor_010622/caffemodel Loaded network /data/models_weight/detection_plaque_valcor_010622/caffemodel About to compute detect_faster_rcnn : len(args) : 1 Inside frcnn step exec : nb paths : 1 image_path : temp/1758728024_1852108_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg image_size (600, 800, 3) [[[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] [[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] [[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] ... [[ 14 16 16] [ 13 15 15] [ 11 13 13] ... [198 206 205] [198 206 205] [198 206 205]] [[ 16 18 18] [ 14 16 16] [ 11 13 13] ... [206 214 213] [206 214 213] [206 214 213]] [[ 13 15 15] [ 12 14 14] [ 9 11 11] ... [210 218 217] [210 218 217] [210 218 217]]] Detection took 0.071s for 300 object proposals c : plaque list_crops.shape (72, 5) proba : 0.06383844 (374.12637, 293.91925, 430.8101, 317.80865) proba : 0.052215073 (382.17734, 297.18808, 552.35864, 344.6578) proba : 0.012271571 (345.35635, 272.43146, 468.85782, 320.72552) We are managing local photo_id len de result frcnn : 1 After datou_step_exec type output : time spend for datou_step_exec : 2.4392876625061035 time spend to save output : 9.846687316894531e-05 total time spend for step 1 : 2.4393861293792725 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : True Inside saveFrcnn : final : True verbose : True threshold to save the result : 0.1 output flattener : [(0, 493029425, 4370, 374, 430, 293, 317, 0.06383844, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.052215073, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.012271571, None)] Warning : no hashtag_ids to insert in the database final : True begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4184', None, '917754606', '0', 0, '0', 493061979, '0', None)] time used for this insertion : 0.019793033599853516 [917754606] map_info['map_portfolio_photo'] : {} final : True mtd_id 4184 list_pids : [917754606] Looping around the photos to save general results len do output : 1 /0 before output type Managing all output in save final without adding information in the mtr_datou_result ('4184', None, None, None, None, None, None, None, None) ('4184', None, '917754606', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4184', None, '917754606', None, None, None, None, None, None)] time used for this insertion : 0.013567924499511719 save_final save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {0: [[(0, 493029425, 4370, 374, 430, 293, 317, 0.06383844, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.052215073, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.012271571, None)], 'temp/1758728024_1852108_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg']} ############################### TEST thcl ################################ TEST THCL Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=2 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=2 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 2 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=2 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 1 thcl is not linked in the step_by_step architecture ! WARNING : step 2 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : thcl, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.view_photos_with_urls_250912 ph WHERE photo_id IN (916235064) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 916235064 download finish for photo 916235064 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.09472298622131348 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:thcl Wed Sep 24 17:33:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1758728027_1852108_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064} map_photo_id_path_extension : {916235064: {'path': 'temp/1758728027_1852108_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Thcl ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'355': 1} we are using the classfication for only one thcl 355 In convert_file_to_np l 337 : 1 l343 1 l357 after caffe.io.load_image dimension du image : (3, (66, 66, 3)) dimension displayed ! time to import caffe and check if the image exist : 0.008603334426879883 time to convert the images to numpy array : 0.0032567977905273438 total time to convert the images to numpy array : 0.01212310791015625 list photo_ids error: [] list photo_ids correct : [916235064] number of photos to traite : 1 try to delete the photos incorrect in DB tagging for thcl : 355 To do loadFromThcl(), then load ParamDescType : thcl355 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (355) thcls : [{'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3390 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3390) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27)) To loadFromThcl() : net_3390 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1193 wait 20 seconds l 3637 free memory gpu now : 1193 max_wait_temp : 1 max_wait : 0 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3390) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27)) param : , param.caffemodel : car_360_1027 None mean_file_type : mean_file_path : prototxt_file_path : model : car_360_1027 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : car_360_1027 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : [] local folder : /data/models_weight/car_360_1027 /data/models_weight/car_360_1027/caffemodel size_local : 542944640 size in s3 : 542944640 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 caffemodel already exist and didn't need to update /data/models_weight/car_360_1027/deploy_conv_normal.prototxt size_local : 4626 size in s3 : 4626 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42 deploy_conv_normal.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/deploy_fc.prototxt size_local : 1132 size in s3 : 1132 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 deploy_fc.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/deploy.prototxt size_local : 5654 size in s3 : 5654 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42 deploy.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/mean.npy size_local : 1572944 size in s3 : 1572944 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:55 mean.npy already exist and didn't need to update /data/models_weight/car_360_1027/synset_words.txt size_local : 13687 size in s3 : 13687 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/:/home/admin/workarea/git/apy/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/car_360_1027/deploy.prototxt caffemodel_filename : /data/models_weight/car_360_1027/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1193 wait 20 seconds l 3637 free memory gpu now : 1193 max_wait_temp : 1 max_wait : 0 dict_keys(['prob', 'pool5']) time used to do the prepocess of the images : 0.014452934265136719 time used to do the prediction : 0.08703160285949707 save descriptor for thcl : 355 (1, 512, 7, 7) Got the blobs of the net to insert : [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] code_as_byte_string:b'0000000000'| time to traite the descriptors : 0.04630780220031738 Testing : ['916235064'] In select_photos_meta_from_ids: SELECT photo_id, url, FROM_UNIXTIME(uploaded_at), latitude, longitude, text FROM MTRBack.view_photos_with_urls_250912 WHERE photo_id IN (916235064) result : {916235064: {'photo_id': 916235064, 'url': 'https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/2017/10/14/6293d1bb790dc6902450e7c572b7d10b.jpg', 'latitude': None, 'longitude': None, 'text': None}} list_photo_exists : [916235064] storage_type for insertDescriptorsMulti : 1 To insert : 916235064 time to insert the descriptors : 0.509610652923584 After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : False verbose : True time used to find the portfolios of the photos select button_legend_list from MTRDatou.classification_theme where id = 355 SAVE THCL, output : {'916235064': [[('916235064', 'c_elysee_1027_gao__port_506302', 0.0018815113, 332, '355'), ('916235064', 'mokka_1027_gao__port_506374', 0.001163315, 332, '355'), ('916235064', 'captur_1027_gao__port_506399', 0.0008157107, 332, '355'), ('916235064', 'sorento_1027_gao__port_506192', 0.0011771553, 332, '355'), ('916235064', 'navara_1027_gao__port_506205', 0.0025850253, 332, '355'), ('916235064', 'xc90_1027_gao__port_506350', 0.004170366, 332, '355'), ('916235064', 'saxo_1027_gao__port_506052', 0.003480675, 332, '355'), ('916235064', 'trafic_1027_gao__port_506295', 0.0073670917, 332, '355'), ('916235064', 'punto_evo_1027_gao__port_506066', 0.0021886874, 332, '355'), ('916235064', '5_1027_gao__port_506117', 0.0005798119, 332, '355'), ('916235064', '250_1027_gao__port_506065', 0.0045901, 332, '355'), ('916235064', 'd_max_1027_gao__port_506125', 0.0031586709, 332, '355'), ('916235064', 'panamera_1027_gao__port_506387', 0.0022506546, 332, '355'), ('916235064', 'alhambra_1027_gao__port_506381', 0.005320306, 332, '355'), ('916235064', 'x6_1027_gao__port_506349', 0.0010997774, 332, '355'), ('916235064', 'vitara_1027_gao__port_506328', 0.0054021315, 332, '355'), ('916235064', 'fiesta_1027_gao__port_506377', 0.0039186855, 332, '355'), ('916235064', 'qashqai_1027_gao__port_506286', 0.0014787662, 332, '355'), ('916235064', '147_1027_gao__port_506124', 0.0019778444, 332, '355'), ('916235064', 'c5_1027_gao__port_506172', 0.0012440633, 332, '355'), ('916235064', 'q5_1027_gao__port_506206', 0.0015049413, 332, '355'), ('916235064', 'giulia_1027_gao__port_506178', 0.0021692866, 332, '355'), ('916235064', 'karl_1027_gao__port_506371', 0.0027078846, 332, '355'), ('916235064', 'mehari_1027_gao__port_506076', 0.004704202, 332, '355'), ('916235064', '911_1027_gao__port_506114', 0.0019418511, 332, '355'), ('916235064', '508_1027_gao__port_506329', 0.00095847726, 332, '355'), ('916235064', 'idea_1027_gao__port_506122', 0.0007699588, 332, '355'), ('916235064', 'megane_1027_gao__port_506220', 0.0019465545, 332, '355'), ('916235064', 'ghibli_1027_gao__port_506174', 0.0013724347, 332, '355'), ('916235064', 'touareg_1027_gao__port_506224', 0.00162028, 332, '355'), ('916235064', 'i10_1027_gao__port_506232', 0.001392329, 332, '355'), ('916235064', 'jumper_1027_gao__port_506234', 0.010045814, 332, '355'), ('916235064', 'classe_clk_1027_gao__port_506173', 0.0010792839, 332, '355'), ('916235064', 'kuga_1027_gao__port_506181', 0.0008446483, 332, '355'), ('916235064', 'ct_1027_gao__port_506323', 0.001251931, 332, '355'), ('916235064', 'leon_1027_gao__port_506326', 0.0025844302, 332, '355'), ('916235064', 'ds5_1027_gao__port_506376', 0.0012429103, 332, '355'), ('916235064', 'cordoba_1027_gao__port_506048', 0.0028650116, 332, '355'), ('916235064', 'classe_cla_1027_gao__port_506400', 0.0012947136, 332, '355'), ('916235064', 'jumpy_1027_gao__port_506179', 0.010339659, 332, '355'), ('916235064', 'avensis_1027_gao__port_506311', 0.001876482, 332, '355'), ('916235064', 'juke_1027_gao__port_506325', 0.0011343497, 332, '355'), ('916235064', '4008_1027_gao__port_506402', 0.0015758062, 332, '355'), ('916235064', '190_series_1027_gao__port_506051', 0.0039800694, 332, '355'), ('916235064', 'serie_3_1027_gao__port_506294', 0.0028737446, 332, '355'), ('916235064', 'q7_1027_gao__port_506318', 0.002335535, 332, '355'), ('916235064', 'glc_1027_gao__port_506303', 0.0012105925, 332, '355'), ('916235064', 'grand_vitara_1027_gao__port_506175', 0.001144549, 332, '355'), ('916235064', 's40_1027_gao__port_506099', 0.0022336806, 332, '355'), ('916235064', 'toledo_1027_gao__port_506061', 0.0017461816, 332, '355'), ('916235064', '5008_1027_gao__port_506337', 0.0046996567, 332, '355'), ('916235064', 'continental_1027_gao__port_506250', 0.0021912383, 332, '355'), ('916235064', 'coupe_1027_gao__port_506082', 0.002263005, 332, '355'), ('916235064', 'iq_1027_gao__port_506166', 0.0018171945, 332, '355'), ('916235064', '407_1027_gao__port_506133', 0.00090562593, 332, '355'), ('916235064', 'touran_1027_gao__port_506308', 0.0020402032, 332, '355'), ('916235064', '300c_1027_gao__port_506078', 0.0025333834, 332, '355'), ('916235064', 'classe_gl_1027_gao__port_506340', 0.0044887974, 332, '355'), ('916235064', 'vivaro_1027_gao__port_506310', 0.0034255348, 332, '355'), ('916235064', 'sl_1027_gao__port_506100', 0.0031349934, 332, '355'), ('916235064', 'elise_1027_gao__port_506121', 0.001025572, 332, '355'), ('916235064', '1007_1027_gao__port_506070', 0.0015353785, 332, '355'), ('916235064', 'i40_1027_gao__port_506218', 0.00059140875, 332, '355'), ('916235064', 'bipper_tepee_1027_gao__port_506227', 0.0040296856, 332, '355'), ('916235064', 'focus_1027_gao__port_506272', 0.0011585873, 332, '355'), ('916235064', 'primera_1027_gao__port_506147', 0.0012155806, 332, '355'), ('916235064', 'r4_1027_gao__port_506160', 0.014970172, 332, '355'), ('916235064', 'a8_1027_gao__port_506265', 0.0011319832, 332, '355'), ('916235064', 'boxer_1027_gao__port_506202', 0.010546597, 332, '355'), ('916235064', 's5_1027_gao__port_506222', 0.0011984826, 332, '355'), ('916235064', 'r21_1027_gao__port_506093', 0.004185648, 332, '355'), ('916235064', 'c3_1027_gao__port_506257', 0.002363495, 332, '355'), ('916235064', 'santa_fe_1027_gao__port_506208', 0.001632363, 332, '355'), ('916235064', 'm4_1027_gao__port_506344', 0.0015566221, 332, '355'), ('916235064', 'safrane_1027_gao__port_506077', 0.0013958625, 332, '355'), ('916235064', 'classe_gle_1027_gao__port_506395', 0.002197566, 332, '355'), ('916235064', '0_1027_gao__port_506094', 0.008827799, 332, '355'), ('916235064', 'ix35_1027_gao__port_506219', 0.0014613883, 332, '355'), ('916235064', 'carens_1027_gao__port_506298', 0.0008823778, 332, '355'), ('916235064', 'classe_a_1027_gao__port_506339', 0.0024708943, 332, '355'), ('916235064', 'ix20_1027_gao__port_506343', 0.001009129, 332, '355'), ('916235064', 'note_1027_gao__port_506365', 0.0015961987, 332, '355'), ('916235064', 'a5_1027_gao__port_506200', 0.0015329652, 332, '355'), ('916235064', 'sx4_1027_gao__port_506348', 0.0014915848, 332, '355'), ('916235064', 'sandero_1027_gao__port_506198', 0.001458451, 332, '355'), ('916235064', '3008_1027_gao__port_506385', 0.0056456886, 332, '355'), ('916235064', 'q50_1027_gao__port_506239', 0.0011163959, 332, '355'), ('916235064', 'latitude_1027_gao__port_506236', 0.0008018479, 332, '355'), ('916235064', 'v40_1027_gao__port_506391', 0.0017144355, 332, '355'), ('916235064', 'xsara_1027_gao__port_506087', 0.000982188, 332, '355'), ('916235064', 'grand_c_max_1027_gao__port_506342', 0.00179581, 332, '355'), ('916235064', 'swift_1027_gao__port_506149', 0.0015019185, 332, '355'), ('916235064', 'serie_1_1027_gao__port_506184', 0.0015136311, 332, '355'), ('916235064', 'xc70_1027_gao__port_506393', 0.0036190618, 332, '355'), ('916235064', 'master_1027_gao__port_506203', 0.007958686, 332, '355'), ('916235064', 'clio_1027_gao__port_506280', 0.002957021, 332, '355'), ('916235064', 'duster_1027_gao__port_506216', 0.0007443225, 332, '355'), ('916235064', 'traveller_1027_gao__port_506403', 0.0042941817, 332, '355'), ('916235064', 'tipo_1027_gao__port_506355', 0.0010929469, 332, '355'), ('916235064', 'rav_4_1027_gao__port_506332', 0.0013603647, 332, '355'), ('916235064', 'coccinelle_1027_gao__port_506259', 0.0034946012, 332, '355'), ('916235064', 'spacetourer_1027_gao__port_506401', 0.0030973898, 332, '355'), ('916235064', 'xe_1027_gao__port_506357', 0.0014469676, 332, '355'), ('916235064', 'ds3_1027_gao__port_506324', 0.0013092798, 332, '355'), ('916235064', 'mx_5_1027_gao__port_506098', 0.0025886078, 332, '355'), ('916235064', 'land_cruiser_1027_gao__port_506315', 0.009531982, 332, '355'), ('916235064', 'classe_b_1027_gao__port_506335', 0.001721351, 332, '355'), ('916235064', '806_1027_gao__port_506088', 0.0025618467, 332, '355'), ('916235064', 'rx_8_1027_gao__port_506046', 0.0036217824, 332, '355'), ('916235064', 'spark_1027_gao__port_506185', 0.0010076317, 332, '355'), ('916235064', '6_1027_gao__port_506171', 0.0011181668, 332, '355'), ('916235064', 'bravo_1027_gao__port_506080', 0.001464951, 332, '355'), ('916235064', 'nx_1027_gao__port_506345', 0.0013683282, 332, '355'), ('916235064', 'sharan_1027_gao__port_506347', 0.00509208, 332, '355'), ('916235064', 'x_type_1027_gao__port_506067', 0.0007801731, 332, '355'), ('916235064', 'jimny_1027_gao__port_506233', 0.0046061235, 332, '355'), ('916235064', 'wrangler_1027_gao__port_506225', 0.0017998074, 332, '355'), ('916235064', 'c_crosser_1027_gao__port_506312', 0.0015927221, 332, '355'), ('916235064', 'v70_1027_gao__port_506278', 0.0019676113, 332, '355'), ('916235064', 'classe_e_1027_gao__port_506300', 0.001736591, 332, '355'), ('916235064', 'classe_v_1027_gao__port_506258', 0.0127316965, 332, '355'), ('916235064', 'm3_1027_gao__port_506182', 0.0023369147, 332, '355'), ('916235064', 'abarth_500_1027_gao__port_506226', 0.0040435684, 332, '355'), ('916235064', 'serie_6_1027_gao__port_506262', 0.0011313177, 332, '355'), ('916235064', 'modus_1027_gao__port_506146', 0.0018292166, 332, '355'), ('916235064', '3_1027_gao__port_506113', 0.0015081387, 332, '355'), ('916235064', '405_1027_gao__port_506108', 0.00805487, 332, '355'), ('916235064', 'allroad_1027_gao__port_506297', 0.0010597152, 332, '355'), ('916235064', 'auris_1027_gao__port_506322', 0.0011522896, 332, '355'), ('916235064', 'galaxy_1027_gao__port_506143', 0.0025148282, 332, '355'), ('916235064', 'giulietta_1027_gao__port_506363', 0.00086508103, 332, '355'), ('916235064', '106_1027_gao__port_506073', 0.008270536, 332, '355'), ('916235064', 'classe_m_1027_gao__port_506154', 0.0030011775, 332, '355'), ('916235064', 'espace_1027_gao__port_506313', 0.0010646643, 332, '355'), ('916235064', 'panda_1027_gao__port_506189', 0.009030673, 332, '355'), ('916235064', 'rcz_1027_gao__port_506197', 0.0011293639, 332, '355'), ('916235064', '4007_1027_gao__port_506162', 0.0006791801, 332, '355'), ('916235064', 'classe_cl_1027_gao__port_506249', 0.0010859603, 332, '355'), ('916235064', 'leaf_1027_gao__port_506139', 0.0018037482, 332, '355'), ('916235064', 'octavia_1027_gao__port_506237', 0.0018598126, 332, '355'), ('916235064', 'ds4_1027_gao__port_506336', 0.002416086, 332, '355'), ('916235064', 'freelander_1027_gao__port_506084', 0.0023472367, 332, '355'), ('916235064', 'evasion_1027_gao__port_506109', 0.0031141187, 332, '355'), ('916235064', 'punto_1027_gao__port_506106', 0.0019494438, 332, '355'), ('916235064', '2cv_1027_gao__port_506045', 0.007974622, 332, '355'), ('916235064', 'x4_1027_gao__port_506392', 0.001794661, 332, '355'), ('916235064', 'antara_1027_gao__port_506247', 0.0012466817, 332, '355'), ('916235064', 'murano_1027_gao__port_506316', 0.0006087389, 332, '355'), ('916235064', 'alto_1027_gao__port_506201', 0.009231275, 332, '355'), ('916235064', 'meriva_1027_gao__port_506353', 0.001376269, 332, '355'), ('916235064', 'orlando_1027_gao__port_506305', 0.0018459444, 332, '355'), ('916235064', 'new_beetle_1027_gao__port_506050', 0.0011638122, 332, '355'), ('916235064', '306_1027_gao__port_506145', 0.0035072158, 332, '355'), ('916235064', 'tiguan_1027_gao__port_506362', 0.0026824663, 332, '355'), ('916235064', 's_type_1027_gao__port_506101', 0.0011381136, 332, '355'), ('916235064', 'c1_1027_gao__port_506128', 0.0027515334, 332, '355'), ('916235064', 'vectra_1027_gao__port_506044', 0.0011988218, 332, '355'), ('916235064', 'outlander_1027_gao__port_506317', 0.0017120439, 332, '355'), ('916235064', '307_1027_gao__port_506074', 0.0020010874, 332, '355'), ('916235064', 'a6_s6_1027_gao__port_506134', 0.001657022, 332, '355'), ('916235064', 'nemo_combi_1027_gao__port_506196', 0.0022665027, 332, '355'), ('916235064', 'berlingo_1027_gao__port_506194', 0.004662823, 332, '355'), ('916235064', 'partner_1027_gao__port_506285', 0.0039414796, 332, '355'), ('916235064', 'cayenne_1027_gao__port_506177', 0.003798104, 332, '355'), ('916235064', 'quattroporte_1027_gao__port_506240', 0.0024437362, 332, '355'), ('916235064', 'c_max_1027_gao__port_506282', 0.0013123387, 332, '355'), ('916235064', 'fabia_1027_gao__port_506396', 0.0052983263, 332, '355'), ('916235064', 'cx_3_1027_gao__port_506281', 0.0014460731, 332, '355'), ('916235064', 'x_trail_1027_gao__port_506264', 0.0018314845, 332, '355'), ('916235064', 'scirocco_1027_gao__port_506276', 0.004790927, 332, '355'), ('916235064', 'matiz_1027_gao__port_506144', 0.0017559973, 332, '355'), ('916235064', 'tigra_1027_gao__port_506069', 0.00085421326, 332, '355'), ('916235064', 'escort_1027_gao__port_506091', 0.0048401495, 332, '355'), ('916235064', 'c2_1027_gao__port_506081', 0.001490417, 332, '355'), ('916235064', 'mini_1027_gao__port_506168', 0.0011922222, 332, '355'), ('916235064', 'i30_1027_gao__port_506291', 0.00063249486, 332, '355'), ('916235064', 'picanto_1027_gao__port_506238', 0.002974838, 332, '355'), ('916235064', 'mito_1027_gao__port_506072', 0.001507763, 332, '355'), ('916235064', 'impreza_1027_gao__port_506085', 0.0020191132, 332, '355'), ('916235064', 'kangoo_1027_gao__port_506235', 0.0065845395, 332, '355'), ('916235064', 'a4_1027_gao__port_506193', 0.001987416, 332, '355'), ('916235064', 'cayman_1027_gao__port_506268', 0.0018138823, 332, '355'), ('916235064', 'sportage_1027_gao__port_506148', 0.0014274553, 332, '355'), ('916235064', 'up_1027_gao__port_506356', 0.0068631433, 332, '355'), ('916235064', 'optima_1027_gao__port_506386', 0.00089178275, 332, '355'), ('916235064', 'defender_1027_gao__port_506229', 0.006722822, 332, '355'), ('916235064', 'serie_2_1027_gao__port_506256', 0.0022272475, 332, '355'), ('916235064', 'edge_1027_gao__port_506187', 0.00087478023, 332, '355'), ('916235064', 'r19_1027_gao__port_506110', 0.004942286, 332, '355'), ('916235064', 'jetta_1027_gao__port_506304', 0.0036189714, 332, '355'), ('916235064', 'eos_1027_gao__port_506115', 0.0038925689, 332, '355'), ('916235064', 'accord_1027_gao__port_506214', 0.0020124852, 332, '355'), ('916235064', 'yaris_1027_gao__port_506334', 0.003233201, 332, '355'), ('916235064', 'classe_cls_1027_gao__port_506289', 0.00078506785, 332, '355'), ('916235064', 'polo_1027_gao__port_506361', 0.004310988, 332, '355'), ('916235064', 'serie_4_1027_gao__port_506366', 0.0011473801, 332, '355'), ('916235064', 'mini_cabriolet_1027_gao__port_506204', 0.0008377641, 332, '355'), ('916235064', 'prius_1027_gao__port_506190', 0.0011488148, 332, '355'), ('916235064', 'lodgy_1027_gao__port_506188', 0.0020174636, 332, '355'), ('916235064', 'serie_7_1027_gao__port_506307', 0.001247414, 332, '355'), ('916235064', 'c15_1027_gao__port_506055', 0.017714119, 332, '355'), ('916235064', 'kadjar_1027_gao__port_506389', 0.0012504567, 332, '355'), ('916235064', 'insignia_1027_gao__port_506364', 0.0016433158, 332, '355'), ('916235064', '308_1027_gao__port_506279', 0.0021235293, 332, '355'), ('916235064', 'roomster_1027_gao__port_506241', 0.001801226, 332, '355'), ('916235064', '80_1027_gao__port_506057', 0.0046132905, 332, '355'), ('916235064', '309_1027_gao__port_506063', 0.013525877, 332, '355'), ('916235064', 'tucson_1027_gao__port_506320', 0.0021234024, 332, '355'), ('916235064', 'x3_1027_gao__port_506212', 0.0008975628, 332, '355'), ('916235064', 'xf_1027_gao__port_506263', 0.001116558, 332, '355'), ('916235064', '2008_1027_gao__port_506394', 0.002640831, 332, '355'), ('916235064', 'passat_1027_gao__port_506306', 0.001497497, 332, '355'), ('916235064', 'compass_1027_gao__port_506260', 0.00325586, 332, '355'), ('916235064', 'twingo_1027_gao__port_506309', 0.006499338, 332, '355'), ('916235064', 'micra_1027_gao__port_506221', 0.00358582, 332, '355'), ('916235064', 'golf_1027_gao__port_506155', 0.0031929978, 332, '355'), ('916235064', 'soul_1027_gao__port_506176', 0.0012889532, 332, '355'), ('916235064', 'rapid_1027_gao__port_506398', 0.0025910158, 332, '355'), ('916235064', 'forester_1027_gao__port_506360', 0.0022760583, 332, '355'), ('916235064', 'slk_1027_gao__port_506210', 0.0015843246, 332, '355'), ('916235064', 'forfour_1027_gao__port_506341', 0.0021795928, 332, '355'), ('916235064', 'serie_5_1027_gao__port_506209', 0.0013693492, 332, '355'), ('916235064', 'xj_1027_gao__port_506170', 0.0026005765, 332, '355'), ('916235064', 'pajero_1027_gao__port_506097', 0.0052117445, 332, '355'), ('916235064', 'agila_1027_gao__port_506119', 0.004850258, 332, '355'), ('916235064', 'a6_1027_gao__port_506163', 0.0018910553, 332, '355'), ('916235064', 'fox_1027_gao__port_506092', 0.0008465188, 332, '355'), ('916235064', 'boxster_1027_gao__port_506267', 0.0015941885, 332, '355'), ('916235064', 'altea_1027_gao__port_506246', 0.002139136, 332, '355'), ('916235064', 'samurai_1027_gao__port_506047', 0.0062523372, 332, '355'), ('916235064', 'trax_1027_gao__port_506296', 0.0019437243, 332, '355'), ('916235064', 'getz_1027_gao__port_506058', 0.0016383658, 332, '355'), ('916235064', 'cherokee_1027_gao__port_506269', 0.0029809233, 332, '355'), ('916235064', 'koleos_1027_gao__port_506378', 0.0015590639, 332, '355'), ('916235064', 'z_series_1027_gao__port_506123', 0.001656101, 332, '355'), ('916235064', 'ecosport_1027_gao__port_506271', 0.0013229665, 332, '355'), ('916235064', 'space_star_1027_gao__port_506277', 0.0021144636, 332, '355'), ('916235064', 'rs3_sportback_1027_gao__port_506207', 0.001911592, 332, '355'), ('916235064', 'civic_1027_gao__port_506141', 0.0026896861, 332, '355'), ('916235064', 'talisman_1027_gao__port_506390', 0.0007612434, 332, '355'), ('916235064', 'f_pace_1027_gao__port_506314', 0.0016163907, 332, '355'), ('916235064', 'classe_c_1027_gao__port_506299', 0.0017938546, 332, '355'), ('916235064', 'tt_1027_gao__port_506075', 0.001393407, 332, '355'), ('916235064', 'pathfinder_1027_gao__port_506183', 0.0016514121, 332, '355'), ('916235064', '156_1027_gao__port_506157', 0.0015443647, 332, '355'), ('916235064', 'cx_5_1027_gao__port_506228', 0.0014412818, 332, '355'), ('916235064', 'scenic_1027_gao__port_506255', 0.0016083465, 332, '355'), ('916235064', 'yeti_1027_gao__port_506358', 0.0020913114, 332, '355'), ('916235064', 'mustang_1027_gao__port_506053', 0.010050809, 332, '355'), ('916235064', 'stilo_1027_gao__port_506060', 0.0010830021, 332, '355'), ('916235064', 'ateca_1027_gao__port_506382', 0.0017009925, 332, '355'), ('916235064', 'fiorino_1027_gao__port_506217', 0.009201011, 332, '355'), ('916235064', 'classe_glk_1027_gao__port_506290', 0.0017014625, 332, '355'), ('916235064', 'fortwo_1027_gao__port_506230', 0.0016010845, 332, '355'), ('916235064', 'cruze_1027_gao__port_506186', 0.0010051578, 332, '355'), ('916235064', '107_1027_gao__port_506213', 0.0016275262, 332, '355'), ('916235064', 'aygo_1027_gao__port_506248', 0.0032430778, 332, '355'), ('916235064', 'rx_1027_gao__port_506354', 0.0010631259, 332, '355'), ('916235064', '500_1027_gao__port_506245', 0.0016354368, 332, '355'), ('916235064', 'bora_1027_gao__port_506104', 0.0038158146, 332, '355'), ('916235064', 'transit_1027_gao__port_506111', 0.0048626335, 332, '355'), ('916235064', 'pt_cruiser_1027_gao__port_506054', 0.0019164268, 332, '355'), ('916235064', 'patrol_1027_gao__port_506068', 0.004240515, 332, '355'), ('916235064', 'r8_1027_gao__port_506156', 0.0012721621, 332, '355'), ('916235064', 'xm_1027_gao__port_506102', 0.0022678368, 332, '355'), ('916235064', 's60_1027_gao__port_506191', 0.0031988972, 332, '355'), ('916235064', 'aveo_1027_gao__port_506158', 0.0038400367, 332, '355'), ('916235064', 'captiva_1027_gao__port_506159', 0.0017187846, 332, '355'), ('916235064', 'ax_1027_gao__port_506153', 0.006896619, 332, '355'), ('916235064', 'rexton_1027_gao__port_506107', 0.0013019813, 332, '355'), ('916235064', 'camaro_1027_gao__port_506056', 0.002490167, 332, '355'), ('916235064', 'ypsilon_1027_gao__port_506131', 0.0019541553, 332, '355'), ('916235064', 'delta_1027_gao__port_506165', 0.0014001308, 332, '355'), ('916235064', 'c4_1027_gao__port_506370', 0.0013008481, 332, '355'), ('916235064', 'zx_1027_gao__port_506161', 0.0045936536, 332, '355'), ('916235064', 'verso_1027_gao__port_506242', 0.00077202497, 332, '355'), ('916235064', 'superb_1027_gao__port_506327', 0.0019938825, 332, '355'), ('916235064', 'r5_1027_gao__port_506253', 0.009546243, 332, '355'), ('916235064', 'caddy_1027_gao__port_506330', 0.013828455, 332, '355'), ('916235064', 'x5_1027_gao__port_506243', 0.0011202438, 332, '355'), ('916235064', 'f_type_1027_gao__port_506231', 0.00082985015, 332, '355'), ('916235064', 'fusion_1027_gao__port_506096', 0.0012670782, 332, '355'), ('916235064', 'dokker_1027_gao__port_506331', 0.005358085, 332, '355'), ('916235064', '205_1027_gao__port_506062', 0.00668516, 332, '355'), ('916235064', 'macan_1027_gao__port_506195', 0.0015572328, 332, '355'), ('916235064', 'tourneo_1027_gao__port_506369', 0.006401825, 332, '355'), ('916235064', '108_1027_gao__port_506384', 0.0052638147, 332, '355'), ('916235064', '9_3_1027_gao__port_506071', 0.0008373029, 332, '355'), ('916235064', 'mondeo_1027_gao__port_506116', 0.0014394878, 332, '355'), ('916235064', 'cr_v_1027_gao__port_506164', 0.0016412373, 332, '355'), ('916235064', 'c30_1027_gao__port_506090', 0.0017486004, 332, '355'), ('916235064', 'pulsar_1027_gao__port_506397', 0.0012018785, 332, '355'), ('916235064', 'ibiza_1027_gao__port_506273', 0.0037229613, 332, '355'), ('916235064', 'a1_1027_gao__port_506338', 0.001234698, 332, '355'), ('916235064', 'matrix_1027_gao__port_506140', 0.00070775097, 332, '355'), ('916235064', 'carnival_1027_gao__port_506136', 0.0022811922, 332, '355'), ('916235064', 'xantia_1027_gao__port_506086', 0.0021959785, 332, '355'), ('916235064', 'terrano_1027_gao__port_506083', 0.0020294364, 332, '355'), ('916235064', 'q3_1027_gao__port_506275', 0.0011264263, 332, '355'), ('916235064', 'hr_v_1027_gao__port_506283', 0.0017806772, 332, '355'), ('916235064', 'expert_1027_gao__port_506142', 0.00737015, 332, '355'), ('916235064', 'multivan_1027_gao__port_506383', 0.0065052267, 332, '355'), ('916235064', 'venga_1027_gao__port_506380', 0.0008003821, 332, '355'), ('916235064', 'scudo_1027_gao__port_506129', 0.005593294, 332, '355'), ('916235064', 'laguna_1027_gao__port_506368', 0.00071336137, 332, '355'), ('916235064', 'vel_satis_1027_gao__port_506130', 0.0027261726, 332, '355'), ('916235064', 'b_max_1027_gao__port_506367', 0.0017246526, 332, '355'), ('916235064', 'ignis_1027_gao__port_506292', 0.0043560043, 332, '355'), ('916235064', '159_1027_gao__port_506064', 0.0010782299, 332, '355'), ('916235064', 'grande_punto_1027_gao__port_506138', 0.0023635176, 332, '355'), ('916235064', 'logan_1027_gao__port_506167', 0.004396976, 332, '355'), ('916235064', 's_max_1027_gao__port_506223', 0.0012526179, 332, '355'), ('916235064', 'caravelle_1027_gao__port_506351', 0.0030299171, 332, '355'), ('916235064', 'adam_1027_gao__port_506079', 0.0010538028, 332, '355'), ('916235064', '406_1027_gao__port_506132', 0.0013574676, 332, '355'), ('916235064', 'q30_1027_gao__port_506293', 0.0009714442, 332, '355'), ('916235064', 'almera_1027_gao__port_506089', 0.0010238797, 332, '355'), ('916235064', 'corsa_1027_gao__port_506095', 0.0025201861, 332, '355'), ('916235064', 'corolla_1027_gao__port_506120', 0.0026817385, 332, '355'), ('916235064', 'xc60_1027_gao__port_506388', 0.0018983919, 332, '355'), ('916235064', 'viano_1027_gao__port_506211', 0.0026945586, 332, '355'), ('916235064', 'pro_cee_d_1027_gao__port_506274', 0.00083197013, 332, '355'), ('916235064', 'a3_1027_gao__port_506321', 0.0037379174, 332, '355'), ('916235064', 'v50_1027_gao__port_506150', 0.0007919513, 332, '355'), ('916235064', 'voyager_1027_gao__port_506169', 0.0030527643, 332, '355'), ('916235064', 'corvette_1027_gao__port_506049', 0.0037229941, 332, '355'), ('916235064', 'rio_1027_gao__port_506379', 0.0017739858, 332, '355'), ('916235064', 'jazz_1027_gao__port_506252', 0.0015302984, 332, '355'), ('916235064', '200_1027_gao__port_506112', 0.0040867026, 332, '355'), ('916235064', 'tts_1027_gao__port_506199', 0.0011861655, 332, '355'), ('916235064', 'zafira_1027_gao__port_506287', 0.0026948329, 332, '355'), ('916235064', 'asx_1027_gao__port_506266', 0.0011406268, 332, '355'), ('916235064', '607_1027_gao__port_506118', 0.0012527801, 332, '355'), ('916235064', '207_1027_gao__port_506103', 0.0015146787, 332, '355'), ('916235064', 'classe_s_1027_gao__port_506301', 0.0031650946, 332, '355'), ('916235064', 'c6_1027_gao__port_506105', 0.0017347817, 332, '355'), ('916235064', 'express_1027_gao__port_506137', 0.01672994, 332, '355'), ('916235064', 'classe_gla_1027_gao__port_506352', 0.0018252939, 332, '355'), ('916235064', 'v60_1027_gao__port_506333', 0.0021457395, 332, '355'), ('916235064', 'ka_1027_gao__port_506180', 0.0014151764, 332, '355'), ('916235064', 'range_rover_1027_gao__port_506254', 0.0020551113, 332, '355'), ('916235064', 'discovery_1027_gao__port_506375', 0.0022962461, 332, '355'), ('916235064', 'classe_r_1027_gao__port_506270', 0.0013941999, 332, '355'), ('916235064', 'transporter_1027_gao__port_506319', 0.011970102, 332, '355'), ('916235064', 'cee_d_1027_gao__port_506288', 0.0010547399, 332, '355'), ('916235064', 'zoe_1027_gao__port_506244', 0.002071428, 332, '355'), ('916235064', 'i20_1027_gao__port_506284', 0.001786827, 332, '355'), ('916235064', 'gtv_1027_gao__port_506059', 0.005722152, 332, '355'), ('916235064', 's4_avant_1027_gao__port_506261', 0.0027664078, 332, '355'), ('916235064', 'x1_1027_gao__port_506372', 0.0017141928, 332, '355'), ('916235064', 'autres_1027_gao__port_506127', 0.0048254435, 332, '355'), ('916235064', '208_1027_gao__port_506359', 0.0018685928, 332, '355'), ('916235064', 'c8_1027_gao__port_506135', 0.0012580492, 332, '355'), ('916235064', 'astra_1027_gao__port_506215', 0.0012623819, 332, '355'), ('916235064', '2_1027_gao__port_506151', 0.00092438277, 332, '355'), ('916235064', 'doblo_1027_gao__port_506251', 0.007466892, 332, '355'), ('916235064', '807_1027_gao__port_506152', 0.00072904193, 332, '355'), ('916235064', '206_1027_gao__port_506126', 0.0010385595, 332, '355'), ('916235064', 'a7_1027_gao__port_506373', 0.00069106463, 332, '355'), ('916235064', 'renegade_1027_gao__port_506346', 0.002141543, 332, '355')]]} begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score) time used for this insertion : 5.245208740234375e-06 save missing photos in datou_result : time spend for datou_step_exec : 46.26535749435425 time spend to save output : 2.1167755126953125 total time spend for step 1 : 48.38213300704956 step2:argmax Wed Sep 24 17:34:35 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1758728027_1852108_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064} map_photo_id_path_extension : {916235064: {'path': 'temp/1758728027_1852108_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step Argmax ! calculate argmax for thcl : 355 After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : True verbose : True photo_id : 916235064 output[photo_id] : [('916235064', 'c15_1027_gao__port_506055', 0.017714119, 332, '355'), 'temp/1758728027_1852108_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg'] begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 1 insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s) insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s) first line : ('916235064', '2049863950', '332') ... last line : ('916235064', '2049863950', '332') time used for this insertion : 0.022987842559814453 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 1 insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score) time used for this insertion : 0.019005775451660156 len list_finale : 1, len picture : 1 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('2', None, '916235064', 'c15_1027_gao__port_506055', None, None, '2049863950', '0.017714119', None)] time used for this insertion : 0.014950990676879883 saving photo_ids in datou_result photo id not in port begin to insert list_values into mtr_datou_result : length of list_values in save_final : 0 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [] time used for this insertion : 3.5762786865234375e-06 save missing photos in datou_result : time spend for datou_step_exec : 0.003474712371826172 time spend to save output : 0.05725288391113281 total time spend for step 2 : 0.060727596282958984 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : {'916235064': [('916235064', 'c15_1027_gao__port_506055', 0.017714119, 332, '355'), 'temp/1758728027_1852108_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg']} ############################### TEST tfhub2 ################################ TEST TFHUB2 ######################## test with use_multi_inputs=0 ######################## Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4567 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4567 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4567 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4567 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 12835 tfhub_classification2 is not linked in the step_by_step architecture ! WARNING : step 12836 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.view_photos_with_urls_250912 ph WHERE photo_id IN (1171252784,1171252764,1171252487) Found this number of photos: 3 ##### Call download_photos : nb_thread : 5 begin to download photo : 1171252487 begin to download photo : 1171252764 begin to download photo : 1171252784 download finish for photo 1171252784 download finish for photo 1171252487 download finish for photo 1171252764 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 ##### After load_data_input time to download the photos : 0.19295215606689453 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:tfhub_classification2 Wed Sep 24 17:34:36 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1758728076_1852108_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg': 1171252784, 'temp/1758728076_1852108_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg': 1171252487, 'temp/1758728076_1852108_1171252764_29d5179a892cc50aadc9d67245534b59.jpg': 1171252764} map_photo_id_path_extension : {1171252784: {'path': 'temp/1758728076_1852108_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg', 'extension': 'jpg'}, 1171252487: {'path': 'temp/1758728076_1852108_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg', 'extension': 'jpg'}, 1171252764: {'path': 'temp/1758728076_1852108_1171252764_29d5179a892cc50aadc9d67245534b59.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step TFHub with tf2 ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'3609': 1} we are using the classfication for only one thcl 3609 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1193 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1193 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1193 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1193 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1193 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1193 wait 20 seconds 2025-09-24 17:36:45.731108: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-09-24 17:36:45.732368: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-24 17:36:45.732465: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-24 17:36:45.732523: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-24 17:36:45.751281: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-24 17:36:45.751408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-24 17:36:45.782147: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-24 17:36:45.786295: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-24 17:36:45.839895: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-24 17:36:45.840913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-24 17:36:45.842048: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-09-24 17:36:45.880703: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-09-24 17:36:45.882254: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fadf8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-09-24 17:36:45.882292: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-09-24 17:36:45.886440: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xb038d2e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-09-24 17:36:45.886463: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-09-24 17:36:45.887533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-24 17:36:45.887647: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-24 17:36:45.887668: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-24 17:36:45.887742: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-24 17:36:45.887771: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-24 17:36:45.887806: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-24 17:36:45.887844: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-24 17:36:45.887883: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-24 17:36:45.888755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-24 17:36:45.889179: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-24 17:36:45.889234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-24 17:36:45.889246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-24 17:36:45.889255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-24 17:36:45.890547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 1193 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3609 To do loadFromThcl(), then load ParamDescType : thcl3609 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (3609) thcls : [{'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'}] thcl {'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'} Update svm_hashtag_type_desc : 5832 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (5832) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5832, 'tfhub_19_06_2023', 1280, 1280, 'tfhub_19_06_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 6, 19, 12, 55, 22), datetime.datetime(2023, 6, 19, 12, 55, 22)) model_name : tfhub_19_06_2023 model_param file didn't exist model_name : tfhub_19_06_2023 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-09-24 17:36:52.980719: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-24 17:36:52.981330: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-24 17:36:52.981901: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2629576704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-24 17:36:52.982393: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.20G (2366618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-24 17:36:52.982883: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.98G (2129957120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-24 17:36:52.983373: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.79G (1916961536 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-24 17:36:52.983860: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.61G (1725265408 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-24 17:36:52.984373: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.45G (1552738816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-24 17:36:52.984912: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.30G (1397465088 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-24 17:36:52.985460: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.17G (1257718528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/tfhub_19_06_2023 /data/models_weight/tfhub_19_06_2023/Confusion_Matrix.png size_local : 57753 size in s3 : 57753 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_jrm.jpg size_local : 79724 size in s3 : 79724 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_jrm.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pcm.jpg size_local : 83556 size in s3 : 83556 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15 Precision_Recall_pcm.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pcnc.jpg size_local : 74107 size in s3 : 74107 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_pcnc.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pehd.jpg size_local : 72705 size in s3 : 72705 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_pehd.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_tapis_vide.jpg size_local : 70874 size in s3 : 70874 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:15 Precision_Recall_tapis_vide.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Result_Summary.txt size_local : 642 size in s3 : 642 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 Result_Summary.txt already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 checkpoint already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216488 size in s3 : 216488 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:21 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.index size_local : 43546 size in s3 : 43546 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:22 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_weights.h5 size_local : 16499144 size in s3 : 16499144 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:15 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2333, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2527, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3139, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 77, in create_tfhub_model hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module"), File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype gen_logging_ops._assert( # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1171252784, 1171252487, 1171252764] map_info['map_portfolio_photo'] : {} final : True mtd_id 4567 list_pids : [1171252784, 1171252487, 1171252764] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4567', None, '1171252784', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252487', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252764', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.02273273468017578 save_final ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse time spend for datou_step_exec : 137.76571583747864 time spend to save output : 0.02882528305053711 total time spend for step 0 : 137.79454112052917 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : None probably due to empty image bug ERROR expected : {'1171252784': [(1171252784, 'jrm', 0.9677492, 4674, '3609'), 'temp/1687511175_1882837_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg'], '1171252764': [(1171252764, 'jrm', 0.9853587, 4674, '3609'), 'temp/1687511175_1882837_1171252764_29d5179a892cc50aadc9d67245534b59.jpg'], '1171252487': [(1171252487, 'jrm', 0.9262757, 4674, '3609'), 'temp/1687511175_1882837_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg']} got : None ######################## test with use_multi_inputs=1 ######################## Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4621 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4621 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4621 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4621 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 12927 tfhub_classification2 is not linked in the step_by_step architecture ! WARNING : step 12928 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.view_photos_with_urls_250912 ph WHERE photo_id IN (1171291875,1171275372,1171275314) Found this number of photos: 3 ##### Call download_photos : nb_thread : 5 begin to download photo : 1171275314 begin to download photo : 1171275372 begin to download photo : 1171291875 download finish for photo 1171291875 download finish for photo 1171275314 download finish for photo 1171275372 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 ##### After load_data_input time to download the photos : 0.2606828212738037 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:tfhub_classification2 Wed Sep 24 17:36:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1758728214_1852108_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg': 1171291875, 'temp/1758728214_1852108_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg': 1171275314, 'temp/1758728214_1852108_1171275372_76d81364ff7df843bff095f45c07ba35.jpg': 1171275372} map_photo_id_path_extension : {1171291875: {'path': 'temp/1758728214_1852108_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg', 'extension': 'jpg'}, 1171275314: {'path': 'temp/1758728214_1852108_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg', 'extension': 'jpg'}, 1171275372: {'path': 'temp/1758728214_1852108_1171275372_76d81364ff7df843bff095f45c07ba35.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step TFHub with tf2 ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'3655': 1} we are using the classfication for only one thcl 3655 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 9 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 9 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 9 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 9 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 9 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 9 wait 20 seconds l 3637 free memory gpu now : 9 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3655 To do loadFromThcl(), then load ParamDescType : thcl3655 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (3655) thcls : [{'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'}] thcl {'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'} Update svm_hashtag_type_desc : 5862 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (5862) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5862, 'tfhub_18_7_2023', 1280, 1280, 'tfhub_18_7_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 7, 18, 22, 46, 29), datetime.datetime(2023, 7, 18, 22, 46, 29)) model_name : tfhub_18_7_2023 model_param file didn't exist model_name : tfhub_18_7_2023 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/tfhub_18_7_2023 /data/models_weight/tfhub_18_7_2023/Confusion_Matrix.png size_local : 54360 size in s3 : 54360 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:28 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_jrm.jpg size_local : 72583 size in s3 : 72583 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_jrm.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pcm.jpg size_local : 81681 size in s3 : 81681 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:17 Precision_Recall_pcm.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pcnc.jpg size_local : 79510 size in s3 : 79510 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_pcnc.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pehd.jpg size_local : 59936 size in s3 : 59936 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_pehd.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_tapis_vide.jpg size_local : 78974 size in s3 : 78974 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17 Precision_Recall_tapis_vide.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Result_Summary.txt size_local : 642 size in s3 : 642 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 Result_Summary.txt already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 checkpoint already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216529 size in s3 : 216529 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279748 size in s3 : 32279748 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.index size_local : 43546 size in s3 : 43546 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_weights.h5 size_local : 16500868 size in s3 : 16500868 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:18 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2333, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2527, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3139, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 62, in create_tfhub_model fe_layer = hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module", File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype gen_logging_ops._assert( # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1171291875, 1171275314, 1171275372] map_info['map_portfolio_photo'] : {} final : True mtd_id 4621 list_pids : [1171291875, 1171275314, 1171275372] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4621', None, '1171291875', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171275314', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171275372', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.01563096046447754 save_final ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse time spend for datou_step_exec : 133.64525938034058 time spend to save output : 0.016034603118896484 total time spend for step 0 : 133.66129398345947 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : None probably due to empty image bug ERROR expected : {'1171291875': [(1171291875, 'tapis_vide', 0.97062814, 4723, '3655'), 'temp/1691745841_1143057_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg'], '1171275372': [(1171275372, 'tapis_vide', 0.9674145, 4723, '3655'), 'temp/1691745841_1143057_1171275372_76d81364ff7df843bff095f45c07ba35.jpg'], '1171275314': [(1171275314, 'tapis_vide', 0.96509415, 4723, '3655'), 'temp/1691745841_1143057_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg']} got : None ERROR tfhub2 FAILED ############################### TEST ordonner ################################ To do loadFromThcl(), then load ParamDescType : thcl358 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (358) thcls : [{'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3392 ['FirstUploadExperveo_vignette__port_505674', 'CAR_EXTERIEUR_Roue__port_503398', 'FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486', 'FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465', 'CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198', 'CAR_EXTERIEUR_Face_avant_axe_droit__port_504451', 'CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235', 'FirstUploadExperveo_vin__port_505675', 'CAR_EXTERIEUR_cote_droite__port_504108', 'CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565', 'FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201', 'cartegrise_orientation__port_505064', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217', 'CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531', 'CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218', 'CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214', 'CAR_EXTERIEUR_Angle_avant_droit__port_504087', 'FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484', 'CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563', 'CAR_EXTERIEUR_Angle_arriere_droit__port_504160', 'CAR_EXTERIEUR_arriere__port_504184', 'CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562', 'INTERIEUR_Compteur_kilometrique__port_503644', 'CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494', 'CAR_EXTERIEUR_Angle_arriere_gauche__port_504170', 'CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226', 'CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202', 'CAR_EXTERIEUR_moteur__port_503704', 'FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487', 'CAR_INTERIEUR_siege_arriere_class_1__port_506551', 'CAR_EXTERIEUR_avant__port_504146', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215', 'CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225', 'CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564', 'FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'CAR_INTERIEUR_coffre__port_503412', 'FirstUploadExperveo_rouetranche__port_505677', 'UploadPhotoImmatBest_class_1__port_505051', 'CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532', 'CAR_EXTERIEUR_angle_avant_gauche__port_504098', 'CAR_EXTERIEUR_face_avant_axe_gauche__port_504236', 'CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540', 'CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233', 'CAR_EXTERIEUR_roue_de_secour__port_503763', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199', 'CAR_EXTERIEUR_cote_gauche__port_504017', 'CAR_INTERIEUR_avant_volant_class_1__port_506503', 'CAR_INTERIEUR_avant_volant_class_2__port_506504', 'CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'] 51 SELECT hashtag_id,hashtag FROM MTRBack.hashtags where hashtag in ('FirstUploadExperveo_vignette__port_505674','CAR_EXTERIEUR_Roue__port_503398','FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486','FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485','CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465','CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198','CAR_EXTERIEUR_Face_avant_axe_droit__port_504451','CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235','FirstUploadExperveo_vin__port_505675','CAR_EXTERIEUR_cote_droite__port_504108','CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565','FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483','CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201','cartegrise_orientation__port_505064','CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217','CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531','CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218','CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214','CAR_EXTERIEUR_Angle_avant_droit__port_504087','FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484','CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563','CAR_EXTERIEUR_Angle_arriere_droit__port_504160','CAR_EXTERIEUR_arriere__port_504184','CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562','INTERIEUR_Compteur_kilometrique__port_503644','CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494','CAR_EXTERIEUR_Angle_arriere_gauche__port_504170','CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226','CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202','CAR_EXTERIEUR_moteur__port_503704','FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487','CAR_INTERIEUR_siege_arriere_class_1__port_506551','CAR_EXTERIEUR_avant__port_504146','CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215','CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225','CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564','FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482','CAR_INTERIEUR_coffre__port_503412','FirstUploadExperveo_rouetranche__port_505677','UploadPhotoImmatBest_class_1__port_505051','CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532','CAR_EXTERIEUR_angle_avant_gauche__port_504098','CAR_EXTERIEUR_face_avant_axe_gauche__port_504236','CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540','CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233','CAR_EXTERIEUR_roue_de_secour__port_503763','CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199','CAR_EXTERIEUR_cote_gauche__port_504017','CAR_INTERIEUR_avant_volant_class_1__port_506503','CAR_INTERIEUR_avant_volant_class_2__port_506504','CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'); 51 dict_keys(['cartegrise_orientation__port_505064', 'car_exterieur_angle_arriere_droit_axe_arriere__port_504217', 'car_exterieur_angle_arriere_droit_axe_droit__port_504215', 'car_exterieur_angle_arriere_droit__port_504160', 'car_exterieur_angle_arriere_gauche_axe_arriere__port_504201', 'car_exterieur_angle_arriere_gauche_axe_gauche__port_504199', 'car_exterieur_angle_arriere_gauche__port_504170', 'car_exterieur_angle_avant_droit_axe_arriere__port_504226', 'car_exterieur_angle_avant_droit_axe_droit__port_504225', 'car_exterieur_angle_avant_droit__port_504087', 'car_exterieur_angle_avant_gauche_axe_avant__port_504235', 'car_exterieur_angle_avant_gauche_axe_gauche__port_504234', 'car_exterieur_angle_avant_gauche__port_504098', 'car_exterieur_arriere__port_504184', 'car_exterieur_avant__port_504146', 'car_exterieur_cote_droite__port_504108', 'car_exterieur_cote_droit_axe_arriere__port_504214', 'car_exterieur_cote_droit_axe_avant__port_504465', 'car_exterieur_cote_gauche_axe_arriere__port_504198', 'car_exterieur_cote_gauche_axe_avant__port_504233', 'car_exterieur_cote_gauche__port_504017', 'car_exterieur_face_arriere_axe_droit__port_504218', 'car_exterieur_face_arriere_axe_gauche__port_504202', 'car_exterieur_face_avant_axe_droit__port_504451', 'car_exterieur_face_avant_axe_gauche__port_504236', 'car_exterieur_moteur__port_503704', 'car_exterieur_roue_de_secour__port_503763', 'car_exterieur_roue__port_503398', 'car_interieur_avant_volant_class_1__port_506503', 'car_interieur_avant_volant_class_2__port_506504', 'car_interieur_avant_volant_class_6_boutonrond__port_506562', 'car_interieur_avant_volant_class_6_class_2__port_506563', 'car_interieur_avant_volant_class_6_ecrangrosplan__port_506564', 'car_interieur_avant_volant_class_6_levierdevitesse__port_506565', 'car_interieur_avant_vue-arriere_class_1__port_506531', 'car_interieur_avant_vue-arriere_class_2__port_506532', 'car_interieur_avant_vue_droite_habitacle_class_1__port_506540', 'car_interieur_avant_vue_gauche_habitacle_class_1__port_506494', 'car_interieur_coffre__port_503412', 'car_interieur_siege_arriere_class_1__port_506551', 'firstuploadexperveo_carrosseriegrosplan_carrosserie__port_506483', 'firstuploadexperveo_carrosseriegrosplan_class_6__port_506487', 'firstuploadexperveo_carrosseriegrosplan_morceauderoue__port_506484', 'firstuploadexperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'firstuploadexperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'firstuploadexperveo_carrosseriegrosplan_vindanslamoquette__port_506486', 'firstuploadexperveo_rouetranche__port_505677', 'firstuploadexperveo_vignette__port_505674', 'firstuploadexperveo_vin__port_505675', 'interieur_compteur_kilometrique__port_503644', 'uploadphotoimmatbest_class_1__port_505051']) select photo_hashtag_type from MTRDatou.classification_theme where id = 358 thcl : 358 photo_hashtag_type : 337 SELECT phi.hashtag_id , phi.photo_id FROM MTRBack.photo_hashtag_ids phi, MTRUser.mtr_portfolio_photos mp where phi.type = 337 and phi.photo_id = mp.mtr_photo_id and mp.mtr_portfolio_id =510365; {510365: [(917973295, 1), (917973297, 1), (917973302, 1), (917973293, 7), (917973296, 11), (917973300, 11), (917973286, 13), (917973289, 13), (917973301, 24), (917973285, 29), (917973290, 29), (917973299, 29), (917973304, 35), (917973287, 36), (917973298, 36), (917973305, 36), (917973292, 37), (917973291, 41), (917973303, 41), (917973294, 42), (917973288, 46)]} ############################### TEST rotate ################################ test rotate only Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=230 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=230 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 230 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=230 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : rotate list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.view_photos_with_urls_250912 ph WHERE photo_id IN (917849322) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917849322 download finish for photo 917849322 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.17980170249938965 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:rotate Wed Sep 24 17:39:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322} map_photo_id_path_extension : {917849322: {'path': 'temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step_rotate ! We are in a linear step without datou_depend ! rotate photos of 90,180,270 degres batch 1 select photo_id, hashtag_id, `type`, x0, x1, y0, y1, score, id from MTRPhoto.crop_hashtag_ids where photo_id in ( 917849322) and `type` in (0) Loaded 0 chid ids of type : 0 SELECT crop_hashtag_id, points FROM MTRPhoto.crop_polygon_points WHERE crop_hashtag_id in () map_chi : {} photo_id in download_rotate_and_save : 917849322 list_chi_loc : 0 Use all angle ! Rotation of photo 917849322 of 90 degree temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 90 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[ 6.123234e-17 1.000000e+00] [-1.000000e+00 6.123234e-17]] 90 [[ 6.123234e-17 1.000000e+00] [-1.000000e+00 6.123234e-17]] shrink_image : False image_rotate : image_path : temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg image_rotate.mode : RGB Rotation of photo 917849322 of 180 degree temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 180 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[-1.0000000e+00 1.2246468e-16] [-1.2246468e-16 -1.0000000e+00]] 180 [[-1.0000000e+00 1.2246468e-16] [-1.2246468e-16 -1.0000000e+00]] shrink_image : False image_rotate : image_path : temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg image_rotate.mode : RGB Rotation of photo 917849322 of 270 degree temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 270 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[-1.8369702e-16 -1.0000000e+00] [ 1.0000000e+00 -1.8369702e-16]] 270 [[-1.8369702e-16 -1.0000000e+00] [ 1.0000000e+00 -1.8369702e-16]] shrink_image : False image_rotate : image_path : temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg image_rotate.mode : RGB About to upload 3 photos upload in portfolio : 551782 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1758728349_1852108 batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first Unexecpected behavior in 07/2025 that can be generalized l287 : type_extension .jpg This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first Unexecpected behavior in 07/2025 that can be generalized l287 : type_extension .jpg This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first Unexecpected behavior in 07/2025 that can be generalized l287 : type_extension .jpg This is a hack ! we have uploaded 3 photos in the portfolio 551782 time of upload the photos Elapsed time : 1.289466142654419 map_filename_photo_id : 3 map_filename_photo_id : {'temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg': 1385927920, 'temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg': 1385927921, 'temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg': 1385927922} Len new_chis : 3 Len list_new_chi_with_photo_id : 0 of type : 0 list_new_chi_with_photo_id : [] After datou_step_exec type output : time spend for datou_step_exec : 1.5058612823486328 time spend to save output : 2.2411346435546875e-05 total time spend for step 1 : 1.5058836936950684 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : True saveOutput not yet implemented for datou_step.type : rotate we use saveGeneral [917849322] map_info['map_portfolio_photo'] : {} final : True mtd_id 230 list_pids : [917849322] Looping around the photos to save general results len do output : 3 /1385927920Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1385927921Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1385927922Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('230', None, None, None, None, None, None, None, None) ('230', None, '917849322', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('230', None, '1385927920', 'None', None, None, None, None, None), ('230', None, '1385927921', 'None', None, None, None, None, None), ('230', None, '1385927922', 'None', None, None, None, None, None), ('230', None, '917849322', None, None, None, None, None, None)] time used for this insertion : 0.01361393928527832 save_final save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {1385927920: ['917849322', 'temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg', []], 1385927921: ['917849322', 'temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg', []], 1385927922: ['917849322', 'temp/1758728348_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg', []]} test rotate only is a success ! test rotate conditionnel Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=233 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=233 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 233 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=233 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : thcl, argmax, rotate list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.view_photos_with_urls_250912 ph WHERE photo_id IN (917849322) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917849322 download finish for photo 917849322 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.17200684547424316 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 3 step1:thcl Wed Sep 24 17:39:10 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1758728350_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322} map_photo_id_path_extension : {917849322: {'path': 'temp/1758728350_1852108_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Thcl ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'500': 1} we are using the classfication for only one thcl 500 In convert_file_to_np l 337 : 1 l343 1 l357 after caffe.io.load_image dimension du image : (3, (2448, 3264, 3)) dimension displayed ! time to import caffe and check if the image exist : 0.0002205371856689453 time to convert the images to numpy array : 0.9088418483734131 total time to convert the images to numpy array : 0.9093396663665771 list photo_ids error: [] list photo_ids correct : [917849322] number of photos to traite : 1 try to delete the photos incorrect in DB tagging for thcl : 500 To do loadFromThcl(), then load ParamDescType : thcl500 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (500) thcls : [{'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'}] thcl {'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'} Update svm_hashtag_type_desc : 3517 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3517) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34)) To loadFromThcl() : net_3517 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 9 wait 20 seconds l 3637 free memory gpu now : 9 max_wait_temp : 1 max_wait : 0 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3517) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34)) param : , param.caffemodel : orientation_carte_grise_all_2 None mean_file_type : mean_file_path : prototxt_file_path : model : orientation_carte_grise_all_2 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : orientation_carte_grise_all_2 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : [] local folder : /data/models_weight/orientation_carte_grise_all_2 /data/models_weight/orientation_carte_grise_all_2/caffemodel size_local : 537110520 size in s3 : 537110520 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:17 caffemodel already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy_conv_normal.prototxt size_local : 4626 size in s3 : 4626 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy_conv_normal.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy_fc.prototxt size_local : 1130 size in s3 : 1130 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy_fc.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt size_local : 5653 size in s3 : 5653 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:31 mean.npy already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/synset_words.txt size_local : 159 size in s3 : 159 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/:/home/admin/workarea/git/apy/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt caffemodel_filename : /data/models_weight/orientation_carte_grise_all_2/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 9 wait 20 seconds WARNING: Logging before InitGoogleLogging() is written to STDERR F0924 17:39:58.360069 1852108 syncedmem.cpp:78] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** Command terminated by signal 6 23.99user 18.35system 6:30.82elapsed 10%CPU (0avgtext+0avgdata 5214380maxresident)k 5227240inputs+21200outputs (13607major+2535077minor)pagefaults 0swaps