python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120 import MySQLdb succeeded Import error (python version) python version = 3 warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data list_job_run_as_list : ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel'] python version used : 3 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')] #&_# BEGIN OF TEST : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/mask_test.py Test mask-detection python version used : 3 ############################### TEST memory used ################################ free memory at begining : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 407 run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.11244606971740723 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Thu May 15 18:35:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 407 wait 20 seconds l 3637 free memory gpu now : 407 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 /home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp 2025-05-15 18:35:51.537080: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-15 18:35:51.563190: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-15 18:35:2025-05-15 18:36:03.525870: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25MiB (rounded to 2359296) Current allocation summary follows. 2025-05-15 18:36:03.525947: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc 2025-05-15 18:36:03.525969: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): Total Chunks: 59, Chunks in use: 59. 14.8KiB allocated for chunks. 14.8KiB in use in bin. 8.8KiB client-requested in use in bin. 2025-05-15 18:36:03.525987: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2025-05-15 18:36:03.526002: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): Total Chunks: 56, Chunks in use: 56. 56.2KiB allocated for chunks. 56.2KiB in use in bin. 56.0KiB client-requested in use in bin. 2025-05-15 18:36:03.526018: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): Total Chunks: 25, Chunks in use: 25. 50.0KiB allocated for chunks. 50.0KiB in use in bin. 50.0KiB client-requested in use in bin. 2025-05-15 18:36:03.526033: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): Total Chunks: 20, Chunks in use: 20. 83.5KiB allocated for chunks. 83.5KiB in use in bin. 80.0KiB client-requested in use in bin. 2025-05-15 18:36:03.526047: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:03.526076: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin. 2025-05-15 18:36:03.526092: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin. 2025-05-15 18:36:03.526107: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): Total Chunks: 7, Chunks in use: 6. 459.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin. 2025-05-15 18:36:03.526123: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): Total Chunks: 4, Chunks in use: 4. 560.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin. 2025-05-15 18:36:03.526138: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): Total Chunks: 9, Chunks in use: 7. 2.41MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin. 2025-05-15 18:36:03.526153: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): Total Chunks: 8, Chunks in use: 6. 4.95MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin. 2025-05-15 18:36:03.526168: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): Total Chunks: 8, Chunks in use: 6. 8.50MiB allocated for chunks. 6.25MiB in use in bin. 6.00MiB client-requested in use in bin. 2025-05-15 18:36:03.526183: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): Total Chunks: 6, Chunks in use: 6. 13.75MiB allocated for chunks. 13.75MiB in use in bin. 13.25MiB client-requested in use in bin. 2025-05-15 18:36:03.526197: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:03.526210: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:03.526224: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:03.526237: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:03.526250: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:03.526264: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:03.526277: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:03.526291: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 2.25MiB was 2.00MiB, Chunk State: 2025-05-15 18:36:03.526303: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16646144 2025-05-15 18:36:03.526321: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e92000000 of size 2359296 next 181 2025-05-15 18:36:03.526333: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e92240000 of size 2097152 next 196 2025-05-15 18:36:03.526346: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e92440000 of size 1048576 next 216 2025-05-15 18:36:03.526358: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9e92540000 of size 1310720 next 209 2025-05-15 18:36:03.526378: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e92680000 of size 2359296 next 208 2025-05-15 18:36:03.526391: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e928c0000 of size 1048576 next 234 2025-05-15 18:36:03.526403: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e929c0000 of size 1310720 next 228 2025-05-15 18:36:03.526415: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e92b00000 of size 2359296 next 227 2025-05-15 18:36:03.526427: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e92d40000 of size 2752512 next 18446744073709551615 2025-05-15 18:36:03.526439: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152 2025-05-15 18:36:03.526452: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94600000 of size 147456 next 55 2025-05-15 18:36:03.526463: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94624000 of size 65536 next 78 2025-05-15 18:36:03.526476: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94634000 of size 4096 next 191 2025-05-15 18:36:03.526487: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94635000 of size 4096 next 192 2025-05-15 18:36:03.526499: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94636000 of size 4096 next 193 2025-05-15 18:36:03.526511: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94637000 of size 256 next 194 2025-05-15 18:36:03.526523: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94637100 of size 256 next 195 2025-05-15 18:36:03.526535: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94637200 of size 4096 next 197 2025-05-15 18:36:03.526547: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94638200 of size 4096 next 198 2025-05-15 18:36:03.526558: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94639200 of size 4096 next 199 2025-05-15 18:36:03.526570: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463a200 of size 4096 next 200 2025-05-15 18:36:03.526582: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463b200 of size 4096 next 201 2025-05-15 18:36:03.526594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463c200 of size 1024 next 202 2025-05-15 18:36:03.526605: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463c600 of size 1024 next 204 2025-05-15 18:36:03.526617: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463ca00 of size 1024 next 205 2025-05-15 18:36:03.526629: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463ce00 of size 1024 next 206 2025-05-15 18:36:03.526640: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463d200 of size 1024 next 207 2025-05-15 18:36:03.526652: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463d600 of size 1024 next 210 2025-05-15 18:36:03.526664: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463da00 of size 1024 next 211 2025-05-15 18:36:03.526675: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463de00 of size 1024 next 212 2025-05-15 18:36:03.526687: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463e200 of size 1024 next 213 2025-05-15 18:36:03.526699: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463e600 of size 1024 next 214 2025-05-15 18:36:03.526711: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463ea00 of size 4096 next 215 2025-05-15 18:36:03.526722: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463fa00 of size 4096 next 217 2025-05-15 18:36:03.526734: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94640a00 of size 4096 next 218 2025-05-15 18:36:03.526746: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94641a00 of size 4096 next 219 2025-05-15 18:36:03.526766: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94642a00 of size 4096 next 220 2025-05-15 18:36:03.526778: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94643a00 of size 1024 next 221 2025-05-15 18:36:03.526789: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94643e00 of size 1024 next 223 2025-05-15 18:36:03.526801: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94644200 of size 1024 next 224 2025-05-15 18:36:03.526813: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94644600 of size 1024 next 225 2025-05-15 18:36:03.526825: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94644a00 of size 1024 next 226 2025-05-15 18:36:03.526836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94644e00 of size 1024 next 229 2025-05-15 18:36:03.526848: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94645200 of size 1024 next 230 2025-05-15 18:36:03.526860: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94645600 of size 1024 next 231 2025-05-15 18:36:03.526871: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94645a00 of size 1024 next 232 2025-05-15 18:36:03.526883: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94645e00 of size 1024 next 233 2025-05-15 18:36:03.526909: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94646200 of size 7680 next 72 2025-05-15 18:36:03.526924: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94648000 of size 147456 next 71 2025-05-15 18:36:03.526936: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9466c000 of size 131072 next 87 2025-05-15 18:36:03.526948: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9e9468c000 of size 262144 next 103 2025-05-15 18:36:03.526960: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e946cc000 of size 262144 next 102 2025-05-15 18:36:03.526972: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9e9470c000 of size 999424 next 18446744073709551615 2025-05-15 18:36:03.526984: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304 2025-05-15 18:36:03.526996: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94800000 of size 589824 next 96 2025-05-15 18:36:03.527008: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94890000 of size 262144 next 117 2025-05-15 18:36:03.527019: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9e948d0000 of size 262144 next 112 2025-05-15 18:36:03.527032: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94910000 of size 524288 next 111 2025-05-15 18:36:03.527043: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94990000 of size 262144 next 135 2025-05-15 18:36:03.527055: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e949d0000 of size 327680 next 124 2025-05-15 18:36:03.527067: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94a20000 of size 589824 next 123 2025-05-15 18:36:03.527079: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94ab0000 of size 262144 next 152 2025-05-15 18:36:03.527091: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94af0000 of size 327680 next 142 2025-05-15 18:36:03.527103: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94b40000 of size 786432 next 18446744073709551615 2025-05-15 18:36:03.527115: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608 2025-05-15 18:36:03.527126: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9e94c00000 of size 589824 next 160 2025-05-15 18:36:03.527138: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94c90000 of size 589824 next 158 2025-05-15 18:36:03.527150: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94d20000 of size 524288 next 171 2025-05-15 18:36:03.527170: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9e94da0000 of size 1048576 next 222 2025-05-15 18:36:03.527182: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94ea0000 of size 1048576 next 190 2025-05-15 18:36:03.527194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94fa0000 of size 1048576 next 189 2025-05-15 18:36:03.527206: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e950a0000 of size 1048576 next 203 2025-05-15 18:36:03.527218: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e951a0000 of size 2490368 next 18446744073709551615 2025-05-15 18:36:03.527229: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576 2025-05-15 18:36:03.527241: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00000 of size 1280 next 1 2025-05-15 18:36:03.527253: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00500 of size 256 next 5 2025-05-15 18:36:03.527265: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00600 of size 256 next 8 2025-05-15 18:36:03.527277: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00700 of size 256 next 9 2025-05-15 18:36:03.527289: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00800 of size 256 next 10 2025-05-15 18:36:03.527300: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00900 of size 256 next 11 2025-05-15 18:36:03.527312: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00a00 of size 256 next 12 2025-05-15 18:36:03.527324: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00b00 of size 256 next 13 2025-05-15 18:36:03.527335: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00c00 of size 256 next 17 2025-05-15 18:36:03.527347: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00d00 of size 256 next 19 2025-05-15 18:36:03.527359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00e00 of size 256 next 20 2025-05-15 18:36:03.527370: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00f00 of size 256 next 21 2025-05-15 18:36:03.527382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01000 of size 256 next 22 2025-05-15 18:36:03.527394: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01100 of size 256 next 24 2025-05-15 18:36:03.527405: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01200 of size 256 next 25 2025-05-15 18:36:03.527417: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01300 of size 256 next 23 2025-05-15 18:36:03.527428: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01400 of size 256 next 28 2025-05-15 18:36:03.527440: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01500 of size 256 next 29 2025-05-15 18:36:03.527452: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01600 of size 256 next 30 2025-05-15 18:36:03.527463: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01700 of size 256 next 31 2025-05-15 18:36:03.527475: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01800 of size 256 next 33 2025-05-15 18:36:03.527487: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01900 of size 256 next 34 2025-05-15 18:36:03.527499: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01a00 of size 1024 next 32 2025-05-15 18:36:03.527511: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01e00 of size 1024 next 37 2025-05-15 18:36:03.527522: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee02200 of size 1024 next 38 2025-05-15 18:36:03.527534: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee02600 of size 1024 next 39 2025-05-15 18:36:03.527546: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee02a00 of size 1024 next 40 2025-05-15 18:36:03.527565: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee02e00 of size 1024 next 41 2025-05-15 18:36:03.527577: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee03200 of size 1024 next 43 2025-05-15 18:36:03.527589: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee03600 of size 1024 next 44 2025-05-15 18:36:03.527601: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee03a00 of size 1024 next 45 2025-05-15 18:36:03.527613: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee03e00 of size 1024 next 46 2025-05-15 18:36:03.527624: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04200 of size 256 next 48 2025-05-15 18:36:03.527636: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04300 of size 256 next 49 2025-05-15 18:36:03.527648: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04400 of size 256 next 50 2025-05-15 18:36:03.527659: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04500 of size 256 next 51 2025-05-15 18:36:03.527671: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04600 of size 256 next 52 2025-05-15 18:36:03.527683: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04700 of size 256 next 53 2025-05-15 18:36:03.527695: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04800 of size 256 next 56 2025-05-15 18:36:03.527707: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04900 of size 256 next 57 2025-05-15 18:36:03.527718: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04a00 of size 256 next 58 2025-05-15 18:36:03.527730: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04b00 of size 256 next 14 2025-05-15 18:36:03.527742: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04c00 of size 256 next 15 2025-05-15 18:36:03.527753: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04d00 of size 256 next 16 2025-05-15 18:36:03.527765: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04e00 of size 1024 next 60 2025-05-15 18:36:03.527777: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee05200 of size 1024 next 61 2025-05-15 18:36:03.527788: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee05600 of size 1024 next 62 2025-05-15 18:36:03.527800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee05a00 of size 1024 next 63 2025-05-15 18:36:03.527812: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee05e00 of size 1024 next 64 2025-05-15 18:36:03.527823: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06200 of size 256 next 66 2025-05-15 18:36:03.527835: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06300 of size 256 next 67 2025-05-15 18:36:03.527847: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06400 of size 256 next 68 2025-05-15 18:36:03.527858: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06500 of size 256 next 69 2025-05-15 18:36:03.527870: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06600 of size 256 next 70 2025-05-15 18:36:03.527882: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06700 of size 256 next 73 2025-05-15 18:36:03.527894: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06800 of size 256 next 74 2025-05-15 18:36:03.527905: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06900 of size 256 next 75 2025-05-15 18:36:03.527917: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06a00 of size 256 next 76 2025-05-15 18:36:03.527929: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06b00 of size 256 next 77 2025-05-15 18:36:03.527948: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06c00 of size 1024 next 79 2025-05-15 18:36:03.527960: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee07000 of size 1024 next 80 2025-05-15 18:36:03.527972: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee07400 of size 1024 next 81 2025-05-15 18:36:03.527984: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee07800 of size 1024 next 82 2025-05-15 18:36:03.527995: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee07c00 of size 1024 next 83 2025-05-15 18:36:03.528007: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08000 of size 256 next 85 2025-05-15 18:36:03.528019: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08100 of size 256 next 86 2025-05-15 18:36:03.528031: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08200 of size 512 next 84 2025-05-15 18:36:03.528042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08400 of size 512 next 88 2025-05-15 18:36:03.528054: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08600 of size 512 next 89 2025-05-15 18:36:03.528066: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08800 of size 512 next 90 2025-05-15 18:36:03.528078: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08a00 of size 512 next 91 2025-05-15 18:36:03.528089: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08c00 of size 256 next 93 2025-05-15 18:36:03.528101: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08d00 of size 256 next 94 2025-05-15 18:36:03.528112: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08e00 of size 512 next 92 2025-05-15 18:36:03.528124: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09000 of size 512 next 97 2025-05-15 18:36:03.528136: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09200 of size 512 next 98 2025-05-15 18:36:03.528147: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09400 of size 512 next 99 2025-05-15 18:36:03.528159: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09600 of size 512 next 2 2025-05-15 18:36:03.528171: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09800 of size 256 next 3 2025-05-15 18:36:03.528182: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09900 of size 256 next 4 2025-05-15 18:36:03.528194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09a00 of size 16384 next 18 2025-05-15 18:36:03.528206: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0da00 of size 256 next 100 2025-05-15 18:36:03.528218: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0db00 of size 256 next 101 2025-05-15 18:36:03.528229: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0dc00 of size 2048 next 104 2025-05-15 18:36:03.528241: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0e400 of size 2048 next 105 2025-05-15 18:36:03.528253: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0ec00 of size 2048 next 106 2025-05-15 18:36:03.528265: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0f400 of size 2048 next 107 2025-05-15 18:36:03.528277: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0fc00 of size 2048 next 108 2025-05-15 18:36:03.528289: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee10400 of size 256 next 109 2025-05-15 18:36:03.528300: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee10500 of size 256 next 110 2025-05-15 18:36:03.528312: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee10600 of size 2048 next 113 2025-05-15 18:36:03.528324: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee10e00 of size 2048 next 114 2025-05-15 18:36:03.528343: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee11600 of size 2048 next 115 2025-05-15 18:36:03.528356: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee11e00 of size 2048 next 116 2025-05-15 18:36:03.528367: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee12600 of size 2048 next 6 2025-05-15 18:36:03.528379: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee12e00 of size 37632 next 7 2025-05-15 18:36:03.528391: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1c100 of size 512 next 118 2025-05-15 18:36:03.528403: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1c300 of size 512 next 119 2025-05-15 18:36:03.528414: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1c500 of size 512 next 120 2025-05-15 18:36:03.528426: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1c700 of size 512 next 121 2025-05-15 18:36:03.528438: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1c900 of size 512 next 122 2025-05-15 18:36:03.528449: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1cb00 of size 512 next 125 2025-05-15 18:36:03.528461: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1cd00 of size 512 next 126 2025-05-15 18:36:03.528473: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1cf00 of size 512 next 127 2025-05-15 18:36:03.528484: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1d100 of size 512 next 128 2025-05-15 18:36:03.528496: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1d300 of size 512 next 129 2025-05-15 18:36:03.528508: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1d500 of size 2048 next 130 2025-05-15 18:36:03.528519: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1dd00 of size 2048 next 131 2025-05-15 18:36:03.528531: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1e500 of size 2048 next 132 2025-05-15 18:36:03.528543: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1ed00 of size 2048 next 133 2025-05-15 18:36:03.528554: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1f500 of size 2048 next 134 2025-05-15 18:36:03.528566: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1fd00 of size 512 next 136 2025-05-15 18:36:03.528578: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1ff00 of size 512 next 137 2025-05-15 18:36:03.528589: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20100 of size 512 next 138 2025-05-15 18:36:03.528601: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20300 of size 512 next 139 2025-05-15 18:36:03.528613: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20500 of size 512 next 140 2025-05-15 18:36:03.528624: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20700 of size 512 next 141 2025-05-15 18:36:03.528636: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20900 of size 512 next 143 2025-05-15 18:36:03.528648: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20b00 of size 512 next 144 2025-05-15 18:36:03.528659: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20d00 of size 512 next 145 2025-05-15 18:36:03.528671: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20f00 of size 512 next 146 2025-05-15 18:36:03.528683: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee21100 of size 2048 next 147 2025-05-15 18:36:03.528694: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee21900 of size 2048 next 148 2025-05-15 18:36:03.528706: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee22100 of size 2048 next 149 2025-05-15 18:36:03.528725: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee22900 of size 2048 next 150 2025-05-15 18:36:03.528737: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee23100 of size 2048 next 151 2025-05-15 18:36:03.528749: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee23900 of size 512 next 153 2025-05-15 18:36:03.528761: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee23b00 of size 512 next 154 2025-05-15 18:36:03.528772: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee23d00 of size 512 next 155 2025-05-15 18:36:03.528784: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee23f00 of size 512 next 156 2025-05-15 18:36:03.528796: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24100 of size 512 next 157 2025-05-15 18:36:03.528807: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24300 of size 512 next 161 2025-05-15 18:36:03.528819: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24500 of size 512 next 162 2025-05-15 18:36:03.528831: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24700 of size 512 next 163 2025-05-15 18:36:03.528842: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24900 of size 512 next 164 2025-05-15 18:36:03.528854: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24b00 of size 512 next 165 2025-05-15 18:36:03.528866: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24d00 of size 2048 next 166 2025-05-15 18:36:03.528877: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee25500 of size 2048 next 167 2025-05-15 18:36:03.528889: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee25d00 of size 2048 next 168 2025-05-15 18:36:03.528901: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee26500 of size 2048 next 169 2025-05-15 18:36:03.528913: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee26d00 of size 2048 next 170 2025-05-15 18:36:03.528924: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee27500 of size 1024 next 172 2025-05-15 18:36:03.528936: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee27900 of size 1024 next 173 2025-05-15 18:36:03.528948: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee27d00 of size 1024 next 174 2025-05-15 18:36:03.528960: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee28100 of size 1024 next 175 2025-05-15 18:36:03.528971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee28500 of size 1024 next 176 2025-05-15 18:36:03.528983: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee28900 of size 256 next 178 2025-05-15 18:36:03.528995: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee28a00 of size 256 next 179 2025-05-15 18:36:03.529006: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee28b00 of size 1024 next 177 2025-05-15 18:36:03.529018: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee28f00 of size 1024 next 182 2025-05-15 18:36:03.529030: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee29300 of size 1024 next 183 2025-05-15 18:36:03.529041: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee29700 of size 1024 next 184 2025-05-15 18:36:03.529053: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee29b00 of size 1024 next 185 2025-05-15 18:36:03.529065: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee29f00 of size 256 next 187 2025-05-15 18:36:03.529076: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee2a000 of size 256 next 188 2025-05-15 18:36:03.529088: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee2a100 of size 4096 next 186 2025-05-15 18:36:03.529100: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee2b100 of size 4096 next 42 2025-05-15 18:36:03.529119: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee2c100 of size 65536 next 36 2025-05-15 18:36:03.529131: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee3c100 of size 65536 next 35 2025-05-15 18:36:03.529143: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee4c100 of size 4096 next 235 2025-05-15 18:36:03.529155: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee4d100 of size 4096 next 236 2025-05-15 18:36:03.529167: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee4e100 of size 4096 next 237 2025-05-15 18:36:03.529178: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee4f100 of size 4096 next 238 2025-05-15 18:36:03.529190: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee50100 of size 1024 next 239 2025-05-15 18:36:03.529201: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee50500 of size 1024 next 240 2025-05-15 18:36:03.529213: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee50900 of size 1024 next 241 2025-05-15 18:36:03.529225: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee50d00 of size 1024 next 242 2025-05-15 18:36:03.529236: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee51100 of size 1024 next 243 2025-05-15 18:36:03.529248: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9fcee51500 of size 76800 next 27 2025-05-15 18:36:03.529260: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee64100 of size 147456 next 26 2025-05-15 18:36:03.529272: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee88100 of size 65536 next 47 2025-05-15 18:36:03.529283: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee98100 of size 65536 next 59 2025-05-15 18:36:03.529295: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fceea8100 of size 65536 next 65 2025-05-15 18:36:03.529307: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fceeb8100 of size 294656 next 18446744073709551615 2025-05-15 18:36:03.529319: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size: 2025-05-15 18:36:03.529335: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 59 Chunks of size 256 totalling 14.8KiB 2025-05-15 18:36:03.529348: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB 2025-05-15 18:36:03.529362: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 55 Chunks of size 1024 totalling 55.0KiB 2025-05-15 18:36:03.529375: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB 2025-05-15 18:36:03.529388: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 2048 totalling 50.0KiB 2025-05-15 18:36:03.529401: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 19 Chunks of size 4096 totalling 76.0KiB 2025-05-15 18:36:03.529414: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB 2025-05-15 18:36:03.529427: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB 2025-05-15 18:36:03.529440: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB 2025-05-15 18:36:03.529453: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB 2025-05-15 18:36:03.529467: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB 2025-05-15 18:36:03.529480: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB 2025-05-15 18:36:03.529493: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB 2025-05-15 18:36:03.529506: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB 2025-05-15 18:36:03.529527: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB 2025-05-15 18:36:03.529540: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB 2025-05-15 18:36:03.529553: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB 2025-05-15 18:36:03.529566: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB 2025-05-15 18:36:03.529579: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 5 Chunks of size 1048576 totalling 5.00MiB 2025-05-15 18:36:03.529592: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1310720 totalling 1.25MiB 2025-05-15 18:36:03.529605: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2097152 totalling 2.00MiB 2025-05-15 18:36:03.529618: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 2359296 totalling 6.75MiB 2025-05-15 18:36:03.529631: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2490368 totalling 2.38MiB 2025-05-15 18:36:03.529643: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2752512 totalling 2.62MiB 2025-05-15 18:36:03.529656: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 26.54MiB 2025-05-15 18:36:03.529669: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 32374784 memory_limit_: 32374784 available bytes: 0 curr_region_allocation_bytes_: 33554432 2025-05-15 18:36:03.529685: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: Limit: 32374784 InUse: 27825152 MaxInUse: 27825408 NumAllocs: 763 MaxAllocSize: 4325376 2025-05-15 18:36:03.529706: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ******************___**********************************__**************_*****__********************* 2025-05-15 18:36:03.533337: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 Exception in mask_detect : OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] we want to redo the detection Using TensorFlow backend. max_time_sub_proc : 3600 erreur pendant la detection Useless call to update_current_state in case -12 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! ERROR : mask output needs to be a dictionnary now ! No output to save, continue without doing anything ! save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : -12 free memory after detection : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 88 ############################### TEST detect object ################################ run mask_detect Inside batchDatouExec : verbose : False Catched exception ! Connect or reconnect ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.19231390953063965 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Thu May 15 18:36:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 254 wait 20 seconds l 3637 free memory gpu now : 254 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-05-15 18:36:27.871733: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-15 18:36:27.903113: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-15 18:36:27.905086: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fa070000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-15 18:36:27.905138: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-15 18:36:27.909036: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-15 18:36:28.061517: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x366d9ec0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-15 18:36:28.061563: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-15 18:36:28.062431: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-15 18:36:28.062829: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 18:36:28.065719: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-15 18:36:28.068381: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-15 18:36:28.068862: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-15 18:36:28.072050: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-15 18:36:28.073099: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-15 18:36:28.077164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-15 18:36:28.078037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-15 18:36:28.078122: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 18:36:28.078575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-15 18:36:28.078590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-15 18:36:28.078599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-15 18:36:28.079397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2025-05-15 18:36:28.701232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-15 18:36:28.701321: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 18:36:28.701347: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-15 18:36:28.701371: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-15 18:36:28.701394: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-15 18:36:28.701416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-15 18:36:28.701438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-15 18:36:28.701461: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-15 18:36:28.702254: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-15 18:36:28.703293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-15 18:36:28.703345: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-15 18:36:28.703370: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-15 18:36:28.703388: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-15 18:36:28.703405: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-15 18:36:28.703423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-15 18:36:28.703440: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-15 18:36:28.703457: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-15 18:36:28.704238: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-15 18:36:28.704278: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-15 18:36:28.704289: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-15 18:36:28.704298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-15 18:36:28.705144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) 2025-05-15 18:36:39.700541: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25MiB (rounded to 2359296) Current allocation summary follows. 2025-05-15 18:36:39.700621: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc 2025-05-15 18:36:39.700643: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): Total Chunks: 59, Chunks in use: 59. 14.8KiB allocated for chunks. 14.8KiB in use in bin. 8.8KiB client-requested in use in bin. 2025-05-15 18:36:39.700660: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2025-05-15 18:36:39.700676: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): Total Chunks: 56, Chunks in use: 56. 56.2KiB allocated for chunks. 56.2KiB in use in bin. 56.0KiB client-requested in use in bin. 2025-05-15 18:36:39.700692: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): Total Chunks: 25, Chunks in use: 25. 50.0KiB allocated for chunks. 50.0KiB in use in bin. 50.0KiB client-requested in use in bin. 2025-05-15 18:36:39.700707: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): Total Chunks: 20, Chunks in use: 20. 83.5KiB allocated for chunks. 83.5KiB in use in bin. 80.0KiB client-requested in use in bin. 2025-05-15 18:36:39.700721: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:39.700737: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin. 2025-05-15 18:36:39.700752: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin. 2025-05-15 18:36:39.700769: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): Total Chunks: 7, Chunks in use: 6. 459.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin. 2025-05-15 18:36:39.700784: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): Total Chunks: 4, Chunks in use: 4. 560.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin. 2025-05-15 18:36:39.700800: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): Total Chunks: 9, Chunks in use: 7. 2.41MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin. 2025-05-15 18:36:39.700815: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): Total Chunks: 8, Chunks in use: 6. 4.95MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin. 2025-05-15 18:36:39.700830: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): Total Chunks: 8, Chunks in use: 6. 8.50MiB allocated for chunks. 6.25MiB in use in bin. 6.00MiB client-requested in use in bin. 2025-05-15 18:36:39.700845: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): Total Chunks: 6, Chunks in use: 6. 13.75MiB allocated for chunks. 13.75MiB in use in bin. 13.25MiB client-requested in use in bin. 2025-05-15 18:36:39.700859: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:39.700873: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:39.700908: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:39.700923: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:39.700936: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:39.700950: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:39.700963: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-05-15 18:36:39.700977: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 2.25MiB was 2.00MiB, Chunk State: 2025-05-15 18:36:39.700990: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16646144 2025-05-15 18:36:39.701011: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e92000000 of size 2359296 next 181 2025-05-15 18:36:39.701023: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e92240000 of size 2097152 next 196 2025-05-15 18:36:39.701035: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e92440000 of size 1048576 next 216 2025-05-15 18:36:39.701047: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9e92540000 of size 1310720 next 209 2025-05-15 18:36:39.701060: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e92680000 of size 2359296 next 208 2025-05-15 18:36:39.701071: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e928c0000 of size 1048576 next 234 2025-05-15 18:36:39.701084: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e929c0000 of size 1310720 next 228 2025-05-15 18:36:39.701096: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e92b00000 of size 2359296 next 227 2025-05-15 18:36:39.701108: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e92d40000 of size 2752512 next 18446744073709551615 2025-05-15 18:36:39.701120: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152 2025-05-15 18:36:39.701133: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94600000 of size 147456 next 55 2025-05-15 18:36:39.701145: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94624000 of size 65536 next 78 2025-05-15 18:36:39.701158: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94634000 of size 4096 next 191 2025-05-15 18:36:39.701170: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94635000 of size 4096 next 192 2025-05-15 18:36:39.701182: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94636000 of size 4096 next 193 2025-05-15 18:36:39.701194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94637000 of size 256 next 194 2025-05-15 18:36:39.701206: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94637100 of size 256 next 195 2025-05-15 18:36:39.701218: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94637200 of size 4096 next 197 2025-05-15 18:36:39.701229: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94638200 of size 4096 next 198 2025-05-15 18:36:39.701241: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94639200 of size 4096 next 199 2025-05-15 18:36:39.701253: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463a200 of size 4096 next 200 2025-05-15 18:36:39.701274: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463b200 of size 4096 next 201 2025-05-15 18:36:39.701287: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463c200 of size 1024 next 202 2025-05-15 18:36:39.701299: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463c600 of size 1024 next 204 2025-05-15 18:36:39.701312: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463ca00 of size 1024 next 205 2025-05-15 18:36:39.701326: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463ce00 of size 1024 next 206 2025-05-15 18:36:39.701340: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463d200 of size 1024 next 207 2025-05-15 18:36:39.701354: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463d600 of size 1024 next 210 2025-05-15 18:36:39.701368: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463da00 of size 1024 next 211 2025-05-15 18:36:39.701380: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463de00 of size 1024 next 212 2025-05-15 18:36:39.701392: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463e200 of size 1024 next 213 2025-05-15 18:36:39.701405: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463e600 of size 1024 next 214 2025-05-15 18:36:39.701417: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463ea00 of size 4096 next 215 2025-05-15 18:36:39.701429: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9463fa00 of size 4096 next 217 2025-05-15 18:36:39.701442: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94640a00 of size 4096 next 218 2025-05-15 18:36:39.701454: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94641a00 of size 4096 next 219 2025-05-15 18:36:39.701466: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94642a00 of size 4096 next 220 2025-05-15 18:36:39.701479: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94643a00 of size 1024 next 221 2025-05-15 18:36:39.701491: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94643e00 of size 1024 next 223 2025-05-15 18:36:39.701504: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94644200 of size 1024 next 224 2025-05-15 18:36:39.701516: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94644600 of size 1024 next 225 2025-05-15 18:36:39.701528: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94644a00 of size 1024 next 226 2025-05-15 18:36:39.701541: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94644e00 of size 1024 next 229 2025-05-15 18:36:39.701553: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94645200 of size 1024 next 230 2025-05-15 18:36:39.701565: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94645600 of size 1024 next 231 2025-05-15 18:36:39.701578: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94645a00 of size 1024 next 232 2025-05-15 18:36:39.701590: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94645e00 of size 1024 next 233 2025-05-15 18:36:39.701603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94646200 of size 7680 next 72 2025-05-15 18:36:39.701615: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94648000 of size 147456 next 71 2025-05-15 18:36:39.701628: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e9466c000 of size 131072 next 87 2025-05-15 18:36:39.701640: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9e9468c000 of size 262144 next 103 2025-05-15 18:36:39.701653: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e946cc000 of size 262144 next 102 2025-05-15 18:36:39.701665: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9e9470c000 of size 999424 next 18446744073709551615 2025-05-15 18:36:39.701686: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304 2025-05-15 18:36:39.701699: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94800000 of size 589824 next 96 2025-05-15 18:36:39.701712: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94890000 of size 262144 next 117 2025-05-15 18:36:39.701724: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9e948d0000 of size 262144 next 112 2025-05-15 18:36:39.701737: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94910000 of size 524288 next 111 2025-05-15 18:36:39.701750: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94990000 of size 262144 next 135 2025-05-15 18:36:39.701762: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e949d0000 of size 327680 next 124 2025-05-15 18:36:39.701775: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94a20000 of size 589824 next 123 2025-05-15 18:36:39.701787: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94ab0000 of size 262144 next 152 2025-05-15 18:36:39.701800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94af0000 of size 327680 next 142 2025-05-15 18:36:39.701813: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94b40000 of size 786432 next 18446744073709551615 2025-05-15 18:36:39.701825: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608 2025-05-15 18:36:39.701837: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9e94c00000 of size 589824 next 160 2025-05-15 18:36:39.701850: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94c90000 of size 589824 next 158 2025-05-15 18:36:39.701862: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94d20000 of size 524288 next 171 2025-05-15 18:36:39.701875: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9e94da0000 of size 1048576 next 222 2025-05-15 18:36:39.701887: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94ea0000 of size 1048576 next 190 2025-05-15 18:36:39.701900: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e94fa0000 of size 1048576 next 189 2025-05-15 18:36:39.701912: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e950a0000 of size 1048576 next 203 2025-05-15 18:36:39.701925: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9e951a0000 of size 2490368 next 18446744073709551615 2025-05-15 18:36:39.701937: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576 2025-05-15 18:36:39.701950: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00000 of size 1280 next 1 2025-05-15 18:36:39.701963: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00500 of size 256 next 5 2025-05-15 18:36:39.701975: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00600 of size 256 next 8 2025-05-15 18:36:39.701987: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00700 of size 256 next 9 2025-05-15 18:36:39.702000: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00800 of size 256 next 10 2025-05-15 18:36:39.702012: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00900 of size 256 next 11 2025-05-15 18:36:39.702025: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00a00 of size 256 next 12 2025-05-15 18:36:39.702037: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00b00 of size 256 next 13 2025-05-15 18:36:39.702049: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00c00 of size 256 next 17 2025-05-15 18:36:39.702062: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00d00 of size 256 next 19 2025-05-15 18:36:39.702074: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00e00 of size 256 next 20 2025-05-15 18:36:39.702095: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee00f00 of size 256 next 21 2025-05-15 18:36:39.702108: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01000 of size 256 next 22 2025-05-15 18:36:39.702120: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01100 of size 256 next 24 2025-05-15 18:36:39.702132: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01200 of size 256 next 25 2025-05-15 18:36:39.702145: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01300 of size 256 next 23 2025-05-15 18:36:39.702157: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01400 of size 256 next 28 2025-05-15 18:36:39.702169: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01500 of size 256 next 29 2025-05-15 18:36:39.702182: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01600 of size 256 next 30 2025-05-15 18:36:39.702194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01700 of size 256 next 31 2025-05-15 18:36:39.702206: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01800 of size 256 next 33 2025-05-15 18:36:39.702219: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01900 of size 256 next 34 2025-05-15 18:36:39.702231: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01a00 of size 1024 next 32 2025-05-15 18:36:39.702243: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee01e00 of size 1024 next 37 2025-05-15 18:36:39.702256: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee02200 of size 1024 next 38 2025-05-15 18:36:39.702268: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee02600 of size 1024 next 39 2025-05-15 18:36:39.702280: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee02a00 of size 1024 next 40 2025-05-15 18:36:39.702293: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee02e00 of size 1024 next 41 2025-05-15 18:36:39.702305: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee03200 of size 1024 next 43 2025-05-15 18:36:39.702317: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee03600 of size 1024 next 44 2025-05-15 18:36:39.702330: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee03a00 of size 1024 next 45 2025-05-15 18:36:39.702342: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee03e00 of size 1024 next 46 2025-05-15 18:36:39.702355: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04200 of size 256 next 48 2025-05-15 18:36:39.702367: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04300 of size 256 next 49 2025-05-15 18:36:39.702379: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04400 of size 256 next 50 2025-05-15 18:36:39.702392: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04500 of size 256 next 51 2025-05-15 18:36:39.702404: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04600 of size 256 next 52 2025-05-15 18:36:39.702416: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04700 of size 256 next 53 2025-05-15 18:36:39.702428: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04800 of size 256 next 56 2025-05-15 18:36:39.702441: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04900 of size 256 next 57 2025-05-15 18:36:39.702453: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04a00 of size 256 next 58 2025-05-15 18:36:39.702465: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04b00 of size 256 next 14 2025-05-15 18:36:39.702478: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04c00 of size 256 next 15 2025-05-15 18:36:39.702490: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04d00 of size 256 next 16 2025-05-15 18:36:39.702511: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee04e00 of size 1024 next 60 2025-05-15 18:36:39.702523: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee05200 of size 1024 next 61 2025-05-15 18:36:39.702536: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee05600 of size 1024 next 62 2025-05-15 18:36:39.702548: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee05a00 of size 1024 next 63 2025-05-15 18:36:39.702561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee05e00 of size 1024 next 64 2025-05-15 18:36:39.702573: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06200 of size 256 next 66 2025-05-15 18:36:39.702585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06300 of size 256 next 67 2025-05-15 18:36:39.702598: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06400 of size 256 next 68 2025-05-15 18:36:39.702610: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06500 of size 256 next 69 2025-05-15 18:36:39.702623: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06600 of size 256 next 70 2025-05-15 18:36:39.702635: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06700 of size 256 next 73 2025-05-15 18:36:39.702647: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06800 of size 256 next 74 2025-05-15 18:36:39.702660: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06900 of size 256 next 75 2025-05-15 18:36:39.702672: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06a00 of size 256 next 76 2025-05-15 18:36:39.702684: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06b00 of size 256 next 77 2025-05-15 18:36:39.702697: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee06c00 of size 1024 next 79 2025-05-15 18:36:39.702709: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee07000 of size 1024 next 80 2025-05-15 18:36:39.702721: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee07400 of size 1024 next 81 2025-05-15 18:36:39.702734: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee07800 of size 1024 next 82 2025-05-15 18:36:39.702746: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee07c00 of size 1024 next 83 2025-05-15 18:36:39.702759: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08000 of size 256 next 85 2025-05-15 18:36:39.702771: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08100 of size 256 next 86 2025-05-15 18:36:39.702784: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08200 of size 512 next 84 2025-05-15 18:36:39.702796: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08400 of size 512 next 88 2025-05-15 18:36:39.702809: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08600 of size 512 next 89 2025-05-15 18:36:39.702821: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08800 of size 512 next 90 2025-05-15 18:36:39.702833: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08a00 of size 512 next 91 2025-05-15 18:36:39.702846: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08c00 of size 256 next 93 2025-05-15 18:36:39.702858: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08d00 of size 256 next 94 2025-05-15 18:36:39.702871: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee08e00 of size 512 next 92 2025-05-15 18:36:39.702883: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09000 of size 512 next 97 2025-05-15 18:36:39.702913: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09200 of size 512 next 98 2025-05-15 18:36:39.702932: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09400 of size 512 next 99 2025-05-15 18:36:39.702954: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09600 of size 512 next 2 2025-05-15 18:36:39.702967: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09800 of size 256 next 3 2025-05-15 18:36:39.702979: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09900 of size 256 next 4 2025-05-15 18:36:39.702992: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee09a00 of size 16384 next 18 2025-05-15 18:36:39.703005: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0da00 of size 256 next 100 2025-05-15 18:36:39.703017: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0db00 of size 256 next 101 2025-05-15 18:36:39.703030: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0dc00 of size 2048 next 104 2025-05-15 18:36:39.703042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0e400 of size 2048 next 105 2025-05-15 18:36:39.703055: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0ec00 of size 2048 next 106 2025-05-15 18:36:39.703067: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0f400 of size 2048 next 107 2025-05-15 18:36:39.703079: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee0fc00 of size 2048 next 108 2025-05-15 18:36:39.703092: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee10400 of size 256 next 109 2025-05-15 18:36:39.703104: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee10500 of size 256 next 110 2025-05-15 18:36:39.703116: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee10600 of size 2048 next 113 2025-05-15 18:36:39.703129: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee10e00 of size 2048 next 114 2025-05-15 18:36:39.703141: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee11600 of size 2048 next 115 2025-05-15 18:36:39.703153: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee11e00 of size 2048 next 116 2025-05-15 18:36:39.703166: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee12600 of size 2048 next 6 2025-05-15 18:36:39.703178: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee12e00 of size 37632 next 7 2025-05-15 18:36:39.703191: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1c100 of size 512 next 118 2025-05-15 18:36:39.703203: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1c300 of size 512 next 119 2025-05-15 18:36:39.703216: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1c500 of size 512 next 120 2025-05-15 18:36:39.703228: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1c700 of size 512 next 121 2025-05-15 18:36:39.703240: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1c900 of size 512 next 122 2025-05-15 18:36:39.703253: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1cb00 of size 512 next 125 2025-05-15 18:36:39.703265: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1cd00 of size 512 next 126 2025-05-15 18:36:39.703277: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1cf00 of size 512 next 127 2025-05-15 18:36:39.703289: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1d100 of size 512 next 128 2025-05-15 18:36:39.703302: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1d300 of size 512 next 129 2025-05-15 18:36:39.703314: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1d500 of size 2048 next 130 2025-05-15 18:36:39.703326: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1dd00 of size 2048 next 131 2025-05-15 18:36:39.703339: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1e500 of size 2048 next 132 2025-05-15 18:36:39.703359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1ed00 of size 2048 next 133 2025-05-15 18:36:39.703372: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1f500 of size 2048 next 134 2025-05-15 18:36:39.703384: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1fd00 of size 512 next 136 2025-05-15 18:36:39.703397: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee1ff00 of size 512 next 137 2025-05-15 18:36:39.703409: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20100 of size 512 next 138 2025-05-15 18:36:39.703421: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20300 of size 512 next 139 2025-05-15 18:36:39.703434: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20500 of size 512 next 140 2025-05-15 18:36:39.703446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20700 of size 512 next 141 2025-05-15 18:36:39.703458: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20900 of size 512 next 143 2025-05-15 18:36:39.703471: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20b00 of size 512 next 144 2025-05-15 18:36:39.703483: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20d00 of size 512 next 145 2025-05-15 18:36:39.703495: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee20f00 of size 512 next 146 2025-05-15 18:36:39.703508: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee21100 of size 2048 next 147 2025-05-15 18:36:39.703520: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee21900 of size 2048 next 148 2025-05-15 18:36:39.703533: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee22100 of size 2048 next 149 2025-05-15 18:36:39.703545: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee22900 of size 2048 next 150 2025-05-15 18:36:39.703557: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee23100 of size 2048 next 151 2025-05-15 18:36:39.703570: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee23900 of size 512 next 153 2025-05-15 18:36:39.703582: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee23b00 of size 512 next 154 2025-05-15 18:36:39.703594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee23d00 of size 512 next 155 2025-05-15 18:36:39.703607: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee23f00 of size 512 next 156 2025-05-15 18:36:39.703619: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24100 of size 512 next 157 2025-05-15 18:36:39.703631: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24300 of size 512 next 161 2025-05-15 18:36:39.703643: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24500 of size 512 next 162 2025-05-15 18:36:39.703656: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24700 of size 512 next 163 2025-05-15 18:36:39.703668: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24900 of size 512 next 164 2025-05-15 18:36:39.703680: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24b00 of size 512 next 165 2025-05-15 18:36:39.703693: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee24d00 of size 2048 next 166 2025-05-15 18:36:39.703705: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee25500 of size 2048 next 167 2025-05-15 18:36:39.703718: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee25d00 of size 2048 next 168 2025-05-15 18:36:39.703730: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee26500 of size 2048 next 169 2025-05-15 18:36:39.703742: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee26d00 of size 2048 next 170 2025-05-15 18:36:39.703755: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee27500 of size 1024 next 172 2025-05-15 18:36:39.703776: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee27900 of size 1024 next 173 2025-05-15 18:36:39.703789: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee27d00 of size 1024 next 174 2025-05-15 18:36:39.703801: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee28100 of size 1024 next 175 2025-05-15 18:36:39.703814: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee28500 of size 1024 next 176 2025-05-15 18:36:39.703826: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee28900 of size 256 next 178 2025-05-15 18:36:39.703838: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee28a00 of size 256 next 179 2025-05-15 18:36:39.703851: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee28b00 of size 1024 next 177 2025-05-15 18:36:39.703863: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee28f00 of size 1024 next 182 2025-05-15 18:36:39.703875: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee29300 of size 1024 next 183 2025-05-15 18:36:39.703888: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee29700 of size 1024 next 184 2025-05-15 18:36:39.703900: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee29b00 of size 1024 next 185 2025-05-15 18:36:39.703912: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee29f00 of size 256 next 187 2025-05-15 18:36:39.703925: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee2a000 of size 256 next 188 2025-05-15 18:36:39.703937: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee2a100 of size 4096 next 186 2025-05-15 18:36:39.703950: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee2b100 of size 4096 next 42 2025-05-15 18:36:39.703962: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee2c100 of size 65536 next 36 2025-05-15 18:36:39.703974: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee3c100 of size 65536 next 35 2025-05-15 18:36:39.703987: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee4c100 of size 4096 next 235 2025-05-15 18:36:39.703999: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee4d100 of size 4096 next 236 2025-05-15 18:36:39.704011: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee4e100 of size 4096 next 237 2025-05-15 18:36:39.704023: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee4f100 of size 4096 next 238 2025-05-15 18:36:39.704036: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee50100 of size 1024 next 239 2025-05-15 18:36:39.704048: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee50500 of size 1024 next 240 2025-05-15 18:36:39.704060: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee50900 of size 1024 next 241 2025-05-15 18:36:39.704073: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee50d00 of size 1024 next 242 2025-05-15 18:36:39.704085: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee51100 of size 1024 next 243 2025-05-15 18:36:39.704097: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f9fcee51500 of size 76800 next 27 2025-05-15 18:36:39.704110: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee64100 of size 147456 next 26 2025-05-15 18:36:39.704122: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee88100 of size 65536 next 47 2025-05-15 18:36:39.704135: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fcee98100 of size 65536 next 59 2025-05-15 18:36:39.704147: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fceea8100 of size 65536 next 65 2025-05-15 18:36:39.704160: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f9fceeb8100 of size 294656 next 18446744073709551615 2025-05-15 18:36:39.704181: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size: 2025-05-15 18:36:39.704197: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 59 Chunks of size 256 totalling 14.8KiB 2025-05-15 18:36:39.704212: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB 2025-05-15 18:36:39.704226: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 55 Chunks of size 1024 totalling 55.0KiB 2025-05-15 18:36:39.704240: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB 2025-05-15 18:36:39.704254: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 2048 totalling 50.0KiB 2025-05-15 18:36:39.704268: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 19 Chunks of size 4096 totalling 76.0KiB 2025-05-15 18:36:39.704281: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB 2025-05-15 18:36:39.704295: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB 2025-05-15 18:36:39.704309: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB 2025-05-15 18:36:39.704322: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB 2025-05-15 18:36:39.704336: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB 2025-05-15 18:36:39.704350: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB 2025-05-15 18:36:39.704364: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB 2025-05-15 18:36:39.704378: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB 2025-05-15 18:36:39.704391: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB 2025-05-15 18:36:39.704405: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB 2025-05-15 18:36:39.704419: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB 2025-05-15 18:36:39.704432: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB 2025-05-15 18:36:39.704446: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 5 Chunks of size 1048576 totalling 5.00MiB 2025-05-15 18:36:39.704459: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1310720 totalling 1.25MiB 2025-05-15 18:36:39.704473: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2097152 totalling 2.00MiB 2025-05-15 18:36:39.704486: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 2359296 totalling 6.75MiB 2025-05-15 18:36:39.704500: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2490368 totalling 2.38MiB 2025-05-15 18:36:39.704513: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2752512 totalling 2.62MiB 2025-05-15 18:36:39.704527: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 26.54MiB 2025-05-15 18:36:39.704539: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 32374784 memory_limit_: 32374784 available bytes: 0 curr_region_allocation_bytes_: 33554432 2025-05-15 18:36:39.704557: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: Limit: 32374784 InUse: 27825152 MaxInUse: 27825408 NumAllocs: 763 MaxAllocSize: 4325376 2025-05-15 18:36:39.704599: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ******************___**********************************__**************_*****__********************* 2025-05-15 18:36:39.704657: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 Exception in mask_detect : OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] we want to redo the detection Using TensorFlow backend. max_time_sub_proc : 3600 erreur pendant la detection Useless call to update_current_state in case -12 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! ERROR : mask output needs to be a dictionnary now ! No output to save, continue without doing anything ! save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : -12 ERROR : 'int' object is not subscriptable reconnect to base ! warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data #&_# TEST FAILED #&_# : tests/mask_test #&_# Error : invalid literal for int() with base 10: "'int' object is not subscriptable" /home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/master_6b55ca732dc5789dd54794d418371b3abe63c01d SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/master_6b55ca732dc5789dd54794d418371b3abe63c01d','{"mask_detection": "fail"}','0','http://marlene.fotonower-preprod.com/job/2025/May/15052025/python_test3//data_2/data_log/job/2025/May/15052025/python_test3/log-python3----short_python3--v--marlene-18:35:01.txt','mask_detection','unknown'); #&_# END OF TEST #&_# : tests/mask_test #&_# #&_# BEGIN OF TEST : tests/datou_test #&_# /home/admin/workarea/git/Velours/python/tests/datou_test.py Datou All Test python version used : 3 ############################### TEST sam ################################ TEST SAM Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4573 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4573 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4573 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : sam list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1189321094) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 1189321094 download finish for photo 1189321094 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.5755302906036377 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! WARNING : we have an input that is not a photo, we should get rid of it Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:sam Thu May 15 18:36:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1747327000_403103_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094} map_photo_id_path_extension : {1189321094: {'path': 'temp/1747327000_403103_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}} map_subphoto_mainphoto : {} Beginning of datou step sam ! pht : 4677 Inside sam : nb paths : 1 (640, 960, 3) time for calcul the mask position with numpy : 0.002605915069580078 nb_pixel_total : 3755 time to create 1 rle with old method : 0.0045986175537109375 time for calcul the mask position with numpy : 0.0020401477813720703 nb_pixel_total : 39231 time to create 1 rle with old method : 0.0460054874420166 time for calcul the mask position with numpy : 0.0016796588897705078 nb_pixel_total : 13940 time to create 1 rle with old method : 0.01629328727722168 time for calcul the mask position with numpy : 0.0015103816986083984 nb_pixel_total : 5629 time to create 1 rle with old method : 0.00669550895690918 time for calcul the mask position with numpy : 0.0015566349029541016 nb_pixel_total : 16399 time to create 1 rle with old method : 0.019222736358642578 time for calcul the mask position with numpy : 0.0024383068084716797 nb_pixel_total : 84345 time to create 1 rle with old method : 0.09595799446105957 time for calcul the mask position with numpy : 0.001674652099609375 nb_pixel_total : 29321 time to create 1 rle with old method : 0.035227298736572266 time for calcul the mask position with numpy : 0.0016109943389892578 nb_pixel_total : 2937 time to create 1 rle with old method : 0.0037314891815185547 time for calcul the mask position with numpy : 0.0014979839324951172 nb_pixel_total : 4264 time to create 1 rle with old method : 0.0049364566802978516 time for calcul the mask position with numpy : 0.0014491081237792969 nb_pixel_total : 1227 time to create 1 rle with old method : 0.0015158653259277344 time for calcul the mask position with numpy : 0.0015587806701660156 nb_pixel_total : 10819 time to create 1 rle with old method : 0.012618780136108398 time for calcul the mask position with numpy : 0.0014612674713134766 nb_pixel_total : 3952 time to create 1 rle with old method : 0.004975080490112305 time for calcul the mask position with numpy : 0.001508951187133789 nb_pixel_total : 2452 time to create 1 rle with old method : 0.0031583309173583984 time for calcul the mask position with numpy : 0.001524209976196289 nb_pixel_total : 6607 time to create 1 rle with old method : 0.008774280548095703 time for calcul the mask position with numpy : 0.001589059829711914 nb_pixel_total : 1246 time to create 1 rle with old method : 0.0016579627990722656 time for calcul the mask position with numpy : 0.0016133785247802734 nb_pixel_total : 692 time to create 1 rle with old method : 0.001134634017944336 time for calcul the mask position with numpy : 0.001844167709350586 nb_pixel_total : 16348 time to create 1 rle with old method : 0.022122621536254883 time for calcul the mask position with numpy : 0.001547098159790039 nb_pixel_total : 8414 time to create 1 rle with old method : 0.010595083236694336 time for calcul the mask position with numpy : 0.0017330646514892578 nb_pixel_total : 2079 time to create 1 rle with old method : 0.0027091503143310547 time for calcul the mask position with numpy : 0.0014846324920654297 nb_pixel_total : 13090 time to create 1 rle with old method : 0.015431404113769531 time for calcul the mask position with numpy : 0.002019166946411133 nb_pixel_total : 7610 time to create 1 rle with old method : 0.00913381576538086 time for calcul the mask position with numpy : 0.001482248306274414 nb_pixel_total : 7757 time to create 1 rle with old method : 0.009130477905273438 time for calcul the mask position with numpy : 0.001489400863647461 nb_pixel_total : 4291 time to create 1 rle with old method : 0.005763053894042969 time for calcul the mask position with numpy : 0.0015118122100830078 nb_pixel_total : 5497 time to create 1 rle with old method : 0.0065708160400390625 time for calcul the mask position with numpy : 0.00147247314453125 nb_pixel_total : 978 time to create 1 rle with old method : 0.0014595985412597656 time for calcul the mask position with numpy : 0.0018329620361328125 nb_pixel_total : 39007 time to create 1 rle with old method : 0.04708147048950195 time for calcul the mask position with numpy : 0.001531362533569336 nb_pixel_total : 8629 time to create 1 rle with old method : 0.010049819946289062 time for calcul the mask position with numpy : 0.001493692398071289 nb_pixel_total : 3532 time to create 1 rle with old method : 0.004266262054443359 time for calcul the mask position with numpy : 0.0015137195587158203 nb_pixel_total : 2385 time to create 1 rle with old method : 0.0030295848846435547 time for calcul the mask position with numpy : 0.0016052722930908203 nb_pixel_total : 11928 time to create 1 rle with old method : 0.013956308364868164 time for calcul the mask position with numpy : 0.0016350746154785156 nb_pixel_total : 14653 time to create 1 rle with old method : 0.01720714569091797 time for calcul the mask position with numpy : 0.0014994144439697266 nb_pixel_total : 2447 time to create 1 rle with old method : 0.0029833316802978516 time for calcul the mask position with numpy : 0.0014390945434570312 nb_pixel_total : 3860 time to create 1 rle with old method : 0.004735708236694336 time for calcul the mask position with numpy : 0.0014700889587402344 nb_pixel_total : 9883 time to create 1 rle with old method : 0.011753082275390625 time for calcul the mask position with numpy : 0.0014524459838867188 nb_pixel_total : 5426 time to create 1 rle with old method : 0.006531476974487305 time for calcul the mask position with numpy : 0.0014367103576660156 nb_pixel_total : 2727 time to create 1 rle with old method : 0.003350973129272461 time for calcul the mask position with numpy : 0.001485586166381836 nb_pixel_total : 13010 time to create 1 rle with old method : 0.019683837890625 time for calcul the mask position with numpy : 0.0015230178833007812 nb_pixel_total : 2782 time to create 1 rle with old method : 0.003339529037475586 time for calcul the mask position with numpy : 0.0014700889587402344 nb_pixel_total : 10643 time to create 1 rle with old method : 0.01274728775024414 time for calcul the mask position with numpy : 0.0014882087707519531 nb_pixel_total : 4102 time to create 1 rle with old method : 0.0049703121185302734 time for calcul the mask position with numpy : 0.0014345645904541016 nb_pixel_total : 3329 time to create 1 rle with old method : 0.003979921340942383 time for calcul the mask position with numpy : 0.0014243125915527344 nb_pixel_total : 1025 time to create 1 rle with old method : 0.0012617111206054688 time for calcul the mask position with numpy : 0.0014176368713378906 nb_pixel_total : 1648 time to create 1 rle with old method : 0.002007722854614258 time for calcul the mask position with numpy : 0.0014176368713378906 nb_pixel_total : 1638 time to create 1 rle with old method : 0.002069234848022461 time for calcul the mask position with numpy : 0.0014231204986572266 nb_pixel_total : 2028 time to create 1 rle with old method : 0.002468585968017578 time for calcul the mask position with numpy : 0.0014142990112304688 nb_pixel_total : 343 time to create 1 rle with old method : 0.000453948974609375 time for calcul the mask position with numpy : 0.0014674663543701172 nb_pixel_total : 4175 time to create 1 rle with old method : 0.005019187927246094 time for calcul the mask position with numpy : 0.0014214515686035156 nb_pixel_total : 851 time to create 1 rle with old method : 0.0011737346649169922 time for calcul the mask position with numpy : 0.0014166831970214844 nb_pixel_total : 874 time to create 1 rle with old method : 0.0011639595031738281 time for calcul the mask position with numpy : 0.0014271736145019531 nb_pixel_total : 595 time to create 1 rle with old method : 0.0010149478912353516 time for calcul the mask position with numpy : 0.0014297962188720703 nb_pixel_total : 2324 time to create 1 rle with old method : 0.0028023719787597656 time for calcul the mask position with numpy : 0.0014045238494873047 nb_pixel_total : 886 time to create 1 rle with old method : 0.0011341571807861328 time for calcul the mask position with numpy : 0.0013966560363769531 nb_pixel_total : 275 time to create 1 rle with old method : 0.000370025634765625 time for calcul the mask position with numpy : 0.0014209747314453125 nb_pixel_total : 577 time to create 1 rle with old method : 0.0007615089416503906 time for calcul the mask position with numpy : 0.001435995101928711 nb_pixel_total : 2407 time to create 1 rle with old method : 0.003158092498779297 time for calcul the mask position with numpy : 0.0014529228210449219 nb_pixel_total : 1704 time to create 1 rle with old method : 0.0021071434020996094 time for calcul the mask position with numpy : 0.001432180404663086 nb_pixel_total : 337 time to create 1 rle with old method : 0.0004673004150390625 time for calcul the mask position with numpy : 0.001447916030883789 nb_pixel_total : 2771 time to create 1 rle with old method : 0.003352642059326172 time for calcul the mask position with numpy : 0.0014395713806152344 nb_pixel_total : 1206 time to create 1 rle with old method : 0.0015041828155517578 time for calcul the mask position with numpy : 0.0014445781707763672 nb_pixel_total : 584 time to create 1 rle with old method : 0.0007634162902832031 time for calcul the mask position with numpy : 0.0014202594757080078 nb_pixel_total : 1068 time to create 1 rle with old method : 0.0013959407806396484 time for calcul the mask position with numpy : 0.001438140869140625 nb_pixel_total : 915 time to create 1 rle with old method : 0.0011217594146728516 time for calcul the mask position with numpy : 0.0013918876647949219 nb_pixel_total : 1075 time to create 1 rle with old method : 0.0013499259948730469 time for calcul the mask position with numpy : 0.0014452934265136719 nb_pixel_total : 3092 time to create 1 rle with old method : 0.0038177967071533203 time for calcul the mask position with numpy : 0.0015025138854980469 nb_pixel_total : 8600 time to create 1 rle with old method : 0.00987863540649414 time for calcul the mask position with numpy : 0.0015702247619628906 nb_pixel_total : 27716 time to create 1 rle with old method : 0.03211069107055664 time for calcul the mask position with numpy : 0.0014476776123046875 nb_pixel_total : 1742 time to create 1 rle with old method : 0.0020749568939208984 time for calcul the mask position with numpy : 0.0015587806701660156 nb_pixel_total : 16671 time to create 1 rle with old method : 0.01997971534729004 time for calcul the mask position with numpy : 0.0014340877532958984 nb_pixel_total : 3167 time to create 1 rle with old method : 0.003721952438354492 time for calcul the mask position with numpy : 0.0014612674713134766 nb_pixel_total : 9077 time to create 1 rle with old method : 0.010418891906738281 time for calcul the mask position with numpy : 0.0014383792877197266 nb_pixel_total : 1513 time to create 1 rle with old method : 0.0018036365509033203 time for calcul the mask position with numpy : 0.0014324188232421875 nb_pixel_total : 713 time to create 1 rle with old method : 0.0009472370147705078 time for calcul the mask position with numpy : 0.0014133453369140625 nb_pixel_total : 267 time to create 1 rle with old method : 0.0003464221954345703 time for calcul the mask position with numpy : 0.0014617443084716797 nb_pixel_total : 9503 time to create 1 rle with old method : 0.011035919189453125 time for calcul the mask position with numpy : 0.0014331340789794922 nb_pixel_total : 1321 time to create 1 rle with old method : 0.0016942024230957031 time for calcul the mask position with numpy : 0.0015299320220947266 nb_pixel_total : 18482 time to create 1 rle with old method : 0.021289825439453125 time for calcul the mask position with numpy : 0.0014514923095703125 nb_pixel_total : 970 time to create 1 rle with old method : 0.0011904239654541016 time for calcul the mask position with numpy : 0.001425027847290039 nb_pixel_total : 1126 time to create 1 rle with old method : 0.0013785362243652344 time for calcul the mask position with numpy : 0.0014176368713378906 nb_pixel_total : 616 time to create 1 rle with old method : 0.0008020401000976562 time for calcul the mask position with numpy : 0.0014104843139648438 nb_pixel_total : 248 time to create 1 rle with old method : 0.0003561973571777344 time for calcul the mask position with numpy : 0.0014209747314453125 nb_pixel_total : 733 time to create 1 rle with old method : 0.0010497570037841797 time for calcul the mask position with numpy : 0.0014150142669677734 nb_pixel_total : 222 time to create 1 rle with old method : 0.00033092498779296875 time for calcul the mask position with numpy : 0.0014500617980957031 nb_pixel_total : 4147 time to create 1 rle with old method : 0.00541996955871582 time for calcul the mask position with numpy : 0.0014262199401855469 nb_pixel_total : 1501 time to create 1 rle with old method : 0.0018591880798339844 time for calcul the mask position with numpy : 0.0014204978942871094 nb_pixel_total : 1637 time to create 1 rle with old method : 0.0020492076873779297 time for calcul the mask position with numpy : 0.001453399658203125 nb_pixel_total : 7499 time to create 1 rle with old method : 0.008766412734985352 time for calcul the mask position with numpy : 0.0014331340789794922 nb_pixel_total : 300 time to create 1 rle with old method : 0.0004611015319824219 time for calcul the mask position with numpy : 0.0014300346374511719 nb_pixel_total : 1092 time to create 1 rle with old method : 0.00138092041015625 time for calcul the mask position with numpy : 0.0014524459838867188 nb_pixel_total : 1442 time to create 1 rle with old method : 0.001840353012084961 time for calcul the mask position with numpy : 0.0014584064483642578 nb_pixel_total : 595 time to create 1 rle with old method : 0.0007960796356201172 time for calcul the mask position with numpy : 0.0014166831970214844 nb_pixel_total : 917 time to create 1 rle with old method : 0.0013117790222167969 time for calcul the mask position with numpy : 0.0014455318450927734 nb_pixel_total : 2197 time to create 1 rle with old method : 0.0028209686279296875 time for calcul the mask position with numpy : 0.0014219284057617188 nb_pixel_total : 889 time to create 1 rle with old method : 0.0011169910430908203 time for calcul the mask position with numpy : 0.0014815330505371094 nb_pixel_total : 1321 time to create 1 rle with old method : 0.0016736984252929688 time for calcul the mask position with numpy : 0.0014166831970214844 nb_pixel_total : 947 time to create 1 rle with old method : 0.0013959407806396484 time for calcul the mask position with numpy : 0.0017485618591308594 nb_pixel_total : 884 time to create 1 rle with old method : 0.0013992786407470703 time for calcul the mask position with numpy : 0.0017507076263427734 nb_pixel_total : 1374 time to create 1 rle with old method : 0.002171039581298828 time for calcul the mask position with numpy : 0.0017552375793457031 nb_pixel_total : 1614 time to create 1 rle with old method : 0.002447366714477539 time for calcul the mask position with numpy : 0.0014154911041259766 nb_pixel_total : 828 time to create 1 rle with old method : 0.0010645389556884766 insert ignore into MTRPhoto.crop_hashtag_ids (photo_id, hashtag_id, `type`,x0,x1,y0,y1,score) VALUES (%s,%s,%s,%s,%s,%s,%s,%s) batch 1 Loaded 99 chid ids of type : 4677 Number RLEs to save : 9171 INSERT IGNORE INTO MTRPhoto.crop_segments (`crop_hashtag_id`, `x0`, `y0`, `length`) VALUES (%s, %s, %s , %s) first line : ('3799452163', '202', '535', '8') ... last line : ('3799452261', '815', '44', '5') INSERT IGNORE INTO MTRPhoto.crop_sum_segments (`crop_hashtag_id`, `sum_segments`) VALUES (%s, %s) TO DO : save crop sub photo not yet done ! After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : True verbose : True saveOutput not yet implemented for datou_step.type : sam we use saveGeneral [1189321094] map_info['map_portfolio_photo'] : {} final : True mtd_id 4573 list_pids : [1189321094] Looping around the photos to save general results len do output : 1 /1189321094Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4573', None, None, None, None, None, None, None, None) ('4573', None, '1189321094', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4573', None, '1189321094', 'None', None, None, None, None, None)] time used for this insertion : 0.012125492095947266 save_final save missing photos in datou_result : time spend for datou_step_exec : 10.768210649490356 time spend to save output : 0.012449264526367188 total time spend for step 1 : 10.780659914016724 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'1189321094': [[, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], 'temp/1747327000_403103_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png']} nb_objects detect : 99 ############################### TEST frcnn ################################ test frcnn Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4184 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4184 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4184 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : frcnn list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917754606) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917754606 download finish for photo 917754606 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.16552329063415527 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:frcnn Thu May 15 18:36:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1747327011_403103_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606} map_photo_id_path_extension : {917754606: {'path': 'temp/1747327011_403103_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Faster rcnn ! classes : ['background', 'plaque'] pht : 4370 caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'} To loadFromThcl() model_param file didn't exist model_name : detection_plaque_valcor_010622 model_type : caffe_faster_rcnn list file need : ['caffemodel', 'test.prototxt'] file exist in s3 : ['caffemodel', 'test.prototxt'] file manque in s3 : [] WARNING: Logging before InitGoogleLogging() is written to STDERR F0515 18:36:53.706946 403103 syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** Command terminated by signal 6 11.71user 9.50system 1:28.94elapsed 23%CPU (0avgtext+0avgdata 3466556maxresident)k 2904800inputs+3440outputs (7820major+1253263minor)pagefaults 0swaps