Hugging Face's logo

Surabhi-K
/
working

PEFT
Safetensors
Generated from Trainer
Model card Files Community
Surabhi-K commited on
Commit
d560551
•
1 Parent(s): 41a8a0d

Surabhi-K/code_llama_library2

Browse files
Files changed (7) hide show
  1. outputs_library.zip +3 -0
  2. wandb/debug-internal.log +663 -0
  3. wandb/debug.log +60 -0
  4. wandb/run-20240416_061855-mrfkrks2/files/output.log +1 -0
  5. wandb/run-20240416_061855-mrfkrks2/logs/debug-internal.log +663 -0
  6. wandb/run-20240416_061855-mrfkrks2/logs/debug.log +60 -0
  7. wandb/run-20240416_061855-mrfkrks2/run-mrfkrks2.wandb +0 -0
outputs_library.zip ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5e4ee88d37f55dc8c4972b83ba3bfc7ea1e52dcedefc5ce4d1b69ee96fbaf958
3
+ size 2456003098
wandb/debug-internal.log CHANGED
@@ -8804,3 +8804,666 @@ subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after
8804
  2024-04-16 11:55:29,199 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8805
  2024-04-16 11:55:34,200 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8806
  2024-04-16 11:55:39,201 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8804
  2024-04-16 11:55:29,199 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8805
  2024-04-16 11:55:34,200 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8806
  2024-04-16 11:55:39,201 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8807
+ 2024-04-16 11:55:43,071 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8808
+ 2024-04-16 11:55:43,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8809
+ 2024-04-16 11:55:43,073 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8810
+ 2024-04-16 11:55:44,225 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8811
+ 2024-04-16 11:55:49,226 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8812
+ 2024-04-16 11:55:54,226 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8813
+ 2024-04-16 11:55:58,071 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8814
+ 2024-04-16 11:55:58,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8815
+ 2024-04-16 11:55:58,073 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8816
+ 2024-04-16 11:56:00,207 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8817
+ 2024-04-16 11:56:05,208 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8818
+ 2024-04-16 11:56:09,162 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8819
+ 2024-04-16 11:56:09,162 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8820
+ 2024-04-16 11:56:09,162 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8821
+ 2024-04-16 11:56:09,162 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8822
+ 2024-04-16 11:56:09,162 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8823
+ 2024-04-16 11:56:09,162 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8824
+ 2024-04-16 11:56:09,163 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8825
+ 2024-04-16 11:56:09,163 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8826
+ 2024-04-16 11:56:09,178 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8827
+ 2024-04-16 11:56:09,179 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8828
+ 2024-04-16 11:56:09,179 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8829
+ 2024-04-16 11:56:09,179 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8830
+ 2024-04-16 11:56:09,179 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8831
+ 2024-04-16 11:56:09,179 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8832
+ 2024-04-16 11:56:09,180 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8833
+ 2024-04-16 11:56:09,180 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8834
+ 2024-04-16 11:56:09,181 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8835
+ 2024-04-16 11:56:09,183 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8836
+ 2024-04-16 11:56:09,183 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8837
+ 2024-04-16 11:56:09,184 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8838
+ 2024-04-16 11:56:09,185 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8839
+ 2024-04-16 11:56:11,182 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8840
+ 2024-04-16 11:56:13,071 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8841
+ 2024-04-16 11:56:13,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8842
+ 2024-04-16 11:56:13,074 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8843
+ 2024-04-16 11:56:16,203 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8844
+ 2024-04-16 11:56:21,203 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8845
+ 2024-04-16 11:56:24,890 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8846
+ 2024-04-16 11:56:24,890 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8847
+ 2024-04-16 11:56:24,890 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8848
+ 2024-04-16 11:56:24,891 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8849
+ 2024-04-16 11:56:24,891 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8850
+ 2024-04-16 11:56:24,891 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8851
+ 2024-04-16 11:56:24,891 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8852
+ 2024-04-16 11:56:24,891 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8853
+ 2024-04-16 11:56:24,906 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8854
+ 2024-04-16 11:56:24,906 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8855
+ 2024-04-16 11:56:24,906 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8856
+ 2024-04-16 11:56:24,906 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8857
+ 2024-04-16 11:56:24,907 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8858
+ 2024-04-16 11:56:24,907 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8859
+ 2024-04-16 11:56:24,907 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8860
+ 2024-04-16 11:56:24,907 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8861
+ 2024-04-16 11:56:24,908 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8862
+ 2024-04-16 11:56:24,909 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8863
+ 2024-04-16 11:56:24,909 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8864
+ 2024-04-16 11:56:24,910 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8865
+ 2024-04-16 11:56:24,911 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8866
+ 2024-04-16 11:56:25,089 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8867
+ 2024-04-16 11:56:25,089 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8868
+ 2024-04-16 11:56:25,089 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8869
+ 2024-04-16 11:56:25,090 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8870
+ 2024-04-16 11:56:25,090 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8871
+ 2024-04-16 11:56:25,090 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8872
+ 2024-04-16 11:56:25,090 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8873
+ 2024-04-16 11:56:25,090 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8874
+ 2024-04-16 11:56:25,103 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8875
+ 2024-04-16 11:56:25,103 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8876
+ 2024-04-16 11:56:25,103 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8877
+ 2024-04-16 11:56:25,103 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8878
+ 2024-04-16 11:56:25,103 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8879
+ 2024-04-16 11:56:25,104 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8880
+ 2024-04-16 11:56:25,104 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8881
+ 2024-04-16 11:56:25,104 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8882
+ 2024-04-16 11:56:25,105 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8883
+ 2024-04-16 11:56:25,105 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8884
+ 2024-04-16 11:56:25,106 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8885
+ 2024-04-16 11:56:25,106 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8886
+ 2024-04-16 11:56:25,107 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8887
+ 2024-04-16 11:56:25,469 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8888
+ 2024-04-16 11:56:25,470 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8889
+ 2024-04-16 11:56:25,470 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8890
+ 2024-04-16 11:56:25,470 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8891
+ 2024-04-16 11:56:25,471 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8892
+ 2024-04-16 11:56:25,471 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8893
+ 2024-04-16 11:56:25,471 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8894
+ 2024-04-16 11:56:25,472 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8895
+ 2024-04-16 11:56:25,483 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8896
+ 2024-04-16 11:56:25,483 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8897
+ 2024-04-16 11:56:25,483 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8898
+ 2024-04-16 11:56:25,484 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8899
+ 2024-04-16 11:56:26,484 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8900
+ 2024-04-16 11:56:28,071 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8901
+ 2024-04-16 11:56:28,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8902
+ 2024-04-16 11:56:28,073 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8903
+ 2024-04-16 11:56:32,175 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8904
+ 2024-04-16 11:56:37,176 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8905
+ 2024-04-16 11:56:37,217 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8906
+ 2024-04-16 11:56:37,217 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8907
+ 2024-04-16 11:56:37,217 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8908
+ 2024-04-16 11:56:37,217 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8909
+ 2024-04-16 11:56:37,218 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8910
+ 2024-04-16 11:56:37,218 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8911
+ 2024-04-16 11:56:37,219 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8912
+ 2024-04-16 11:56:37,220 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8913
+ 2024-04-16 11:56:37,220 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8914
+ 2024-04-16 11:56:37,330 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8915
+ 2024-04-16 11:56:37,330 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8916
+ 2024-04-16 11:56:37,330 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8917
+ 2024-04-16 11:56:37,330 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8918
+ 2024-04-16 11:56:37,330 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8919
+ 2024-04-16 11:56:37,331 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8920
+ 2024-04-16 11:56:37,331 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8921
+ 2024-04-16 11:56:37,331 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8922
+ 2024-04-16 11:56:37,342 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8923
+ 2024-04-16 11:56:37,342 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8924
+ 2024-04-16 11:56:37,342 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8925
+ 2024-04-16 11:56:37,343 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8926
+ 2024-04-16 11:56:42,344 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8927
+ 2024-04-16 11:56:43,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8928
+ 2024-04-16 11:56:43,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8929
+ 2024-04-16 11:56:43,074 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8930
+ 2024-04-16 11:56:48,162 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8931
+ 2024-04-16 11:56:53,163 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8932
+ 2024-04-16 11:56:55,143 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8933
+ 2024-04-16 11:56:55,143 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8934
+ 2024-04-16 11:56:55,143 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8935
+ 2024-04-16 11:56:55,143 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8936
+ 2024-04-16 11:56:55,144 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8937
+ 2024-04-16 11:56:55,145 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8938
+ 2024-04-16 11:56:55,146 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8939
+ 2024-04-16 11:56:55,147 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8940
+ 2024-04-16 11:56:55,147 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8941
+ 2024-04-16 11:56:58,018 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8942
+ 2024-04-16 11:56:58,018 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8943
+ 2024-04-16 11:56:58,018 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8944
+ 2024-04-16 11:56:58,018 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8945
+ 2024-04-16 11:56:58,018 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8946
+ 2024-04-16 11:56:58,018 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8947
+ 2024-04-16 11:56:58,019 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8948
+ 2024-04-16 11:56:58,019 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8949
+ 2024-04-16 11:56:58,031 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8950
+ 2024-04-16 11:56:58,031 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8951
+ 2024-04-16 11:56:58,032 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8952
+ 2024-04-16 11:56:58,032 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8953
+ 2024-04-16 11:56:58,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8954
+ 2024-04-16 11:56:58,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8955
+ 2024-04-16 11:56:58,074 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8956
+ 2024-04-16 11:56:58,617 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8957
+ 2024-04-16 11:57:03,618 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8958
+ 2024-04-16 11:57:08,619 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8959
+ 2024-04-16 11:57:11,340 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8960
+ 2024-04-16 11:57:11,340 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8961
+ 2024-04-16 11:57:11,340 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8962
+ 2024-04-16 11:57:11,341 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8963
+ 2024-04-16 11:57:11,341 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8964
+ 2024-04-16 11:57:11,342 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8965
+ 2024-04-16 11:57:11,343 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8966
+ 2024-04-16 11:57:11,343 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8967
+ 2024-04-16 11:57:11,344 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8968
+ 2024-04-16 11:57:11,918 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8969
+ 2024-04-16 11:57:11,918 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8970
+ 2024-04-16 11:57:11,918 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8971
+ 2024-04-16 11:57:11,919 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8972
+ 2024-04-16 11:57:11,919 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8973
+ 2024-04-16 11:57:11,920 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8974
+ 2024-04-16 11:57:11,920 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8975
+ 2024-04-16 11:57:11,920 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8976
+ 2024-04-16 11:57:11,931 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8977
+ 2024-04-16 11:57:11,931 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8978
+ 2024-04-16 11:57:11,931 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8979
+ 2024-04-16 11:57:11,932 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8980
+ 2024-04-16 11:57:13,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8981
+ 2024-04-16 11:57:13,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8982
+ 2024-04-16 11:57:13,074 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8983
+ 2024-04-16 11:57:14,177 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8984
+ 2024-04-16 11:57:14,597 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8985
+ 2024-04-16 11:57:14,597 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8986
+ 2024-04-16 11:57:14,597 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8987
+ 2024-04-16 11:57:14,597 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8988
+ 2024-04-16 11:57:14,598 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8989
+ 2024-04-16 11:57:14,599 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8990
+ 2024-04-16 11:57:14,600 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8991
+ 2024-04-16 11:57:14,601 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8992
+ 2024-04-16 11:57:14,602 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8993
+ 2024-04-16 11:57:15,118 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8994
+ 2024-04-16 11:57:15,118 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8995
+ 2024-04-16 11:57:15,118 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8996
+ 2024-04-16 11:57:15,118 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8997
+ 2024-04-16 11:57:15,119 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8998
+ 2024-04-16 11:57:15,119 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8999
+ 2024-04-16 11:57:15,119 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9000
+ 2024-04-16 11:57:15,120 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9001
+ 2024-04-16 11:57:15,131 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9002
+ 2024-04-16 11:57:15,132 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9003
+ 2024-04-16 11:57:15,132 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9004
+ 2024-04-16 11:57:15,132 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9005
+ 2024-04-16 11:57:20,133 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9006
+ 2024-04-16 11:57:23,879 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9007
+ 2024-04-16 11:57:23,879 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9008
+ 2024-04-16 11:57:23,879 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9009
+ 2024-04-16 11:57:23,880 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9010
+ 2024-04-16 11:57:23,880 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9011
+ 2024-04-16 11:57:23,881 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9012
+ 2024-04-16 11:57:23,882 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9013
+ 2024-04-16 11:57:23,882 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9014
+ 2024-04-16 11:57:23,883 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9015
+ 2024-04-16 11:57:24,280 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9016
+ 2024-04-16 11:57:24,281 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9017
+ 2024-04-16 11:57:24,281 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9018
+ 2024-04-16 11:57:24,281 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9019
+ 2024-04-16 11:57:24,282 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9020
+ 2024-04-16 11:57:24,282 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9021
+ 2024-04-16 11:57:24,282 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9022
+ 2024-04-16 11:57:24,283 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9023
+ 2024-04-16 11:57:24,294 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9024
+ 2024-04-16 11:57:24,294 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9025
+ 2024-04-16 11:57:24,294 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9026
+ 2024-04-16 11:57:24,294 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9027
+ 2024-04-16 11:57:25,295 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9028
+ 2024-04-16 11:57:28,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9029
+ 2024-04-16 11:57:28,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9030
+ 2024-04-16 11:57:28,074 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9031
+ 2024-04-16 11:57:31,197 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9032
+ 2024-04-16 11:57:33,336 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9033
+ 2024-04-16 11:57:33,336 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9034
+ 2024-04-16 11:57:33,336 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9035
+ 2024-04-16 11:57:33,336 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9036
+ 2024-04-16 11:57:33,337 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9037
+ 2024-04-16 11:57:33,337 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9038
+ 2024-04-16 11:57:33,338 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9039
+ 2024-04-16 11:57:33,340 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9040
+ 2024-04-16 11:57:33,341 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9041
+ 2024-04-16 11:57:33,738 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9042
+ 2024-04-16 11:57:33,738 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9043
+ 2024-04-16 11:57:33,738 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9044
+ 2024-04-16 11:57:33,738 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9045
+ 2024-04-16 11:57:33,738 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9046
+ 2024-04-16 11:57:33,739 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9047
+ 2024-04-16 11:57:33,739 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9048
+ 2024-04-16 11:57:33,743 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9049
+ 2024-04-16 11:57:33,751 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9050
+ 2024-04-16 11:57:33,752 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9051
+ 2024-04-16 11:57:33,752 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9052
+ 2024-04-16 11:57:33,752 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9053
+ 2024-04-16 11:57:36,753 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9054
+ 2024-04-16 11:57:41,754 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9055
+ 2024-04-16 11:57:43,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9056
+ 2024-04-16 11:57:43,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9057
+ 2024-04-16 11:57:43,075 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9058
+ 2024-04-16 11:57:47,212 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9059
+ 2024-04-16 11:57:50,313 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9060
+ 2024-04-16 11:57:50,313 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9061
+ 2024-04-16 11:57:50,313 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9062
+ 2024-04-16 11:57:50,313 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9063
+ 2024-04-16 11:57:50,314 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9064
+ 2024-04-16 11:57:50,315 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9065
+ 2024-04-16 11:57:50,315 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9066
+ 2024-04-16 11:57:50,316 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9067
+ 2024-04-16 11:57:50,317 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9068
+ 2024-04-16 11:57:50,725 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9069
+ 2024-04-16 11:57:50,725 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9070
+ 2024-04-16 11:57:50,725 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9071
+ 2024-04-16 11:57:50,725 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9072
+ 2024-04-16 11:57:50,726 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9073
+ 2024-04-16 11:57:50,726 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9074
+ 2024-04-16 11:57:50,726 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9075
+ 2024-04-16 11:57:50,727 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9076
+ 2024-04-16 11:57:50,738 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9077
+ 2024-04-16 11:57:50,739 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9078
+ 2024-04-16 11:57:50,739 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9079
+ 2024-04-16 11:57:50,739 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9080
+ 2024-04-16 11:57:52,740 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9081
+ 2024-04-16 11:57:57,018 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9082
+ 2024-04-16 11:57:57,018 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9083
+ 2024-04-16 11:57:57,018 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9084
+ 2024-04-16 11:57:57,019 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9085
+ 2024-04-16 11:57:57,022 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9086
+ 2024-04-16 11:57:57,022 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9087
+ 2024-04-16 11:57:57,023 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9088
+ 2024-04-16 11:57:57,023 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9089
+ 2024-04-16 11:57:57,024 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9090
+ 2024-04-16 11:57:57,419 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9091
+ 2024-04-16 11:57:57,419 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9092
+ 2024-04-16 11:57:57,419 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9093
+ 2024-04-16 11:57:57,420 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9094
+ 2024-04-16 11:57:57,420 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9095
+ 2024-04-16 11:57:57,420 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9096
+ 2024-04-16 11:57:57,420 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9097
+ 2024-04-16 11:57:57,421 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9098
+ 2024-04-16 11:57:57,432 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9099
+ 2024-04-16 11:57:57,433 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9100
+ 2024-04-16 11:57:57,433 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9101
+ 2024-04-16 11:57:57,437 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9102
+ 2024-04-16 11:57:58,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9103
+ 2024-04-16 11:57:58,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9104
+ 2024-04-16 11:57:58,075 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9105
+ 2024-04-16 11:57:58,075 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9106
+ 2024-04-16 11:58:03,175 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9107
+ 2024-04-16 11:58:08,176 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9108
+ 2024-04-16 11:58:12,387 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9109
+ 2024-04-16 11:58:12,388 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9110
+ 2024-04-16 11:58:12,388 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9111
+ 2024-04-16 11:58:12,388 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9112
+ 2024-04-16 11:58:12,389 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9113
+ 2024-04-16 11:58:12,392 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9114
+ 2024-04-16 11:58:12,395 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9115
+ 2024-04-16 11:58:12,396 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9116
+ 2024-04-16 11:58:12,397 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9117
+ 2024-04-16 11:58:12,861 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9118
+ 2024-04-16 11:58:12,861 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9119
+ 2024-04-16 11:58:12,861 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9120
+ 2024-04-16 11:58:12,861 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9121
+ 2024-04-16 11:58:12,861 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9122
+ 2024-04-16 11:58:12,861 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9123
+ 2024-04-16 11:58:12,862 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9124
+ 2024-04-16 11:58:12,862 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9125
+ 2024-04-16 11:58:12,874 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9126
+ 2024-04-16 11:58:12,874 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9127
+ 2024-04-16 11:58:12,874 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9128
+ 2024-04-16 11:58:12,875 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9129
+ 2024-04-16 11:58:13,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9130
+ 2024-04-16 11:58:13,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9131
+ 2024-04-16 11:58:13,074 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9132
+ 2024-04-16 11:58:14,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9133
+ 2024-04-16 11:58:17,866 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9134
+ 2024-04-16 11:58:17,866 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9135
+ 2024-04-16 11:58:17,866 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9136
+ 2024-04-16 11:58:17,866 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9137
+ 2024-04-16 11:58:17,866 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9138
+ 2024-04-16 11:58:17,867 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9139
+ 2024-04-16 11:58:17,867 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9140
+ 2024-04-16 11:58:17,868 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9141
+ 2024-04-16 11:58:17,869 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9142
+ 2024-04-16 11:58:18,263 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9143
+ 2024-04-16 11:58:18,264 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9144
+ 2024-04-16 11:58:18,264 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9145
+ 2024-04-16 11:58:18,264 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9146
+ 2024-04-16 11:58:18,264 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9147
+ 2024-04-16 11:58:18,264 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9148
+ 2024-04-16 11:58:18,264 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9149
+ 2024-04-16 11:58:18,266 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9150
+ 2024-04-16 11:58:18,277 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9151
+ 2024-04-16 11:58:18,277 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9152
+ 2024-04-16 11:58:18,277 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9153
+ 2024-04-16 11:58:18,277 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9154
+ 2024-04-16 11:58:19,278 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9155
+ 2024-04-16 11:58:19,343 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9156
+ 2024-04-16 11:58:19,343 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9157
+ 2024-04-16 11:58:19,343 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9158
+ 2024-04-16 11:58:19,343 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9159
+ 2024-04-16 11:58:19,344 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9160
+ 2024-04-16 11:58:19,344 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9161
+ 2024-04-16 11:58:19,345 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9162
+ 2024-04-16 11:58:19,346 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9163
+ 2024-04-16 11:58:19,347 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9164
+ 2024-04-16 11:58:21,805 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9165
+ 2024-04-16 11:58:21,805 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9166
+ 2024-04-16 11:58:21,805 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9167
+ 2024-04-16 11:58:21,805 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9168
+ 2024-04-16 11:58:21,805 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9169
+ 2024-04-16 11:58:21,805 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9170
+ 2024-04-16 11:58:21,806 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9171
+ 2024-04-16 11:58:21,807 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9172
+ 2024-04-16 11:58:21,817 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9173
+ 2024-04-16 11:58:21,817 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9174
+ 2024-04-16 11:58:21,817 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9175
+ 2024-04-16 11:58:21,818 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9176
+ 2024-04-16 11:58:24,819 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9177
+ 2024-04-16 11:58:28,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9178
+ 2024-04-16 11:58:28,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9179
+ 2024-04-16 11:58:28,075 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9180
+ 2024-04-16 11:58:30,180 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9181
+ 2024-04-16 11:58:35,181 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9182
+ 2024-04-16 11:58:40,182 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9183
+ 2024-04-16 11:58:43,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9184
+ 2024-04-16 11:58:43,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9185
+ 2024-04-16 11:58:43,075 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9186
+ 2024-04-16 11:58:46,142 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9187
+ 2024-04-16 11:58:51,143 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9188
+ 2024-04-16 11:58:56,144 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9189
+ 2024-04-16 11:58:58,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9190
+ 2024-04-16 11:58:58,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9191
+ 2024-04-16 11:58:58,075 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9192
+ 2024-04-16 11:59:01,171 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9193
+ 2024-04-16 11:59:06,172 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9194
+ 2024-04-16 11:59:07,576 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9195
+ 2024-04-16 11:59:07,577 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9196
+ 2024-04-16 11:59:07,577 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9197
+ 2024-04-16 11:59:07,577 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9198
+ 2024-04-16 11:59:07,578 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9199
+ 2024-04-16 11:59:07,578 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9200
+ 2024-04-16 11:59:07,580 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9201
+ 2024-04-16 11:59:07,581 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9202
+ 2024-04-16 11:59:07,581 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9203
+ 2024-04-16 11:59:11,173 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9204
+ 2024-04-16 11:59:13,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9205
+ 2024-04-16 11:59:13,076 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9206
+ 2024-04-16 11:59:13,076 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9207
+ 2024-04-16 11:59:15,135 INFO Thread-12 :79 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240416_061855-mrfkrks2/files/output.log
9208
+ 2024-04-16 11:59:16,237 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9209
+ 2024-04-16 11:59:21,239 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9210
+ 2024-04-16 11:59:26,239 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9211
+ 2024-04-16 11:59:28,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9212
+ 2024-04-16 11:59:28,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9213
+ 2024-04-16 11:59:28,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9214
+ 2024-04-16 11:59:31,247 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9215
+ 2024-04-16 11:59:33,715 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9216
+ 2024-04-16 11:59:33,715 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9217
+ 2024-04-16 11:59:33,715 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9218
+ 2024-04-16 11:59:33,715 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9219
+ 2024-04-16 11:59:33,716 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9220
+ 2024-04-16 11:59:33,716 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9221
+ 2024-04-16 11:59:33,717 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9222
+ 2024-04-16 11:59:33,717 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9223
+ 2024-04-16 11:59:33,728 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9224
+ 2024-04-16 11:59:33,728 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9225
+ 2024-04-16 11:59:33,728 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9226
+ 2024-04-16 11:59:33,729 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9227
+ 2024-04-16 11:59:36,730 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9228
+ 2024-04-16 11:59:41,731 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9229
+ 2024-04-16 11:59:43,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9230
+ 2024-04-16 11:59:43,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9231
+ 2024-04-16 11:59:43,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9232
+ 2024-04-16 11:59:47,290 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9233
+ 2024-04-16 11:59:52,291 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9234
+ 2024-04-16 11:59:57,292 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9235
+ 2024-04-16 11:59:58,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9236
+ 2024-04-16 11:59:58,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9237
+ 2024-04-16 11:59:58,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9238
+ 2024-04-16 12:00:03,224 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9239
+ 2024-04-16 12:00:08,225 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9240
+ 2024-04-16 12:00:13,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9241
+ 2024-04-16 12:00:13,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9242
+ 2024-04-16 12:00:13,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9243
+ 2024-04-16 12:00:13,288 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9244
+ 2024-04-16 12:00:18,289 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9245
+ 2024-04-16 12:00:23,290 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9246
+ 2024-04-16 12:00:28,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9247
+ 2024-04-16 12:00:28,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9248
+ 2024-04-16 12:00:28,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9249
+ 2024-04-16 12:00:29,252 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9250
+ 2024-04-16 12:00:34,253 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9251
+ 2024-04-16 12:00:39,254 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9252
+ 2024-04-16 12:00:43,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9253
+ 2024-04-16 12:00:43,115 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9254
+ 2024-04-16 12:00:43,156 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9255
+ 2024-04-16 12:00:45,237 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9256
+ 2024-04-16 12:00:50,238 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9257
+ 2024-04-16 12:00:55,239 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9258
+ 2024-04-16 12:00:58,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9259
+ 2024-04-16 12:00:58,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9260
+ 2024-04-16 12:00:58,156 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9261
+ 2024-04-16 12:01:01,235 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9262
+ 2024-04-16 12:01:06,236 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9263
+ 2024-04-16 12:01:11,242 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9264
+ 2024-04-16 12:01:13,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9265
+ 2024-04-16 12:01:13,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9266
+ 2024-04-16 12:01:13,117 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9267
+ 2024-04-16 12:01:17,177 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9268
+ 2024-04-16 12:01:22,177 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9269
+ 2024-04-16 12:01:27,178 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9270
+ 2024-04-16 12:01:28,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9271
+ 2024-04-16 12:01:28,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9272
+ 2024-04-16 12:01:28,117 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9273
+ 2024-04-16 12:01:32,287 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9274
+ 2024-04-16 12:01:37,288 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9275
+ 2024-04-16 12:01:42,289 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9276
+ 2024-04-16 12:01:43,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9277
+ 2024-04-16 12:01:43,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9278
+ 2024-04-16 12:01:43,156 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9279
+ 2024-04-16 12:01:48,259 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9280
+ 2024-04-16 12:01:48,994 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9281
+ 2024-04-16 12:01:48,994 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9282
+ 2024-04-16 12:01:48,994 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9283
+ 2024-04-16 12:01:48,994 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9284
+ 2024-04-16 12:01:48,995 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9285
+ 2024-04-16 12:01:48,996 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9286
+ 2024-04-16 12:01:48,996 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9287
+ 2024-04-16 12:01:48,997 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9288
+ 2024-04-16 12:01:48,997 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9289
+ 2024-04-16 12:01:49,048 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9290
+ 2024-04-16 12:01:49,049 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9291
+ 2024-04-16 12:01:49,049 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9292
+ 2024-04-16 12:01:49,049 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9293
+ 2024-04-16 12:01:49,049 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9294
+ 2024-04-16 12:01:49,049 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9295
+ 2024-04-16 12:01:49,050 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9296
+ 2024-04-16 12:01:49,050 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9297
+ 2024-04-16 12:01:49,062 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9298
+ 2024-04-16 12:01:49,062 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9299
+ 2024-04-16 12:01:49,062 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9300
+ 2024-04-16 12:01:49,063 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9301
+ 2024-04-16 12:01:54,064 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9302
+ 2024-04-16 12:01:58,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9303
+ 2024-04-16 12:01:58,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9304
+ 2024-04-16 12:01:58,118 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9305
+ 2024-04-16 12:01:59,261 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9306
+ 2024-04-16 12:02:00,262 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9307
+ 2024-04-16 12:02:00,262 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9308
+ 2024-04-16 12:02:00,262 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9309
+ 2024-04-16 12:02:00,262 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9310
+ 2024-04-16 12:02:00,263 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9311
+ 2024-04-16 12:02:00,264 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9312
+ 2024-04-16 12:02:00,265 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9313
+ 2024-04-16 12:02:00,266 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9314
+ 2024-04-16 12:02:00,267 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9315
+ 2024-04-16 12:02:04,262 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9316
+ 2024-04-16 12:02:09,263 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9317
+ 2024-04-16 12:02:13,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9318
+ 2024-04-16 12:02:13,117 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9319
+ 2024-04-16 12:02:13,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9320
+ 2024-04-16 12:02:14,281 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9321
+ 2024-04-16 12:02:19,282 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9322
+ 2024-04-16 12:02:24,283 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9323
+ 2024-04-16 12:02:28,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9324
+ 2024-04-16 12:02:28,118 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9325
+ 2024-04-16 12:02:28,158 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9326
+ 2024-04-16 12:02:30,207 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9327
+ 2024-04-16 12:02:35,208 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9328
+ 2024-04-16 12:02:40,209 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9329
+ 2024-04-16 12:02:43,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9330
+ 2024-04-16 12:02:43,117 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9331
+ 2024-04-16 12:02:43,157 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9332
+ 2024-04-16 12:02:45,256 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9333
+ 2024-04-16 12:02:50,257 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9334
+ 2024-04-16 12:02:55,258 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9335
+ 2024-04-16 12:02:58,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9336
+ 2024-04-16 12:02:58,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9337
+ 2024-04-16 12:02:58,118 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9338
+ 2024-04-16 12:03:00,268 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9339
+ 2024-04-16 12:03:00,269 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9340
+ 2024-04-16 12:03:00,273 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9341
+ 2024-04-16 12:03:05,272 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9342
+ 2024-04-16 12:03:10,273 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9343
+ 2024-04-16 12:03:13,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9344
+ 2024-04-16 12:03:13,117 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9345
+ 2024-04-16 12:03:13,157 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9346
+ 2024-04-16 12:03:16,214 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9347
+ 2024-04-16 12:03:21,218 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9348
+ 2024-04-16 12:03:26,219 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9349
+ 2024-04-16 12:03:28,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9350
+ 2024-04-16 12:03:28,118 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9351
+ 2024-04-16 12:03:28,157 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9352
+ 2024-04-16 12:03:30,270 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9353
+ 2024-04-16 12:03:31,271 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9354
+ 2024-04-16 12:03:32,893 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9355
+ 2024-04-16 12:03:32,893 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9356
+ 2024-04-16 12:03:32,893 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9357
+ 2024-04-16 12:03:32,893 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9358
+ 2024-04-16 12:03:32,894 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9359
+ 2024-04-16 12:03:32,894 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9360
+ 2024-04-16 12:03:32,894 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9361
+ 2024-04-16 12:03:32,907 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9362
+ 2024-04-16 12:03:32,907 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9363
+ 2024-04-16 12:03:32,907 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9364
+ 2024-04-16 12:03:32,908 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9365
+ 2024-04-16 12:03:35,341 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9366
+ 2024-04-16 12:03:35,341 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9367
+ 2024-04-16 12:03:35,342 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9368
+ 2024-04-16 12:03:35,342 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9369
+ 2024-04-16 12:03:35,342 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9370
+ 2024-04-16 12:03:35,343 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9371
+ 2024-04-16 12:03:35,344 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9372
+ 2024-04-16 12:03:35,344 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9373
+ 2024-04-16 12:03:35,346 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9374
+ 2024-04-16 12:03:35,802 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9375
+ 2024-04-16 12:03:35,802 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9376
+ 2024-04-16 12:03:35,802 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9377
+ 2024-04-16 12:03:35,802 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9378
+ 2024-04-16 12:03:35,802 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9379
+ 2024-04-16 12:03:35,803 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9380
+ 2024-04-16 12:03:35,803 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9381
+ 2024-04-16 12:03:35,804 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9382
+ 2024-04-16 12:03:35,815 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9383
+ 2024-04-16 12:03:35,815 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9384
+ 2024-04-16 12:03:35,815 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9385
+ 2024-04-16 12:03:35,815 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9386
+ 2024-04-16 12:03:36,816 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9387
+ 2024-04-16 12:03:39,240 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9388
+ 2024-04-16 12:03:39,240 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9389
+ 2024-04-16 12:03:39,240 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9390
+ 2024-04-16 12:03:39,241 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9391
+ 2024-04-16 12:03:39,241 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9392
+ 2024-04-16 12:03:39,242 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9393
+ 2024-04-16 12:03:39,242 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9394
+ 2024-04-16 12:03:39,243 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9395
+ 2024-04-16 12:03:39,244 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9396
+ 2024-04-16 12:03:41,817 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9397
+ 2024-04-16 12:03:43,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9398
+ 2024-04-16 12:03:43,168 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9399
+ 2024-04-16 12:03:43,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9400
+ 2024-04-16 12:03:47,261 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9401
+ 2024-04-16 12:03:52,262 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9402
+ 2024-04-16 12:03:57,263 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9403
+ 2024-04-16 12:03:58,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9404
+ 2024-04-16 12:03:58,168 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9405
+ 2024-04-16 12:03:58,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9406
+ 2024-04-16 12:04:02,302 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9407
+ 2024-04-16 12:04:02,423 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9408
+ 2024-04-16 12:04:02,423 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9409
+ 2024-04-16 12:04:02,423 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9410
+ 2024-04-16 12:04:02,424 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9411
+ 2024-04-16 12:04:02,424 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9412
+ 2024-04-16 12:04:02,424 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9413
+ 2024-04-16 12:04:02,425 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9414
+ 2024-04-16 12:04:02,425 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9415
+ 2024-04-16 12:04:02,437 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9416
+ 2024-04-16 12:04:02,437 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9417
+ 2024-04-16 12:04:02,438 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9418
+ 2024-04-16 12:04:02,438 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9419
+ 2024-04-16 12:04:07,439 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9420
+ 2024-04-16 12:04:12,440 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9421
+ 2024-04-16 12:04:13,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9422
+ 2024-04-16 12:04:13,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9423
+ 2024-04-16 12:04:13,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9424
+ 2024-04-16 12:04:18,272 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9425
+ 2024-04-16 12:04:23,274 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9426
+ 2024-04-16 12:04:28,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9427
+ 2024-04-16 12:04:28,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9428
+ 2024-04-16 12:04:28,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9429
+ 2024-04-16 12:04:29,229 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9430
+ 2024-04-16 12:04:34,230 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9431
+ 2024-04-16 12:04:39,230 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9432
+ 2024-04-16 12:04:43,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9433
+ 2024-04-16 12:04:43,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9434
+ 2024-04-16 12:04:43,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9435
+ 2024-04-16 12:04:44,315 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9436
+ 2024-04-16 12:04:49,316 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9437
+ 2024-04-16 12:04:54,318 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9438
+ 2024-04-16 12:04:54,814 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9439
+ 2024-04-16 12:04:54,814 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9440
+ 2024-04-16 12:04:54,814 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9441
+ 2024-04-16 12:04:54,815 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9442
+ 2024-04-16 12:04:54,815 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9443
+ 2024-04-16 12:04:54,816 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9444
+ 2024-04-16 12:04:54,817 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9445
+ 2024-04-16 12:04:54,817 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9446
+ 2024-04-16 12:04:54,818 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9447
+ 2024-04-16 12:04:58,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9448
+ 2024-04-16 12:04:58,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9449
+ 2024-04-16 12:04:58,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9450
+ 2024-04-16 12:05:00,242 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9451
+ 2024-04-16 12:05:05,243 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9452
+ 2024-04-16 12:05:10,244 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9453
+ 2024-04-16 12:05:13,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9454
+ 2024-04-16 12:05:13,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9455
+ 2024-04-16 12:05:13,208 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9456
+ 2024-04-16 12:05:15,333 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9457
+ 2024-04-16 12:05:20,334 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9458
+ 2024-04-16 12:05:25,335 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9459
+ 2024-04-16 12:05:28,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9460
+ 2024-04-16 12:05:28,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9461
+ 2024-04-16 12:05:28,209 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9462
+ 2024-04-16 12:05:31,317 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9463
+ 2024-04-16 12:05:36,318 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9464
+ 2024-04-16 12:05:41,324 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9465
+ 2024-04-16 12:05:43,168 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9466
+ 2024-04-16 12:05:43,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9467
+ 2024-04-16 12:05:43,208 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9468
+ 2024-04-16 12:05:46,337 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9469
+ 2024-04-16 12:05:51,338 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
wandb/debug.log CHANGED
@@ -39,3 +39,63 @@ config: {}
39
  2024-04-16 11:52:52,066 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
40
  2024-04-16 11:52:52,066 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
41
  2024-04-16 11:55:22,701 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
  2024-04-16 11:52:52,066 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
40
  2024-04-16 11:52:52,066 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
41
  2024-04-16 11:55:22,701 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
42
+ 2024-04-16 11:56:09,161 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
43
+ 2024-04-16 11:56:09,161 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
44
+ 2024-04-16 11:56:09,168 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
45
+ 2024-04-16 11:56:24,889 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
46
+ 2024-04-16 11:56:24,889 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
47
+ 2024-04-16 11:56:24,898 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
48
+ 2024-04-16 11:56:25,088 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
49
+ 2024-04-16 11:56:25,088 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
50
+ 2024-04-16 11:56:25,097 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
51
+ 2024-04-16 11:56:25,469 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
52
+ 2024-04-16 11:56:25,469 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
53
+ 2024-04-16 11:56:37,216 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
54
+ 2024-04-16 11:56:37,329 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
55
+ 2024-04-16 11:56:37,329 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
56
+ 2024-04-16 11:56:55,142 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
57
+ 2024-04-16 11:56:58,017 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
58
+ 2024-04-16 11:56:58,017 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
59
+ 2024-04-16 11:57:11,339 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
60
+ 2024-04-16 11:57:11,917 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
61
+ 2024-04-16 11:57:11,918 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
62
+ 2024-04-16 11:57:14,596 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
63
+ 2024-04-16 11:57:15,117 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
64
+ 2024-04-16 11:57:15,117 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
65
+ 2024-04-16 11:57:23,878 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
66
+ 2024-04-16 11:57:24,280 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
67
+ 2024-04-16 11:57:24,280 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
68
+ 2024-04-16 11:57:33,335 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
69
+ 2024-04-16 11:57:33,737 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
70
+ 2024-04-16 11:57:33,737 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
71
+ 2024-04-16 11:57:50,312 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
72
+ 2024-04-16 11:57:50,724 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
73
+ 2024-04-16 11:57:50,724 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
74
+ 2024-04-16 11:57:57,017 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
75
+ 2024-04-16 11:57:57,418 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
76
+ 2024-04-16 11:57:57,418 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
77
+ 2024-04-16 11:58:12,387 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
78
+ 2024-04-16 11:58:12,860 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
79
+ 2024-04-16 11:58:12,860 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
80
+ 2024-04-16 11:58:17,865 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
81
+ 2024-04-16 11:58:18,263 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
82
+ 2024-04-16 11:58:18,263 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
83
+ 2024-04-16 11:58:19,342 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
84
+ 2024-04-16 11:58:21,804 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
85
+ 2024-04-16 11:58:21,804 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
86
+ 2024-04-16 11:59:07,575 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
87
+ 2024-04-16 11:59:33,714 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
88
+ 2024-04-16 11:59:33,714 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
89
+ 2024-04-16 12:01:48,993 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
90
+ 2024-04-16 12:01:49,048 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
91
+ 2024-04-16 12:01:49,048 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
92
+ 2024-04-16 12:02:00,261 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
93
+ 2024-04-16 12:03:32,892 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
94
+ 2024-04-16 12:03:32,892 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
95
+ 2024-04-16 12:03:35,341 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
96
+ 2024-04-16 12:03:35,801 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
97
+ 2024-04-16 12:03:35,801 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
98
+ 2024-04-16 12:03:39,240 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
99
+ 2024-04-16 12:04:02,423 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
100
+ 2024-04-16 12:04:02,423 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
101
+ 2024-04-16 12:04:54,813 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
wandb/run-20240416_061855-mrfkrks2/files/output.log CHANGED
@@ -1 +1,2 @@
1
 
 
 
1
 
2
+ Training completed in 20006.813757896423 seconds.
wandb/run-20240416_061855-mrfkrks2/logs/debug-internal.log CHANGED
@@ -8804,3 +8804,666 @@ subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after
8804
  2024-04-16 11:55:29,199 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8805
  2024-04-16 11:55:34,200 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8806
  2024-04-16 11:55:39,201 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8804
  2024-04-16 11:55:29,199 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8805
  2024-04-16 11:55:34,200 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8806
  2024-04-16 11:55:39,201 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8807
+ 2024-04-16 11:55:43,071 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8808
+ 2024-04-16 11:55:43,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8809
+ 2024-04-16 11:55:43,073 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8810
+ 2024-04-16 11:55:44,225 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8811
+ 2024-04-16 11:55:49,226 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8812
+ 2024-04-16 11:55:54,226 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8813
+ 2024-04-16 11:55:58,071 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8814
+ 2024-04-16 11:55:58,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8815
+ 2024-04-16 11:55:58,073 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8816
+ 2024-04-16 11:56:00,207 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8817
+ 2024-04-16 11:56:05,208 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8818
+ 2024-04-16 11:56:09,162 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8819
+ 2024-04-16 11:56:09,162 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8820
+ 2024-04-16 11:56:09,162 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8821
+ 2024-04-16 11:56:09,162 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8822
+ 2024-04-16 11:56:09,162 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8823
+ 2024-04-16 11:56:09,162 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8824
+ 2024-04-16 11:56:09,163 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8825
+ 2024-04-16 11:56:09,163 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8826
+ 2024-04-16 11:56:09,178 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8827
+ 2024-04-16 11:56:09,179 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8828
+ 2024-04-16 11:56:09,179 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8829
+ 2024-04-16 11:56:09,179 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8830
+ 2024-04-16 11:56:09,179 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8831
+ 2024-04-16 11:56:09,179 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8832
+ 2024-04-16 11:56:09,180 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8833
+ 2024-04-16 11:56:09,180 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8834
+ 2024-04-16 11:56:09,181 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8835
+ 2024-04-16 11:56:09,183 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8836
+ 2024-04-16 11:56:09,183 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8837
+ 2024-04-16 11:56:09,184 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8838
+ 2024-04-16 11:56:09,185 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8839
+ 2024-04-16 11:56:11,182 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8840
+ 2024-04-16 11:56:13,071 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8841
+ 2024-04-16 11:56:13,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8842
+ 2024-04-16 11:56:13,074 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8843
+ 2024-04-16 11:56:16,203 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8844
+ 2024-04-16 11:56:21,203 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8845
+ 2024-04-16 11:56:24,890 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8846
+ 2024-04-16 11:56:24,890 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8847
+ 2024-04-16 11:56:24,890 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8848
+ 2024-04-16 11:56:24,891 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8849
+ 2024-04-16 11:56:24,891 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8850
+ 2024-04-16 11:56:24,891 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8851
+ 2024-04-16 11:56:24,891 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8852
+ 2024-04-16 11:56:24,891 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8853
+ 2024-04-16 11:56:24,906 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8854
+ 2024-04-16 11:56:24,906 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8855
+ 2024-04-16 11:56:24,906 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8856
+ 2024-04-16 11:56:24,906 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8857
+ 2024-04-16 11:56:24,907 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8858
+ 2024-04-16 11:56:24,907 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8859
+ 2024-04-16 11:56:24,907 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8860
+ 2024-04-16 11:56:24,907 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8861
+ 2024-04-16 11:56:24,908 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8862
+ 2024-04-16 11:56:24,909 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8863
+ 2024-04-16 11:56:24,909 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8864
+ 2024-04-16 11:56:24,910 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8865
+ 2024-04-16 11:56:24,911 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8866
+ 2024-04-16 11:56:25,089 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8867
+ 2024-04-16 11:56:25,089 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8868
+ 2024-04-16 11:56:25,089 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8869
+ 2024-04-16 11:56:25,090 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8870
+ 2024-04-16 11:56:25,090 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8871
+ 2024-04-16 11:56:25,090 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8872
+ 2024-04-16 11:56:25,090 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8873
+ 2024-04-16 11:56:25,090 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8874
+ 2024-04-16 11:56:25,103 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8875
+ 2024-04-16 11:56:25,103 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8876
+ 2024-04-16 11:56:25,103 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8877
+ 2024-04-16 11:56:25,103 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8878
+ 2024-04-16 11:56:25,103 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8879
+ 2024-04-16 11:56:25,104 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8880
+ 2024-04-16 11:56:25,104 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8881
+ 2024-04-16 11:56:25,104 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8882
+ 2024-04-16 11:56:25,105 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8883
+ 2024-04-16 11:56:25,105 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8884
+ 2024-04-16 11:56:25,106 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8885
+ 2024-04-16 11:56:25,106 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8886
+ 2024-04-16 11:56:25,107 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8887
+ 2024-04-16 11:56:25,469 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8888
+ 2024-04-16 11:56:25,470 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8889
+ 2024-04-16 11:56:25,470 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8890
+ 2024-04-16 11:56:25,470 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8891
+ 2024-04-16 11:56:25,471 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8892
+ 2024-04-16 11:56:25,471 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8893
+ 2024-04-16 11:56:25,471 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8894
+ 2024-04-16 11:56:25,472 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8895
+ 2024-04-16 11:56:25,483 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8896
+ 2024-04-16 11:56:25,483 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8897
+ 2024-04-16 11:56:25,483 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8898
+ 2024-04-16 11:56:25,484 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8899
+ 2024-04-16 11:56:26,484 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8900
+ 2024-04-16 11:56:28,071 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8901
+ 2024-04-16 11:56:28,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8902
+ 2024-04-16 11:56:28,073 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8903
+ 2024-04-16 11:56:32,175 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8904
+ 2024-04-16 11:56:37,176 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8905
+ 2024-04-16 11:56:37,217 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8906
+ 2024-04-16 11:56:37,217 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8907
+ 2024-04-16 11:56:37,217 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8908
+ 2024-04-16 11:56:37,217 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8909
+ 2024-04-16 11:56:37,218 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8910
+ 2024-04-16 11:56:37,218 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8911
+ 2024-04-16 11:56:37,219 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8912
+ 2024-04-16 11:56:37,220 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8913
+ 2024-04-16 11:56:37,220 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8914
+ 2024-04-16 11:56:37,330 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8915
+ 2024-04-16 11:56:37,330 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8916
+ 2024-04-16 11:56:37,330 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8917
+ 2024-04-16 11:56:37,330 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8918
+ 2024-04-16 11:56:37,330 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8919
+ 2024-04-16 11:56:37,331 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8920
+ 2024-04-16 11:56:37,331 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8921
+ 2024-04-16 11:56:37,331 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8922
+ 2024-04-16 11:56:37,342 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8923
+ 2024-04-16 11:56:37,342 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8924
+ 2024-04-16 11:56:37,342 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8925
+ 2024-04-16 11:56:37,343 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8926
+ 2024-04-16 11:56:42,344 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8927
+ 2024-04-16 11:56:43,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8928
+ 2024-04-16 11:56:43,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8929
+ 2024-04-16 11:56:43,074 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8930
+ 2024-04-16 11:56:48,162 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8931
+ 2024-04-16 11:56:53,163 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8932
+ 2024-04-16 11:56:55,143 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8933
+ 2024-04-16 11:56:55,143 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8934
+ 2024-04-16 11:56:55,143 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8935
+ 2024-04-16 11:56:55,143 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8936
+ 2024-04-16 11:56:55,144 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8937
+ 2024-04-16 11:56:55,145 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8938
+ 2024-04-16 11:56:55,146 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8939
+ 2024-04-16 11:56:55,147 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8940
+ 2024-04-16 11:56:55,147 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8941
+ 2024-04-16 11:56:58,018 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8942
+ 2024-04-16 11:56:58,018 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8943
+ 2024-04-16 11:56:58,018 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8944
+ 2024-04-16 11:56:58,018 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8945
+ 2024-04-16 11:56:58,018 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8946
+ 2024-04-16 11:56:58,018 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8947
+ 2024-04-16 11:56:58,019 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8948
+ 2024-04-16 11:56:58,019 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8949
+ 2024-04-16 11:56:58,031 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8950
+ 2024-04-16 11:56:58,031 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8951
+ 2024-04-16 11:56:58,032 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8952
+ 2024-04-16 11:56:58,032 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8953
+ 2024-04-16 11:56:58,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8954
+ 2024-04-16 11:56:58,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8955
+ 2024-04-16 11:56:58,074 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8956
+ 2024-04-16 11:56:58,617 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8957
+ 2024-04-16 11:57:03,618 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8958
+ 2024-04-16 11:57:08,619 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8959
+ 2024-04-16 11:57:11,340 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8960
+ 2024-04-16 11:57:11,340 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8961
+ 2024-04-16 11:57:11,340 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8962
+ 2024-04-16 11:57:11,341 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8963
+ 2024-04-16 11:57:11,341 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8964
+ 2024-04-16 11:57:11,342 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8965
+ 2024-04-16 11:57:11,343 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8966
+ 2024-04-16 11:57:11,343 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8967
+ 2024-04-16 11:57:11,344 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8968
+ 2024-04-16 11:57:11,918 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8969
+ 2024-04-16 11:57:11,918 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8970
+ 2024-04-16 11:57:11,918 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8971
+ 2024-04-16 11:57:11,919 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8972
+ 2024-04-16 11:57:11,919 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
8973
+ 2024-04-16 11:57:11,920 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8974
+ 2024-04-16 11:57:11,920 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8975
+ 2024-04-16 11:57:11,920 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
8976
+ 2024-04-16 11:57:11,931 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
8977
+ 2024-04-16 11:57:11,931 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
8978
+ 2024-04-16 11:57:11,931 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
8979
+ 2024-04-16 11:57:11,932 DEBUG SenderThread:79 [sender.py:send():379] send: stats
8980
+ 2024-04-16 11:57:13,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
8981
+ 2024-04-16 11:57:13,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
8982
+ 2024-04-16 11:57:13,074 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
8983
+ 2024-04-16 11:57:14,177 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
8984
+ 2024-04-16 11:57:14,597 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
8985
+ 2024-04-16 11:57:14,597 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
8986
+ 2024-04-16 11:57:14,597 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
8987
+ 2024-04-16 11:57:14,597 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
8988
+ 2024-04-16 11:57:14,598 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
8989
+ 2024-04-16 11:57:14,599 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
8990
+ 2024-04-16 11:57:14,600 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
8991
+ 2024-04-16 11:57:14,601 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
8992
+ 2024-04-16 11:57:14,602 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
8993
+ 2024-04-16 11:57:15,118 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
8994
+ 2024-04-16 11:57:15,118 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
8995
+ 2024-04-16 11:57:15,118 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
8996
+ 2024-04-16 11:57:15,118 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
8997
+ 2024-04-16 11:57:15,119 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
8998
+ 2024-04-16 11:57:15,119 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
8999
+ 2024-04-16 11:57:15,119 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9000
+ 2024-04-16 11:57:15,120 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9001
+ 2024-04-16 11:57:15,131 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9002
+ 2024-04-16 11:57:15,132 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9003
+ 2024-04-16 11:57:15,132 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9004
+ 2024-04-16 11:57:15,132 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9005
+ 2024-04-16 11:57:20,133 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9006
+ 2024-04-16 11:57:23,879 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9007
+ 2024-04-16 11:57:23,879 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9008
+ 2024-04-16 11:57:23,879 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9009
+ 2024-04-16 11:57:23,880 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9010
+ 2024-04-16 11:57:23,880 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9011
+ 2024-04-16 11:57:23,881 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9012
+ 2024-04-16 11:57:23,882 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9013
+ 2024-04-16 11:57:23,882 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9014
+ 2024-04-16 11:57:23,883 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9015
+ 2024-04-16 11:57:24,280 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9016
+ 2024-04-16 11:57:24,281 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9017
+ 2024-04-16 11:57:24,281 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9018
+ 2024-04-16 11:57:24,281 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9019
+ 2024-04-16 11:57:24,282 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9020
+ 2024-04-16 11:57:24,282 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9021
+ 2024-04-16 11:57:24,282 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9022
+ 2024-04-16 11:57:24,283 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9023
+ 2024-04-16 11:57:24,294 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9024
+ 2024-04-16 11:57:24,294 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9025
+ 2024-04-16 11:57:24,294 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9026
+ 2024-04-16 11:57:24,294 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9027
+ 2024-04-16 11:57:25,295 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9028
+ 2024-04-16 11:57:28,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9029
+ 2024-04-16 11:57:28,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9030
+ 2024-04-16 11:57:28,074 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9031
+ 2024-04-16 11:57:31,197 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9032
+ 2024-04-16 11:57:33,336 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9033
+ 2024-04-16 11:57:33,336 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9034
+ 2024-04-16 11:57:33,336 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9035
+ 2024-04-16 11:57:33,336 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9036
+ 2024-04-16 11:57:33,337 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9037
+ 2024-04-16 11:57:33,337 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9038
+ 2024-04-16 11:57:33,338 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9039
+ 2024-04-16 11:57:33,340 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9040
+ 2024-04-16 11:57:33,341 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9041
+ 2024-04-16 11:57:33,738 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9042
+ 2024-04-16 11:57:33,738 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9043
+ 2024-04-16 11:57:33,738 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9044
+ 2024-04-16 11:57:33,738 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9045
+ 2024-04-16 11:57:33,738 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9046
+ 2024-04-16 11:57:33,739 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9047
+ 2024-04-16 11:57:33,739 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9048
+ 2024-04-16 11:57:33,743 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9049
+ 2024-04-16 11:57:33,751 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9050
+ 2024-04-16 11:57:33,752 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9051
+ 2024-04-16 11:57:33,752 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9052
+ 2024-04-16 11:57:33,752 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9053
+ 2024-04-16 11:57:36,753 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9054
+ 2024-04-16 11:57:41,754 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9055
+ 2024-04-16 11:57:43,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9056
+ 2024-04-16 11:57:43,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9057
+ 2024-04-16 11:57:43,075 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9058
+ 2024-04-16 11:57:47,212 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9059
+ 2024-04-16 11:57:50,313 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9060
+ 2024-04-16 11:57:50,313 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9061
+ 2024-04-16 11:57:50,313 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9062
+ 2024-04-16 11:57:50,313 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9063
+ 2024-04-16 11:57:50,314 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9064
+ 2024-04-16 11:57:50,315 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9065
+ 2024-04-16 11:57:50,315 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9066
+ 2024-04-16 11:57:50,316 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9067
+ 2024-04-16 11:57:50,317 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9068
+ 2024-04-16 11:57:50,725 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9069
+ 2024-04-16 11:57:50,725 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9070
+ 2024-04-16 11:57:50,725 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9071
+ 2024-04-16 11:57:50,725 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9072
+ 2024-04-16 11:57:50,726 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9073
+ 2024-04-16 11:57:50,726 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9074
+ 2024-04-16 11:57:50,726 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9075
+ 2024-04-16 11:57:50,727 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9076
+ 2024-04-16 11:57:50,738 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9077
+ 2024-04-16 11:57:50,739 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9078
+ 2024-04-16 11:57:50,739 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9079
+ 2024-04-16 11:57:50,739 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9080
+ 2024-04-16 11:57:52,740 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9081
+ 2024-04-16 11:57:57,018 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9082
+ 2024-04-16 11:57:57,018 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9083
+ 2024-04-16 11:57:57,018 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9084
+ 2024-04-16 11:57:57,019 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9085
+ 2024-04-16 11:57:57,022 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9086
+ 2024-04-16 11:57:57,022 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9087
+ 2024-04-16 11:57:57,023 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9088
+ 2024-04-16 11:57:57,023 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9089
+ 2024-04-16 11:57:57,024 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9090
+ 2024-04-16 11:57:57,419 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9091
+ 2024-04-16 11:57:57,419 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9092
+ 2024-04-16 11:57:57,419 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9093
+ 2024-04-16 11:57:57,420 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9094
+ 2024-04-16 11:57:57,420 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9095
+ 2024-04-16 11:57:57,420 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9096
+ 2024-04-16 11:57:57,420 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9097
+ 2024-04-16 11:57:57,421 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9098
+ 2024-04-16 11:57:57,432 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9099
+ 2024-04-16 11:57:57,433 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9100
+ 2024-04-16 11:57:57,433 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9101
+ 2024-04-16 11:57:57,437 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9102
+ 2024-04-16 11:57:58,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9103
+ 2024-04-16 11:57:58,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9104
+ 2024-04-16 11:57:58,075 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9105
+ 2024-04-16 11:57:58,075 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9106
+ 2024-04-16 11:58:03,175 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9107
+ 2024-04-16 11:58:08,176 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9108
+ 2024-04-16 11:58:12,387 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9109
+ 2024-04-16 11:58:12,388 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9110
+ 2024-04-16 11:58:12,388 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9111
+ 2024-04-16 11:58:12,388 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9112
+ 2024-04-16 11:58:12,389 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9113
+ 2024-04-16 11:58:12,392 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9114
+ 2024-04-16 11:58:12,395 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9115
+ 2024-04-16 11:58:12,396 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9116
+ 2024-04-16 11:58:12,397 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9117
+ 2024-04-16 11:58:12,861 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9118
+ 2024-04-16 11:58:12,861 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9119
+ 2024-04-16 11:58:12,861 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9120
+ 2024-04-16 11:58:12,861 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9121
+ 2024-04-16 11:58:12,861 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9122
+ 2024-04-16 11:58:12,861 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9123
+ 2024-04-16 11:58:12,862 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9124
+ 2024-04-16 11:58:12,862 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9125
+ 2024-04-16 11:58:12,874 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9126
+ 2024-04-16 11:58:12,874 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9127
+ 2024-04-16 11:58:12,874 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9128
+ 2024-04-16 11:58:12,875 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9129
+ 2024-04-16 11:58:13,072 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9130
+ 2024-04-16 11:58:13,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9131
+ 2024-04-16 11:58:13,074 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9132
+ 2024-04-16 11:58:14,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9133
+ 2024-04-16 11:58:17,866 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9134
+ 2024-04-16 11:58:17,866 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9135
+ 2024-04-16 11:58:17,866 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9136
+ 2024-04-16 11:58:17,866 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9137
+ 2024-04-16 11:58:17,866 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9138
+ 2024-04-16 11:58:17,867 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9139
+ 2024-04-16 11:58:17,867 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9140
+ 2024-04-16 11:58:17,868 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9141
+ 2024-04-16 11:58:17,869 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9142
+ 2024-04-16 11:58:18,263 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9143
+ 2024-04-16 11:58:18,264 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9144
+ 2024-04-16 11:58:18,264 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9145
+ 2024-04-16 11:58:18,264 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9146
+ 2024-04-16 11:58:18,264 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9147
+ 2024-04-16 11:58:18,264 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9148
+ 2024-04-16 11:58:18,264 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9149
+ 2024-04-16 11:58:18,266 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9150
+ 2024-04-16 11:58:18,277 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9151
+ 2024-04-16 11:58:18,277 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9152
+ 2024-04-16 11:58:18,277 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9153
+ 2024-04-16 11:58:18,277 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9154
+ 2024-04-16 11:58:19,278 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9155
+ 2024-04-16 11:58:19,343 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9156
+ 2024-04-16 11:58:19,343 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9157
+ 2024-04-16 11:58:19,343 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9158
+ 2024-04-16 11:58:19,343 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9159
+ 2024-04-16 11:58:19,344 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9160
+ 2024-04-16 11:58:19,344 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9161
+ 2024-04-16 11:58:19,345 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9162
+ 2024-04-16 11:58:19,346 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9163
+ 2024-04-16 11:58:19,347 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9164
+ 2024-04-16 11:58:21,805 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9165
+ 2024-04-16 11:58:21,805 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9166
+ 2024-04-16 11:58:21,805 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9167
+ 2024-04-16 11:58:21,805 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9168
+ 2024-04-16 11:58:21,805 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9169
+ 2024-04-16 11:58:21,805 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9170
+ 2024-04-16 11:58:21,806 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9171
+ 2024-04-16 11:58:21,807 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9172
+ 2024-04-16 11:58:21,817 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9173
+ 2024-04-16 11:58:21,817 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9174
+ 2024-04-16 11:58:21,817 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9175
+ 2024-04-16 11:58:21,818 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9176
+ 2024-04-16 11:58:24,819 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9177
+ 2024-04-16 11:58:28,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9178
+ 2024-04-16 11:58:28,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9179
+ 2024-04-16 11:58:28,075 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9180
+ 2024-04-16 11:58:30,180 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9181
+ 2024-04-16 11:58:35,181 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9182
+ 2024-04-16 11:58:40,182 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9183
+ 2024-04-16 11:58:43,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9184
+ 2024-04-16 11:58:43,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9185
+ 2024-04-16 11:58:43,075 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9186
+ 2024-04-16 11:58:46,142 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9187
+ 2024-04-16 11:58:51,143 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9188
+ 2024-04-16 11:58:56,144 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9189
+ 2024-04-16 11:58:58,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9190
+ 2024-04-16 11:58:58,074 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9191
+ 2024-04-16 11:58:58,075 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9192
+ 2024-04-16 11:59:01,171 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9193
+ 2024-04-16 11:59:06,172 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9194
+ 2024-04-16 11:59:07,576 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9195
+ 2024-04-16 11:59:07,577 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9196
+ 2024-04-16 11:59:07,577 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9197
+ 2024-04-16 11:59:07,577 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9198
+ 2024-04-16 11:59:07,578 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9199
+ 2024-04-16 11:59:07,578 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9200
+ 2024-04-16 11:59:07,580 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9201
+ 2024-04-16 11:59:07,581 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9202
+ 2024-04-16 11:59:07,581 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9203
+ 2024-04-16 11:59:11,173 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9204
+ 2024-04-16 11:59:13,073 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9205
+ 2024-04-16 11:59:13,076 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9206
+ 2024-04-16 11:59:13,076 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9207
+ 2024-04-16 11:59:15,135 INFO Thread-12 :79 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240416_061855-mrfkrks2/files/output.log
9208
+ 2024-04-16 11:59:16,237 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9209
+ 2024-04-16 11:59:21,239 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9210
+ 2024-04-16 11:59:26,239 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9211
+ 2024-04-16 11:59:28,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9212
+ 2024-04-16 11:59:28,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9213
+ 2024-04-16 11:59:28,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9214
+ 2024-04-16 11:59:31,247 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9215
+ 2024-04-16 11:59:33,715 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9216
+ 2024-04-16 11:59:33,715 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9217
+ 2024-04-16 11:59:33,715 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9218
+ 2024-04-16 11:59:33,715 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9219
+ 2024-04-16 11:59:33,716 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9220
+ 2024-04-16 11:59:33,716 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9221
+ 2024-04-16 11:59:33,717 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9222
+ 2024-04-16 11:59:33,717 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9223
+ 2024-04-16 11:59:33,728 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9224
+ 2024-04-16 11:59:33,728 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9225
+ 2024-04-16 11:59:33,728 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9226
+ 2024-04-16 11:59:33,729 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9227
+ 2024-04-16 11:59:36,730 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9228
+ 2024-04-16 11:59:41,731 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9229
+ 2024-04-16 11:59:43,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9230
+ 2024-04-16 11:59:43,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9231
+ 2024-04-16 11:59:43,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9232
+ 2024-04-16 11:59:47,290 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9233
+ 2024-04-16 11:59:52,291 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9234
+ 2024-04-16 11:59:57,292 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9235
+ 2024-04-16 11:59:58,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9236
+ 2024-04-16 11:59:58,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9237
+ 2024-04-16 11:59:58,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9238
+ 2024-04-16 12:00:03,224 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9239
+ 2024-04-16 12:00:08,225 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9240
+ 2024-04-16 12:00:13,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9241
+ 2024-04-16 12:00:13,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9242
+ 2024-04-16 12:00:13,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9243
+ 2024-04-16 12:00:13,288 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9244
+ 2024-04-16 12:00:18,289 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9245
+ 2024-04-16 12:00:23,290 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9246
+ 2024-04-16 12:00:28,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9247
+ 2024-04-16 12:00:28,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9248
+ 2024-04-16 12:00:28,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9249
+ 2024-04-16 12:00:29,252 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9250
+ 2024-04-16 12:00:34,253 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9251
+ 2024-04-16 12:00:39,254 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9252
+ 2024-04-16 12:00:43,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9253
+ 2024-04-16 12:00:43,115 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9254
+ 2024-04-16 12:00:43,156 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9255
+ 2024-04-16 12:00:45,237 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9256
+ 2024-04-16 12:00:50,238 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9257
+ 2024-04-16 12:00:55,239 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9258
+ 2024-04-16 12:00:58,115 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9259
+ 2024-04-16 12:00:58,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9260
+ 2024-04-16 12:00:58,156 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9261
+ 2024-04-16 12:01:01,235 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9262
+ 2024-04-16 12:01:06,236 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9263
+ 2024-04-16 12:01:11,242 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9264
+ 2024-04-16 12:01:13,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9265
+ 2024-04-16 12:01:13,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9266
+ 2024-04-16 12:01:13,117 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9267
+ 2024-04-16 12:01:17,177 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9268
+ 2024-04-16 12:01:22,177 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9269
+ 2024-04-16 12:01:27,178 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9270
+ 2024-04-16 12:01:28,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9271
+ 2024-04-16 12:01:28,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9272
+ 2024-04-16 12:01:28,117 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9273
+ 2024-04-16 12:01:32,287 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9274
+ 2024-04-16 12:01:37,288 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9275
+ 2024-04-16 12:01:42,289 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9276
+ 2024-04-16 12:01:43,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9277
+ 2024-04-16 12:01:43,116 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9278
+ 2024-04-16 12:01:43,156 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9279
+ 2024-04-16 12:01:48,259 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9280
+ 2024-04-16 12:01:48,994 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9281
+ 2024-04-16 12:01:48,994 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9282
+ 2024-04-16 12:01:48,994 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9283
+ 2024-04-16 12:01:48,994 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9284
+ 2024-04-16 12:01:48,995 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9285
+ 2024-04-16 12:01:48,996 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9286
+ 2024-04-16 12:01:48,996 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9287
+ 2024-04-16 12:01:48,997 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9288
+ 2024-04-16 12:01:48,997 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9289
+ 2024-04-16 12:01:49,048 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9290
+ 2024-04-16 12:01:49,049 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9291
+ 2024-04-16 12:01:49,049 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9292
+ 2024-04-16 12:01:49,049 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9293
+ 2024-04-16 12:01:49,049 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9294
+ 2024-04-16 12:01:49,049 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9295
+ 2024-04-16 12:01:49,050 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9296
+ 2024-04-16 12:01:49,050 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9297
+ 2024-04-16 12:01:49,062 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9298
+ 2024-04-16 12:01:49,062 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9299
+ 2024-04-16 12:01:49,062 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9300
+ 2024-04-16 12:01:49,063 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9301
+ 2024-04-16 12:01:54,064 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9302
+ 2024-04-16 12:01:58,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9303
+ 2024-04-16 12:01:58,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9304
+ 2024-04-16 12:01:58,118 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9305
+ 2024-04-16 12:01:59,261 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9306
+ 2024-04-16 12:02:00,262 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9307
+ 2024-04-16 12:02:00,262 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9308
+ 2024-04-16 12:02:00,262 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9309
+ 2024-04-16 12:02:00,262 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9310
+ 2024-04-16 12:02:00,263 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9311
+ 2024-04-16 12:02:00,264 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9312
+ 2024-04-16 12:02:00,265 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9313
+ 2024-04-16 12:02:00,266 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9314
+ 2024-04-16 12:02:00,267 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9315
+ 2024-04-16 12:02:04,262 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9316
+ 2024-04-16 12:02:09,263 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9317
+ 2024-04-16 12:02:13,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9318
+ 2024-04-16 12:02:13,117 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9319
+ 2024-04-16 12:02:13,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9320
+ 2024-04-16 12:02:14,281 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9321
+ 2024-04-16 12:02:19,282 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9322
+ 2024-04-16 12:02:24,283 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9323
+ 2024-04-16 12:02:28,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9324
+ 2024-04-16 12:02:28,118 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9325
+ 2024-04-16 12:02:28,158 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9326
+ 2024-04-16 12:02:30,207 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9327
+ 2024-04-16 12:02:35,208 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9328
+ 2024-04-16 12:02:40,209 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9329
+ 2024-04-16 12:02:43,116 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9330
+ 2024-04-16 12:02:43,117 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9331
+ 2024-04-16 12:02:43,157 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9332
+ 2024-04-16 12:02:45,256 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9333
+ 2024-04-16 12:02:50,257 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9334
+ 2024-04-16 12:02:55,258 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9335
+ 2024-04-16 12:02:58,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9336
+ 2024-04-16 12:02:58,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9337
+ 2024-04-16 12:02:58,118 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9338
+ 2024-04-16 12:03:00,268 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9339
+ 2024-04-16 12:03:00,269 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9340
+ 2024-04-16 12:03:00,273 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9341
+ 2024-04-16 12:03:05,272 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9342
+ 2024-04-16 12:03:10,273 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9343
+ 2024-04-16 12:03:13,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9344
+ 2024-04-16 12:03:13,117 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9345
+ 2024-04-16 12:03:13,157 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9346
+ 2024-04-16 12:03:16,214 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9347
+ 2024-04-16 12:03:21,218 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9348
+ 2024-04-16 12:03:26,219 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9349
+ 2024-04-16 12:03:28,117 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9350
+ 2024-04-16 12:03:28,118 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9351
+ 2024-04-16 12:03:28,157 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9352
+ 2024-04-16 12:03:30,270 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9353
+ 2024-04-16 12:03:31,271 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9354
+ 2024-04-16 12:03:32,893 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9355
+ 2024-04-16 12:03:32,893 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9356
+ 2024-04-16 12:03:32,893 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9357
+ 2024-04-16 12:03:32,893 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9358
+ 2024-04-16 12:03:32,894 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9359
+ 2024-04-16 12:03:32,894 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9360
+ 2024-04-16 12:03:32,894 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9361
+ 2024-04-16 12:03:32,907 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9362
+ 2024-04-16 12:03:32,907 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9363
+ 2024-04-16 12:03:32,907 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9364
+ 2024-04-16 12:03:32,908 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9365
+ 2024-04-16 12:03:35,341 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9366
+ 2024-04-16 12:03:35,341 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9367
+ 2024-04-16 12:03:35,342 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9368
+ 2024-04-16 12:03:35,342 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9369
+ 2024-04-16 12:03:35,342 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9370
+ 2024-04-16 12:03:35,343 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9371
+ 2024-04-16 12:03:35,344 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9372
+ 2024-04-16 12:03:35,344 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9373
+ 2024-04-16 12:03:35,346 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9374
+ 2024-04-16 12:03:35,802 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9375
+ 2024-04-16 12:03:35,802 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9376
+ 2024-04-16 12:03:35,802 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9377
+ 2024-04-16 12:03:35,802 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9378
+ 2024-04-16 12:03:35,802 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9379
+ 2024-04-16 12:03:35,803 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9380
+ 2024-04-16 12:03:35,803 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9381
+ 2024-04-16 12:03:35,804 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9382
+ 2024-04-16 12:03:35,815 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9383
+ 2024-04-16 12:03:35,815 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9384
+ 2024-04-16 12:03:35,815 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9385
+ 2024-04-16 12:03:35,815 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9386
+ 2024-04-16 12:03:36,816 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9387
+ 2024-04-16 12:03:39,240 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9388
+ 2024-04-16 12:03:39,240 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9389
+ 2024-04-16 12:03:39,240 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9390
+ 2024-04-16 12:03:39,241 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9391
+ 2024-04-16 12:03:39,241 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9392
+ 2024-04-16 12:03:39,242 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9393
+ 2024-04-16 12:03:39,242 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9394
+ 2024-04-16 12:03:39,243 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9395
+ 2024-04-16 12:03:39,244 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9396
+ 2024-04-16 12:03:41,817 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9397
+ 2024-04-16 12:03:43,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9398
+ 2024-04-16 12:03:43,168 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9399
+ 2024-04-16 12:03:43,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9400
+ 2024-04-16 12:03:47,261 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9401
+ 2024-04-16 12:03:52,262 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9402
+ 2024-04-16 12:03:57,263 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9403
+ 2024-04-16 12:03:58,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9404
+ 2024-04-16 12:03:58,168 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9405
+ 2024-04-16 12:03:58,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9406
+ 2024-04-16 12:04:02,302 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9407
+ 2024-04-16 12:04:02,423 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: pause
9408
+ 2024-04-16 12:04:02,423 INFO HandlerThread:79 [handler.py:handle_request_pause():708] stopping system metrics thread
9409
+ 2024-04-16 12:04:02,423 INFO HandlerThread:79 [system_monitor.py:finish():203] Stopping system monitor
9410
+ 2024-04-16 12:04:02,424 INFO HandlerThread:79 [interfaces.py:finish():202] Joined cpu monitor
9411
+ 2024-04-16 12:04:02,424 DEBUG SystemMonitor:79 [system_monitor.py:_start():172] Starting system metrics aggregation loop
9412
+ 2024-04-16 12:04:02,424 DEBUG SystemMonitor:79 [system_monitor.py:_start():179] Finished system metrics aggregation loop
9413
+ 2024-04-16 12:04:02,425 INFO HandlerThread:79 [interfaces.py:finish():202] Joined disk monitor
9414
+ 2024-04-16 12:04:02,425 DEBUG SystemMonitor:79 [system_monitor.py:_start():183] Publishing last batch of metrics
9415
+ 2024-04-16 12:04:02,437 INFO HandlerThread:79 [interfaces.py:finish():202] Joined gpu monitor
9416
+ 2024-04-16 12:04:02,437 INFO HandlerThread:79 [interfaces.py:finish():202] Joined memory monitor
9417
+ 2024-04-16 12:04:02,438 INFO HandlerThread:79 [interfaces.py:finish():202] Joined network monitor
9418
+ 2024-04-16 12:04:02,438 DEBUG SenderThread:79 [sender.py:send():379] send: stats
9419
+ 2024-04-16 12:04:07,439 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9420
+ 2024-04-16 12:04:12,440 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9421
+ 2024-04-16 12:04:13,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9422
+ 2024-04-16 12:04:13,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9423
+ 2024-04-16 12:04:13,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9424
+ 2024-04-16 12:04:18,272 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9425
+ 2024-04-16 12:04:23,274 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9426
+ 2024-04-16 12:04:28,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9427
+ 2024-04-16 12:04:28,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9428
+ 2024-04-16 12:04:28,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9429
+ 2024-04-16 12:04:29,229 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9430
+ 2024-04-16 12:04:34,230 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9431
+ 2024-04-16 12:04:39,230 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9432
+ 2024-04-16 12:04:43,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9433
+ 2024-04-16 12:04:43,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9434
+ 2024-04-16 12:04:43,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9435
+ 2024-04-16 12:04:44,315 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9436
+ 2024-04-16 12:04:49,316 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9437
+ 2024-04-16 12:04:54,318 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9438
+ 2024-04-16 12:04:54,814 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: resume
9439
+ 2024-04-16 12:04:54,814 INFO HandlerThread:79 [handler.py:handle_request_resume():699] starting system metrics thread
9440
+ 2024-04-16 12:04:54,814 INFO HandlerThread:79 [system_monitor.py:start():194] Starting system monitor
9441
+ 2024-04-16 12:04:54,815 INFO SystemMonitor:79 [system_monitor.py:_start():158] Starting system asset monitoring threads
9442
+ 2024-04-16 12:04:54,815 INFO SystemMonitor:79 [interfaces.py:start():190] Started cpu monitoring
9443
+ 2024-04-16 12:04:54,816 INFO SystemMonitor:79 [interfaces.py:start():190] Started disk monitoring
9444
+ 2024-04-16 12:04:54,817 INFO SystemMonitor:79 [interfaces.py:start():190] Started gpu monitoring
9445
+ 2024-04-16 12:04:54,817 INFO SystemMonitor:79 [interfaces.py:start():190] Started memory monitoring
9446
+ 2024-04-16 12:04:54,818 INFO SystemMonitor:79 [interfaces.py:start():190] Started network monitoring
9447
+ 2024-04-16 12:04:58,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9448
+ 2024-04-16 12:04:58,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9449
+ 2024-04-16 12:04:58,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9450
+ 2024-04-16 12:05:00,242 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9451
+ 2024-04-16 12:05:05,243 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9452
+ 2024-04-16 12:05:10,244 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9453
+ 2024-04-16 12:05:13,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9454
+ 2024-04-16 12:05:13,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9455
+ 2024-04-16 12:05:13,208 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9456
+ 2024-04-16 12:05:15,333 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9457
+ 2024-04-16 12:05:20,334 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9458
+ 2024-04-16 12:05:25,335 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9459
+ 2024-04-16 12:05:28,167 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9460
+ 2024-04-16 12:05:28,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9461
+ 2024-04-16 12:05:28,209 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9462
+ 2024-04-16 12:05:31,317 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9463
+ 2024-04-16 12:05:36,318 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9464
+ 2024-04-16 12:05:41,324 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9465
+ 2024-04-16 12:05:43,168 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: stop_status
9466
+ 2024-04-16 12:05:43,168 DEBUG SenderThread:79 [sender.py:send_request():406] send_request: stop_status
9467
+ 2024-04-16 12:05:43,208 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: internal_messages
9468
+ 2024-04-16 12:05:46,337 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
9469
+ 2024-04-16 12:05:51,338 DEBUG HandlerThread:79 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240416_061855-mrfkrks2/logs/debug.log CHANGED
@@ -39,3 +39,63 @@ config: {}
39
  2024-04-16 11:52:52,066 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
40
  2024-04-16 11:52:52,066 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
41
  2024-04-16 11:55:22,701 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
  2024-04-16 11:52:52,066 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
40
  2024-04-16 11:52:52,066 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
41
  2024-04-16 11:55:22,701 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
42
+ 2024-04-16 11:56:09,161 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
43
+ 2024-04-16 11:56:09,161 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
44
+ 2024-04-16 11:56:09,168 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
45
+ 2024-04-16 11:56:24,889 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
46
+ 2024-04-16 11:56:24,889 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
47
+ 2024-04-16 11:56:24,898 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
48
+ 2024-04-16 11:56:25,088 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
49
+ 2024-04-16 11:56:25,088 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
50
+ 2024-04-16 11:56:25,097 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
51
+ 2024-04-16 11:56:25,469 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
52
+ 2024-04-16 11:56:25,469 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
53
+ 2024-04-16 11:56:37,216 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
54
+ 2024-04-16 11:56:37,329 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
55
+ 2024-04-16 11:56:37,329 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
56
+ 2024-04-16 11:56:55,142 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
57
+ 2024-04-16 11:56:58,017 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
58
+ 2024-04-16 11:56:58,017 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
59
+ 2024-04-16 11:57:11,339 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
60
+ 2024-04-16 11:57:11,917 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
61
+ 2024-04-16 11:57:11,918 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
62
+ 2024-04-16 11:57:14,596 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
63
+ 2024-04-16 11:57:15,117 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
64
+ 2024-04-16 11:57:15,117 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
65
+ 2024-04-16 11:57:23,878 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
66
+ 2024-04-16 11:57:24,280 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
67
+ 2024-04-16 11:57:24,280 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
68
+ 2024-04-16 11:57:33,335 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
69
+ 2024-04-16 11:57:33,737 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
70
+ 2024-04-16 11:57:33,737 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
71
+ 2024-04-16 11:57:50,312 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
72
+ 2024-04-16 11:57:50,724 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
73
+ 2024-04-16 11:57:50,724 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
74
+ 2024-04-16 11:57:57,017 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
75
+ 2024-04-16 11:57:57,418 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
76
+ 2024-04-16 11:57:57,418 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
77
+ 2024-04-16 11:58:12,387 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
78
+ 2024-04-16 11:58:12,860 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
79
+ 2024-04-16 11:58:12,860 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
80
+ 2024-04-16 11:58:17,865 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
81
+ 2024-04-16 11:58:18,263 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
82
+ 2024-04-16 11:58:18,263 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
83
+ 2024-04-16 11:58:19,342 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
84
+ 2024-04-16 11:58:21,804 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
85
+ 2024-04-16 11:58:21,804 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
86
+ 2024-04-16 11:59:07,575 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
87
+ 2024-04-16 11:59:33,714 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
88
+ 2024-04-16 11:59:33,714 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
89
+ 2024-04-16 12:01:48,993 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
90
+ 2024-04-16 12:01:49,048 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
91
+ 2024-04-16 12:01:49,048 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
92
+ 2024-04-16 12:02:00,261 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
93
+ 2024-04-16 12:03:32,892 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
94
+ 2024-04-16 12:03:32,892 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
95
+ 2024-04-16 12:03:35,341 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
96
+ 2024-04-16 12:03:35,801 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
97
+ 2024-04-16 12:03:35,801 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
98
+ 2024-04-16 12:03:39,240 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
99
+ 2024-04-16 12:04:02,423 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
100
+ 2024-04-16 12:04:02,423 INFO MainThread:34 [wandb_init.py:_pause_backend():438] pausing backend
101
+ 2024-04-16 12:04:54,813 INFO MainThread:34 [wandb_init.py:_resume_backend():443] resuming backend
wandb/run-20240416_061855-mrfkrks2/run-mrfkrks2.wandb CHANGED
Binary files a/wandb/run-20240416_061855-mrfkrks2/run-mrfkrks2.wandb and b/wandb/run-20240416_061855-mrfkrks2/run-mrfkrks2.wandb differ