2024-02-08 18:00:20,766 INFO    StreamThr :1998 [internal.py:wandb_internal():86] W&B internal server running at pid: 1998, started at: 2024-02-08 18:00:20.765271
2024-02-08 18:00:20,767 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: status
2024-02-08 18:00:20,770 INFO    WriterThread:1998 [datastore.py:open_for_write():85] open: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/run-3wdew33h.wandb
2024-02-08 18:00:20,771 DEBUG   SenderThread:1998 [sender.py:send():382] send: header
2024-02-08 18:00:20,771 DEBUG   SenderThread:1998 [sender.py:send():382] send: run
2024-02-08 18:00:21,104 INFO    SenderThread:1998 [dir_watcher.py:__init__():211] watching files in: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files
2024-02-08 18:00:21,104 INFO    SenderThread:1998 [sender.py:_start_run_threads():1136] run started: 3wdew33h with start time 1707415220.764703
2024-02-08 18:00:21,108 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: check_version
2024-02-08 18:00:21,108 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: check_version
2024-02-08 18:00:21,151 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: run_start
2024-02-08 18:00:21,179 DEBUG   HandlerThread:1998 [system_info.py:__init__():32] System info init
2024-02-08 18:00:21,179 DEBUG   HandlerThread:1998 [system_info.py:__init__():47] System info init done
2024-02-08 18:00:21,179 INFO    HandlerThread:1998 [system_monitor.py:start():194] Starting system monitor
2024-02-08 18:00:21,180 INFO    SystemMonitor:1998 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-02-08 18:00:21,181 INFO    HandlerThread:1998 [system_monitor.py:probe():214] Collecting system info
2024-02-08 18:00:21,181 INFO    SystemMonitor:1998 [interfaces.py:start():190] Started cpu monitoring
2024-02-08 18:00:21,182 INFO    SystemMonitor:1998 [interfaces.py:start():190] Started disk monitoring
2024-02-08 18:00:21,182 INFO    SystemMonitor:1998 [interfaces.py:start():190] Started gpu monitoring
2024-02-08 18:00:21,184 INFO    SystemMonitor:1998 [interfaces.py:start():190] Started memory monitoring
2024-02-08 18:00:21,184 INFO    SystemMonitor:1998 [interfaces.py:start():190] Started network monitoring
2024-02-08 18:00:21,239 DEBUG   HandlerThread:1998 [system_info.py:probe():196] Probing system
2024-02-08 18:00:21,241 DEBUG   HandlerThread:1998 [gitlib.py:_init_repo():56] git repository is invalid
2024-02-08 18:00:21,241 DEBUG   HandlerThread:1998 [system_info.py:probe():244] Probing system done
2024-02-08 18:00:21,241 DEBUG   HandlerThread:1998 [system_monitor.py:probe():223] {'os': 'Linux-4.14.336-253.554.amzn2.x86_64-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2024-02-08T18:00:21.239172', 'startedAt': '2024-02-08T18:00:20.760995', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': '/home/sagemaker-user/output-7b-26k-lora/../lora_finetuning_push_to_hub_save_local.py', 'codePathLocal': None, 'host': 'default', 'username': 'sagemaker-user', 'executable': '/opt/conda/bin/python3', 'cpu_count': 96, 'cpu_count_logical': 192, 'cpu_freq': {'current': 3011.898104166666, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2708.532, 'min': 0.0, 'max': 0.0}, {'current': 2549.917, 'min': 0.0, 'max': 0.0}, {'current': 2649.342, 'min': 0.0, 'max': 0.0}, {'current': 2617.578, 'min': 0.0, 'max': 0.0}, {'current': 2514.64, 'min': 0.0, 'max': 0.0}, {'current': 3300.018, 'min': 0.0, 'max': 0.0}, {'current': 2514.814, 'min': 0.0, 'max': 0.0}, {'current': 2823.715, 'min': 0.0, 'max': 0.0}, {'current': 2444.923, 'min': 0.0, 'max': 0.0}, {'current': 2450.425, 'min': 0.0, 'max': 0.0}, {'current': 2405.045, 'min': 0.0, 'max': 0.0}, {'current': 3300.611, 'min': 0.0, 'max': 0.0}, {'current': 2521.7, 'min': 0.0, 'max': 0.0}, {'current': 2536.269, 'min': 0.0, 'max': 0.0}, {'current': 2454.76, 'min': 0.0, 'max': 0.0}, {'current': 2452.125, 'min': 0.0, 'max': 0.0}, {'current': 2578.243, 'min': 0.0, 'max': 0.0}, {'current': 2588.493, 'min': 0.0, 'max': 0.0}, {'current': 2597.93, 'min': 0.0, 'max': 0.0}, {'current': 2362.191, 'min': 0.0, 'max': 0.0}, {'current': 2603.048, 'min': 0.0, 'max': 0.0}, {'current': 1726.583, 'min': 0.0, 'max': 0.0}, {'current': 2010.457, 'min': 0.0, 'max': 0.0}, {'current': 1983.584, 'min': 0.0, 'max': 0.0}, {'current': 2585.543, 'min': 0.0, 'max': 0.0}, {'current': 2448.628, 'min': 0.0, 'max': 0.0}, {'current': 2473.293, 'min': 0.0, 'max': 0.0}, {'current': 2326.422, 'min': 0.0, 'max': 0.0}, {'current': 2469.471, 'min': 0.0, 'max': 0.0}, {'current': 1679.725, 'min': 0.0, 'max': 0.0}, {'current': 1707.891, 'min': 0.0, 'max': 0.0}, {'current': 2608.243, 'min': 0.0, 'max': 0.0}, {'current': 1825.706, 'min': 0.0, 'max': 0.0}, {'current': 1852.851, 'min': 0.0, 'max': 0.0}, {'current': 2741.597, 'min': 0.0, 'max': 0.0}, {'current': 2763.617, 'min': 0.0, 'max': 0.0}, {'current': 2668.973, 'min': 0.0, 'max': 0.0}, {'current': 1848.906, 'min': 0.0, 'max': 0.0}, {'current': 1859.662, 'min': 0.0, 'max': 0.0}, {'current': 1872.243, 'min': 0.0, 'max': 0.0}, {'current': 2750.258, 'min': 0.0, 'max': 0.0}, {'current': 2171.724, 'min': 0.0, 'max': 0.0}, {'current': 2673.366, 'min': 0.0, 'max': 0.0}, {'current': 2686.6, 'min': 0.0, 'max': 0.0}, {'current': 2254.628, 'min': 0.0, 'max': 0.0}, {'current': 1922.174, 'min': 0.0, 'max': 0.0}, {'current': 1919.649, 'min': 0.0, 'max': 0.0}, {'current': 1914.156, 'min': 0.0, 'max': 0.0}, {'current': 3299.749, 'min': 0.0, 'max': 0.0}, {'current': 3299.574, 'min': 0.0, 'max': 0.0}, {'current': 3299.179, 'min': 0.0, 'max': 0.0}, {'current': 3300.775, 'min': 0.0, 'max': 0.0}, {'current': 3299.087, 'min': 0.0, 'max': 0.0}, {'current': 3299.329, 'min': 0.0, 'max': 0.0}, {'current': 2200.612, 'min': 0.0, 'max': 0.0}, {'current': 2744.304, 'min': 0.0, 'max': 0.0}, {'current': 3300.722, 'min': 0.0, 'max': 0.0}, {'current': 3299.792, 'min': 0.0, 'max': 0.0}, {'current': 3299.956, 'min': 0.0, 'max': 0.0}, {'current': 3301.117, 'min': 0.0, 'max': 0.0}, {'current': 2226.489, 'min': 0.0, 'max': 0.0}, {'current': 3070.859, 'min': 0.0, 'max': 0.0}, {'current': 1992.465, 'min': 0.0, 'max': 0.0}, {'current': 3048.294, 'min': 0.0, 'max': 0.0}, {'current': 2694.481, 'min': 0.0, 'max': 0.0}, {'current': 3299.499, 'min': 0.0, 'max': 0.0}, {'current': 2794.422, 'min': 0.0, 'max': 0.0}, {'current': 3292.924, 'min': 0.0, 'max': 0.0}, {'current': 3299.232, 'min': 0.0, 'max': 0.0}, {'current': 3297.378, 'min': 0.0, 'max': 0.0}, {'current': 3300.519, 'min': 0.0, 'max': 0.0}, {'current': 3300.987, 'min': 0.0, 'max': 0.0}, {'current': 3298.537, 'min': 0.0, 'max': 0.0}, {'current': 2474.441, 'min': 0.0, 'max': 0.0}, {'current': 2795.47, 'min': 0.0, 'max': 0.0}, {'current': 2412.968, 'min': 0.0, 'max': 0.0}, {'current': 2550.34, 'min': 0.0, 'max': 0.0}, {'current': 3009.464, 'min': 0.0, 'max': 0.0}, {'current': 2578.135, 'min': 0.0, 'max': 0.0}, {'current': 2404.379, 'min': 0.0, 'max': 0.0}, {'current': 3020.541, 'min': 0.0, 'max': 0.0}, {'current': 3058.728, 'min': 0.0, 'max': 0.0}, {'current': 2989.512, 'min': 0.0, 'max': 0.0}, {'current': 3059.601, 'min': 0.0, 'max': 0.0}, {'current': 2714.737, 'min': 0.0, 'max': 0.0}, {'current': 2694.293, 'min': 0.0, 'max': 0.0}, {'current': 3261.941, 'min': 0.0, 'max': 0.0}, {'current': 2586.965, 'min': 0.0, 'max': 0.0}, {'current': 3294.605, 'min': 0.0, 'max': 0.0}, {'current': 3260.796, 'min': 0.0, 'max': 0.0}, {'current': 3295.156, 'min': 0.0, 'max': 0.0}, {'current': 3253.521, 'min': 0.0, 'max': 0.0}, {'current': 2832.733, 'min': 0.0, 'max': 0.0}, {'current': 3264.153, 'min': 0.0, 'max': 0.0}, {'current': 2767.538, 'min': 0.0, 'max': 0.0}, {'current': 3300.784, 'min': 0.0, 'max': 0.0}, {'current': 2342.793, 'min': 0.0, 'max': 0.0}, {'current': 2291.218, 'min': 0.0, 'max': 0.0}, {'current': 2343.318, 'min': 0.0, 'max': 0.0}, {'current': 2377.449, 'min': 0.0, 'max': 0.0}, {'current': 2182.822, 'min': 0.0, 'max': 0.0}, {'current': 3300.271, 'min': 0.0, 'max': 0.0}, {'current': 2180.405, 'min': 0.0, 'max': 0.0}, {'current': 2708.15, 'min': 0.0, 'max': 0.0}, {'current': 2217.841, 'min': 0.0, 'max': 0.0}, {'current': 2223.228, 'min': 0.0, 'max': 0.0}, {'current': 2332.486, 'min': 0.0, 'max': 0.0}, {'current': 2585.795, 'min': 0.0, 'max': 0.0}, {'current': 2332.572, 'min': 0.0, 'max': 0.0}, {'current': 2256.902, 'min': 0.0, 'max': 0.0}, {'current': 2334.544, 'min': 0.0, 'max': 0.0}, {'current': 2350.832, 'min': 0.0, 'max': 0.0}, {'current': 2424.323, 'min': 0.0, 'max': 0.0}, {'current': 2456.004, 'min': 0.0, 'max': 0.0}, {'current': 2457.45, 'min': 0.0, 'max': 0.0}, {'current': 2579.468, 'min': 0.0, 'max': 0.0}, {'current': 2458.733, 'min': 0.0, 'max': 0.0}, {'current': 1860.983, 'min': 0.0, 'max': 0.0}, {'current': 2198.204, 'min': 0.0, 'max': 0.0}, {'current': 2131.955, 'min': 0.0, 'max': 0.0}, {'current': 2416.616, 'min': 0.0, 'max': 0.0}, {'current': 2498.284, 'min': 0.0, 'max': 0.0}, {'current': 2409.271, 'min': 0.0, 'max': 0.0}, {'current': 2442.917, 'min': 0.0, 'max': 0.0}, {'current': 2387.494, 'min': 0.0, 'max': 0.0}, {'current': 1840.589, 'min': 0.0, 'max': 0.0}, {'current': 1851.316, 'min': 0.0, 'max': 0.0}, {'current': 2572.071, 'min': 0.0, 'max': 0.0}, {'current': 1846.514, 'min': 0.0, 'max': 0.0}, {'current': 1838.129, 'min': 0.0, 'max': 0.0}, {'current': 2458.548, 'min': 0.0, 'max': 0.0}, {'current': 2468.97, 'min': 0.0, 'max': 0.0}, {'current': 2573.521, 'min': 0.0, 'max': 0.0}, {'current': 1854.263, 'min': 0.0, 'max': 0.0}, {'current': 1852.806, 'min': 0.0, 'max': 0.0}, {'current': 1826.095, 'min': 0.0, 'max': 0.0}, {'current': 2445.646, 'min': 0.0, 'max': 0.0}, {'current': 2189.591, 'min': 0.0, 'max': 0.0}, {'current': 2264.349, 'min': 0.0, 'max': 0.0}, {'current': 2377.441, 'min': 0.0, 'max': 0.0}, {'current': 2012.723, 'min': 0.0, 'max': 0.0}, {'current': 1906.662, 'min': 0.0, 'max': 0.0}, {'current': 2124.071, 'min': 0.0, 'max': 0.0}, {'current': 1895.988, 'min': 0.0, 'max': 0.0}, {'current': 3300.029, 'min': 0.0, 'max': 0.0}, {'current': 2525.577, 'min': 0.0, 'max': 0.0}, {'current': 3299.997, 'min': 0.0, 'max': 0.0}, {'current': 2670.847, 'min': 0.0, 'max': 0.0}, {'current': 3299.699, 'min': 0.0, 'max': 0.0}, {'current': 3299.175, 'min': 0.0, 'max': 0.0}, {'current': 2197.86, 'min': 0.0, 'max': 0.0}, {'current': 2528.763, 'min': 0.0, 'max': 0.0}, {'current': 3197.634, 'min': 0.0, 'max': 0.0}, {'current': 3211.437, 'min': 0.0, 'max': 0.0}, {'current': 3251.423, 'min': 0.0, 'max': 0.0}, {'current': 2959.085, 'min': 0.0, 'max': 0.0}, {'current': 2065.145, 'min': 0.0, 'max': 0.0}, {'current': 2974.061, 'min': 0.0, 'max': 0.0}, {'current': 2068.626, 'min': 0.0, 'max': 0.0}, {'current': 2986.901, 'min': 0.0, 'max': 0.0}, {'current': 2340.617, 'min': 0.0, 'max': 0.0}, {'current': 3032.214, 'min': 0.0, 'max': 0.0}, {'current': 2431.142, 'min': 0.0, 'max': 0.0}, {'current': 3302.396, 'min': 0.0, 'max': 0.0}, {'current': 2944.793, 'min': 0.0, 'max': 0.0}, {'current': 3299.677, 'min': 0.0, 'max': 0.0}, {'current': 3298.69, 'min': 0.0, 'max': 0.0}, {'current': 2920.67, 'min': 0.0, 'max': 0.0}, {'current': 3090.867, 'min': 0.0, 'max': 0.0}, {'current': 2503.478, 'min': 0.0, 'max': 0.0}, {'current': 2473.681, 'min': 0.0, 'max': 0.0}, {'current': 2485.275, 'min': 0.0, 'max': 0.0}, {'current': 2488.179, 'min': 0.0, 'max': 0.0}, {'current': 2990.001, 'min': 0.0, 'max': 0.0}, {'current': 2452.206, 'min': 0.0, 'max': 0.0}, {'current': 2729.116, 'min': 0.0, 'max': 0.0}, {'current': 3001.225, 'min': 0.0, 'max': 0.0}, {'current': 3299.389, 'min': 0.0, 'max': 0.0}, {'current': 3299.765, 'min': 0.0, 'max': 0.0}, {'current': 3299.125, 'min': 0.0, 'max': 0.0}, {'current': 2002.363, 'min': 0.0, 'max': 0.0}, {'current': 2302.227, 'min': 0.0, 'max': 0.0}, {'current': 3132.468, 'min': 0.0, 'max': 0.0}, {'current': 2770.449, 'min': 0.0, 'max': 0.0}, {'current': 3288.126, 'min': 0.0, 'max': 0.0}, {'current': 3298.022, 'min': 0.0, 'max': 0.0}, {'current': 3298.691, 'min': 0.0, 'max': 0.0}, {'current': 3297.894, 'min': 0.0, 'max': 0.0}, {'current': 2828.863, 'min': 0.0, 'max': 0.0}, {'current': 3299.914, 'min': 0.0, 'max': 0.0}, {'current': 2721.595, 'min': 0.0, 'max': 0.0}, {'current': 3299.427, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 32.0, 'used': 0.01256561279296875}}, 'gpu': 'NVIDIA A10G', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA A10G', 'memory_total': 24146608128}, {'name': 'NVIDIA A10G', 'memory_total': 24146608128}, {'name': 'NVIDIA A10G', 'memory_total': 24146608128}, {'name': 'NVIDIA A10G', 'memory_total': 24146608128}, {'name': 'NVIDIA A10G', 'memory_total': 24146608128}, {'name': 'NVIDIA A10G', 'memory_total': 24146608128}, {'name': 'NVIDIA A10G', 'memory_total': 24146608128}, {'name': 'NVIDIA A10G', 'memory_total': 24146608128}], 'memory': {'total': 747.9597625732422}}
2024-02-08 18:00:21,241 INFO    HandlerThread:1998 [system_monitor.py:probe():224] Finished collecting system info
2024-02-08 18:00:21,241 INFO    HandlerThread:1998 [system_monitor.py:probe():227] Publishing system info
2024-02-08 18:00:21,241 DEBUG   HandlerThread:1998 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
2024-02-08 18:00:21,242 DEBUG   HandlerThread:1998 [system_info.py:_save_pip():68] Saving pip packages done
2024-02-08 18:00:21,242 DEBUG   HandlerThread:1998 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
2024-02-08 18:00:22,105 INFO    Thread-12 :1998 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/requirements.txt
2024-02-08 18:00:22,105 INFO    Thread-12 :1998 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/conda-environment.yaml
2024-02-08 18:00:35,540 DEBUG   HandlerThread:1998 [system_info.py:_save_conda():87] Saving conda packages done
2024-02-08 18:00:35,542 INFO    HandlerThread:1998 [system_monitor.py:probe():229] Finished publishing system info
2024-02-08 18:00:35,545 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: status_report
2024-02-08 18:00:35,545 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: keepalive
2024-02-08 18:00:35,545 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: status_report
2024-02-08 18:00:35,545 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: keepalive
2024-02-08 18:00:35,546 DEBUG   SenderThread:1998 [sender.py:send():382] send: files
2024-02-08 18:00:35,546 INFO    SenderThread:1998 [sender.py:_save_file():1392] saving file wandb-metadata.json with policy now
2024-02-08 18:00:35,550 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: stop_status
2024-02-08 18:00:35,550 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: stop_status
2024-02-08 18:00:35,557 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-08 18:00:35,673 DEBUG   SenderThread:1998 [sender.py:send():382] send: telemetry
2024-02-08 18:00:35,673 DEBUG   SenderThread:1998 [sender.py:send():382] send: config
2024-02-08 18:00:35,674 DEBUG   SenderThread:1998 [sender.py:send():382] send: metric
2024-02-08 18:00:35,674 DEBUG   SenderThread:1998 [sender.py:send():382] send: telemetry
2024-02-08 18:00:35,674 DEBUG   SenderThread:1998 [sender.py:send():382] send: metric
2024-02-08 18:00:35,674 WARNING SenderThread:1998 [sender.py:send_metric():1343] Seen metric with glob (shouldn't happen)
2024-02-08 18:00:35,893 INFO    wandb-upload_0:1998 [upload_job.py:push():131] Uploaded file /tmp/tmp_9e35467wandb/s8xuyfes-wandb-metadata.json
2024-02-08 18:00:36,107 INFO    Thread-12 :1998 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/conda-environment.yaml
2024-02-08 18:00:36,107 INFO    Thread-12 :1998 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/output.log
2024-02-08 18:00:36,108 INFO    Thread-12 :1998 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/wandb-metadata.json
2024-02-08 18:00:36,534 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: status_report
2024-02-08 18:00:38,108 INFO    Thread-12 :1998 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/output.log
2024-02-08 18:00:38,575 DEBUG   SenderThread:1998 [sender.py:send():382] send: exit
2024-02-08 18:00:38,575 INFO    SenderThread:1998 [sender.py:send_exit():589] handling exit code: 1
2024-02-08 18:00:38,575 INFO    SenderThread:1998 [sender.py:send_exit():591] handling runtime: 17
2024-02-08 18:00:38,575 INFO    SenderThread:1998 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-08 18:00:38,575 INFO    SenderThread:1998 [sender.py:send_exit():597] send defer
2024-02-08 18:00:38,575 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:38,576 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 0
2024-02-08 18:00:38,576 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:38,576 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 0
2024-02-08 18:00:38,576 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 1
2024-02-08 18:00:38,576 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:38,576 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 1
2024-02-08 18:00:38,576 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:38,576 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 1
2024-02-08 18:00:38,576 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 2
2024-02-08 18:00:38,576 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:38,576 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 2
2024-02-08 18:00:38,576 INFO    HandlerThread:1998 [system_monitor.py:finish():203] Stopping system monitor
2024-02-08 18:00:38,578 INFO    HandlerThread:1998 [interfaces.py:finish():202] Joined cpu monitor
2024-02-08 18:00:38,578 INFO    HandlerThread:1998 [interfaces.py:finish():202] Joined disk monitor
2024-02-08 18:00:38,578 DEBUG   SystemMonitor:1998 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-02-08 18:00:38,578 DEBUG   SystemMonitor:1998 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-02-08 18:00:38,578 DEBUG   SystemMonitor:1998 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-02-08 18:00:38,618 INFO    HandlerThread:1998 [interfaces.py:finish():202] Joined gpu monitor
2024-02-08 18:00:38,618 INFO    HandlerThread:1998 [interfaces.py:finish():202] Joined memory monitor
2024-02-08 18:00:38,618 INFO    HandlerThread:1998 [interfaces.py:finish():202] Joined network monitor
2024-02-08 18:00:38,618 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:38,619 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 2
2024-02-08 18:00:38,619 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 3
2024-02-08 18:00:38,619 DEBUG   SenderThread:1998 [sender.py:send():382] send: stats
2024-02-08 18:00:38,620 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:38,620 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 3
2024-02-08 18:00:38,620 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:38,620 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 3
2024-02-08 18:00:38,620 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 4
2024-02-08 18:00:38,620 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:38,620 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 4
2024-02-08 18:00:38,620 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:38,620 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 4
2024-02-08 18:00:38,620 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 5
2024-02-08 18:00:38,620 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:38,621 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 5
2024-02-08 18:00:38,621 DEBUG   SenderThread:1998 [sender.py:send():382] send: summary
2024-02-08 18:00:38,633 INFO    SenderThread:1998 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
2024-02-08 18:00:38,633 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:38,633 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 5
2024-02-08 18:00:38,633 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 6
2024-02-08 18:00:38,634 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:38,634 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 6
2024-02-08 18:00:38,634 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:38,634 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 6
2024-02-08 18:00:38,638 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: status_report
2024-02-08 18:00:38,773 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 7
2024-02-08 18:00:38,773 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:38,773 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 7
2024-02-08 18:00:38,774 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:38,774 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 7
2024-02-08 18:00:39,108 INFO    Thread-12 :1998 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/config.yaml
2024-02-08 18:00:39,108 INFO    Thread-12 :1998 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/wandb-summary.json
2024-02-08 18:00:39,575 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: poll_exit
2024-02-08 18:00:39,685 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 8
2024-02-08 18:00:39,685 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: poll_exit
2024-02-08 18:00:39,686 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:39,686 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 8
2024-02-08 18:00:39,686 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:39,686 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 8
2024-02-08 18:00:39,686 INFO    SenderThread:1998 [job_builder.py:build():298] Attempting to build job artifact
2024-02-08 18:00:39,688 INFO    SenderThread:1998 [job_builder.py:_get_source_type():439] no source found
2024-02-08 18:00:39,688 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 9
2024-02-08 18:00:39,688 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:39,688 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 9
2024-02-08 18:00:39,688 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:39,688 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 9
2024-02-08 18:00:39,688 INFO    SenderThread:1998 [dir_watcher.py:finish():358] shutting down directory watcher
2024-02-08 18:00:40,109 INFO    Thread-12 :1998 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/output.log
2024-02-08 18:00:40,109 INFO    SenderThread:1998 [dir_watcher.py:finish():388] scan: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files
2024-02-08 18:00:40,109 INFO    SenderThread:1998 [dir_watcher.py:finish():402] scan save: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/config.yaml config.yaml
2024-02-08 18:00:40,109 INFO    SenderThread:1998 [dir_watcher.py:finish():402] scan save: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/requirements.txt requirements.txt
2024-02-08 18:00:40,109 INFO    SenderThread:1998 [dir_watcher.py:finish():402] scan save: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/conda-environment.yaml conda-environment.yaml
2024-02-08 18:00:40,109 INFO    SenderThread:1998 [dir_watcher.py:finish():402] scan save: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/wandb-metadata.json wandb-metadata.json
2024-02-08 18:00:40,110 INFO    SenderThread:1998 [dir_watcher.py:finish():402] scan save: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/output.log output.log
2024-02-08 18:00:40,111 INFO    SenderThread:1998 [dir_watcher.py:finish():402] scan save: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/wandb-summary.json wandb-summary.json
2024-02-08 18:00:40,111 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 10
2024-02-08 18:00:40,112 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:40,112 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 10
2024-02-08 18:00:40,114 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:40,114 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 10
2024-02-08 18:00:40,114 INFO    SenderThread:1998 [file_pusher.py:finish():175] shutting down file pusher
2024-02-08 18:00:40,385 INFO    wandb-upload_0:1998 [upload_job.py:push():131] Uploaded file /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/config.yaml
2024-02-08 18:00:40,437 INFO    wandb-upload_3:1998 [upload_job.py:push():131] Uploaded file /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/output.log
2024-02-08 18:00:40,470 INFO    wandb-upload_2:1998 [upload_job.py:push():131] Uploaded file /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/conda-environment.yaml
2024-02-08 18:00:40,560 INFO    wandb-upload_1:1998 [upload_job.py:push():131] Uploaded file /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/requirements.txt
2024-02-08 18:00:40,575 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: poll_exit
2024-02-08 18:00:40,576 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: poll_exit
2024-02-08 18:00:40,674 INFO    wandb-upload_4:1998 [upload_job.py:push():131] Uploaded file /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/files/wandb-summary.json
2024-02-08 18:00:40,875 INFO    Thread-11 (_thread_body):1998 [sender.py:transition_state():617] send defer: 11
2024-02-08 18:00:40,875 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:40,875 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 11
2024-02-08 18:00:40,875 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:40,875 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 11
2024-02-08 18:00:40,875 INFO    SenderThread:1998 [file_pusher.py:join():181] waiting for file pusher
2024-02-08 18:00:40,876 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 12
2024-02-08 18:00:40,876 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:40,876 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 12
2024-02-08 18:00:40,876 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:40,876 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 12
2024-02-08 18:00:40,876 INFO    SenderThread:1998 [file_stream.py:finish():595] file stream finish called
2024-02-08 18:00:40,954 INFO    SenderThread:1998 [file_stream.py:finish():599] file stream finish is done
2024-02-08 18:00:40,954 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 13
2024-02-08 18:00:40,954 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:40,954 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 13
2024-02-08 18:00:40,954 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:40,954 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 13
2024-02-08 18:00:40,954 INFO    SenderThread:1998 [sender.py:transition_state():617] send defer: 14
2024-02-08 18:00:40,955 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: defer
2024-02-08 18:00:40,955 INFO    HandlerThread:1998 [handler.py:handle_request_defer():172] handle defer: 14
2024-02-08 18:00:40,955 DEBUG   SenderThread:1998 [sender.py:send():382] send: final
2024-02-08 18:00:40,955 DEBUG   SenderThread:1998 [sender.py:send():382] send: footer
2024-02-08 18:00:40,955 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: defer
2024-02-08 18:00:40,955 INFO    SenderThread:1998 [sender.py:send_request_defer():613] handle sender defer: 14
2024-02-08 18:00:40,955 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: poll_exit
2024-02-08 18:00:40,956 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: poll_exit
2024-02-08 18:00:40,956 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: poll_exit
2024-02-08 18:00:40,956 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: server_info
2024-02-08 18:00:40,956 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: poll_exit
2024-02-08 18:00:40,957 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: server_info
2024-02-08 18:00:40,959 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: get_summary
2024-02-08 18:00:40,959 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: sampled_history
2024-02-08 18:00:40,959 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-08 18:00:40,959 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: job_info
2024-02-08 18:00:41,013 DEBUG   SenderThread:1998 [sender.py:send_request():409] send_request: job_info
2024-02-08 18:00:41,013 INFO    MainThread:1998 [wandb_run.py:_footer_history_summary_info():3837] rendering history
2024-02-08 18:00:41,013 INFO    MainThread:1998 [wandb_run.py:_footer_history_summary_info():3869] rendering summary
2024-02-08 18:00:41,013 INFO    MainThread:1998 [wandb_run.py:_footer_sync_info():3796] logging synced files
2024-02-08 18:00:41,014 DEBUG   HandlerThread:1998 [handler.py:handle_request():146] handle_request: shutdown
2024-02-08 18:00:41,014 INFO    HandlerThread:1998 [handler.py:finish():866] shutting down handler
2024-02-08 18:00:41,960 INFO    WriterThread:1998 [datastore.py:close():294] close: /home/sagemaker-user/output-7b-26k-lora/wandb/run-20240208_180020-3wdew33h/run-3wdew33h.wandb
2024-02-08 18:00:42,013 INFO    SenderThread:1998 [sender.py:finish():1548] shutting down sender
2024-02-08 18:00:42,013 INFO    SenderThread:1998 [file_pusher.py:finish():175] shutting down file pusher
2024-02-08 18:00:42,013 INFO    SenderThread:1998 [file_pusher.py:join():181] waiting for file pusher