2024-07-16 15:40:33,688 | INFO: Total Parameters: 924095168, Total Trainable Parameters: 495869568
2024-07-16 15:40:33,688 | INFO: Trainable Parameters:
2024-07-16 15:40:33,695 | WARNING: Detected kernel version 3.10.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2024-07-16 15:40:39,768 | INFO: Added key: store_based_barrier_key:2 to store for rank: 6
2024-07-16 15:40:43,128 | INFO: Added key: store_based_barrier_key:2 to store for rank: 0
2024-07-16 15:40:43,235 | INFO: Added key: store_based_barrier_key:2 to store for rank: 2
2024-07-16 15:40:43,777 | INFO: Added key: store_based_barrier_key:2 to store for rank: 5
2024-07-16 15:40:44,174 | INFO: Added key: store_based_barrier_key:2 to store for rank: 4
2024-07-16 15:40:44,799 | INFO: Added key: store_based_barrier_key:2 to store for rank: 1
2024-07-16 15:40:45,214 | INFO: Added key: store_based_barrier_key:2 to store for rank: 3
2024-07-16 15:40:46,146 | INFO: Added key: store_based_barrier_key:2 to store for rank: 7
2024-07-16 15:40:46,146 | INFO: Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:40:46,149 | INFO: Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:40:46,150 | INFO: Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:40:46,152 | INFO: Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:40:46,154 | INFO: Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:40:46,155 | INFO: Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:40:46,155 | INFO: Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:40:46,155 | INFO: Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:44:35,860 | INFO: Added key: store_based_barrier_key:2 to store for rank: 5
2024-07-16 15:44:36,299 | INFO: Total Parameters: 924095168, Total Trainable Parameters: 495869568
2024-07-16 15:44:36,299 | INFO: Trainable Parameters:
2024-07-16 15:44:36,306 | WARNING: Detected kernel version 3.10.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2024-07-16 15:44:39,116 | INFO: Added key: store_based_barrier_key:2 to store for rank: 2
2024-07-16 15:44:43,212 | INFO: Added key: store_based_barrier_key:2 to store for rank: 1
2024-07-16 15:44:43,924 | INFO: Added key: store_based_barrier_key:2 to store for rank: 7
2024-07-16 15:44:43,926 | INFO: Added key: store_based_barrier_key:2 to store for rank: 3
2024-07-16 15:44:44,783 | INFO: Added key: store_based_barrier_key:2 to store for rank: 6
2024-07-16 15:44:45,226 | INFO: Added key: store_based_barrier_key:2 to store for rank: 0
2024-07-16 15:44:45,864 | INFO: Waiting in store based barrier to initialize process group for rank: 5, key: store_based_barrier_key:2 (world_size=8, worker_count=7, timeout=0:30:00)
2024-07-16 15:44:45,964 | INFO: Added key: store_based_barrier_key:2 to store for rank: 4
2024-07-16 15:44:45,965 | INFO: Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:44:45,965 | INFO: Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:44:45,965 | INFO: Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:44:45,965 | INFO: Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:44:45,965 | INFO: Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:44:45,967 | INFO: Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:44:45,969 | INFO: Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:44:45,971 | INFO: Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:48:33,508 | INFO: Total Parameters: 924095168, Total Trainable Parameters: 495869568
2024-07-16 15:48:33,508 | INFO: Trainable Parameters:
2024-07-16 15:48:33,515 | WARNING: Detected kernel version 3.10.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2024-07-16 15:48:40,753 | INFO: Added key: store_based_barrier_key:2 to store for rank: 2
2024-07-16 15:48:42,601 | INFO: Added key: store_based_barrier_key:2 to store for rank: 1
2024-07-16 15:48:43,084 | INFO: Added key: store_based_barrier_key:2 to store for rank: 0
2024-07-16 15:48:43,546 | INFO: Added key: store_based_barrier_key:2 to store for rank: 6
2024-07-16 15:48:43,832 | INFO: Added key: store_based_barrier_key:2 to store for rank: 7
2024-07-16 15:48:43,948 | INFO: Added key: store_based_barrier_key:2 to store for rank: 4
2024-07-16 15:48:44,337 | INFO: Added key: store_based_barrier_key:2 to store for rank: 5
2024-07-16 15:48:44,821 | INFO: Added key: store_based_barrier_key:2 to store for rank: 3
2024-07-16 15:48:44,821 | INFO: Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:48:44,823 | INFO: Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:48:44,823 | INFO: Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:48:44,825 | INFO: Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:48:44,828 | INFO: Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:48:44,828 | INFO: Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:48:44,829 | INFO: Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:48:44,830 | INFO: Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:51:35,846 | INFO: Total Parameters: 924095168, Total Trainable Parameters: 495869568
2024-07-16 15:51:35,846 | INFO: Trainable Parameters:
2024-07-16 15:51:35,852 | WARNING: Detected kernel version 3.10.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2024-07-16 15:51:40,320 | INFO: Added key: store_based_barrier_key:2 to store for rank: 3
2024-07-16 15:51:43,850 | INFO: Added key: store_based_barrier_key:2 to store for rank: 5
2024-07-16 15:51:44,522 | INFO: Added key: store_based_barrier_key:2 to store for rank: 6
2024-07-16 15:51:44,688 | INFO: Added key: store_based_barrier_key:2 to store for rank: 7
2024-07-16 15:51:45,242 | INFO: Added key: store_based_barrier_key:2 to store for rank: 0
2024-07-16 15:51:45,726 | INFO: Added key: store_based_barrier_key:2 to store for rank: 4
2024-07-16 15:51:45,814 | INFO: Added key: store_based_barrier_key:2 to store for rank: 1
2024-07-16 15:51:46,384 | INFO: Added key: store_based_barrier_key:2 to store for rank: 2
2024-07-16 15:51:46,385 | INFO: Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:51:46,386 | INFO: Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:51:46,387 | INFO: Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:51:46,387 | INFO: Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:51:46,391 | INFO: Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:51:46,394 | INFO: Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:51:46,394 | INFO: Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2024-07-16 15:51:46,394 | INFO: Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.