Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bugfix in communication trace parsing #104

Closed
wants to merge 1 commit into from
Closed

Bugfix in communication trace parsing #104

wants to merge 1 commit into from

Conversation

TaekyungHeo
Copy link
Contributor

@TaekyungHeo TaekyungHeo commented Apr 23, 2024 •
edited
Loading

Summary

Bugfix in communication trace parsing

Test Plan

I ran the following command.

srun --mpi=pmix \
        -N 2 \ 
        --ntasks-per-node 8 \ 
        --container-image=... \ 
        --container-mounts=... \ 
        --output=stdout.txt \ 
        --error=stderr.txt \ 
        run.sh

You can find run.sh below.

#!/bin/bash 

# Log the process ID and local ID 
echo "Starting script on $(hostname)" 
echo "SLURM_PROCID: $SLURM_PROCID, SLURM_LOCALID: $SLURM_LOCALID" 
  
if [ "$SLURM_LOCALID" -eq 0 ]; then 
    # SLURM_LOCALID is zero, indicating this is the process designated for installation 
    echo "Performing installation on $(hostname) [SLURM_PROCID: $SLURM_PROCID, SLURM_LOCALID: $SLURM_LOCALID]" 
     
    cd /workspace 
    git clone https://github.com/TaekyungHeo/param.git 
    cd param 
    git checkout comms-bugfix 
    pip install -r requirements.txt 
    cd /workspace/param/train/comms/pt 
    pip install . 
    cd /workspace/param/train/compute/python 
    pip install -r requirements.txt 
    pip install . 
     
    echo "Installation completed on $(hostname) [SLURM_PROCID: $SLURM_PROCID, SLURM_LOCALID: $SLURM_LOCALID]" 
else 
    # For all other processes, wait until the designated package is installed 
    echo "Waiting for installation to complete on $(hostname) [SLURM_PROCID: $SLURM_PROCID, SLURM_LOCALID: $SLURM_LOCALID]" 
     
    sleep 120 
fi 
  
# Now proceed to execute your main process 
echo "Starting process on $(hostname)" 
python /workspace/param/train/comms/pt/commsTraceReplay.py --trace-type et --trace-path <trace path> --backend nccl --device cuda --reuse-tensors

stderr.txt

...
Dtype=None with PG: default group
[2024-04-23 15:46:33,427][root][INFO][Rank 10] - [Replay 0][Rank  10] [15551 / 15552] Replaying barrier: NumElemsIn=None, NumElemsOut=None, Dtype=None with PG: default group
[2024-04-23 15:46:33,427][root][INFO][Rank  0] - [Replay 0][15551 / 15552] Replayed barrier in block [Unamed/Unknown]... 60.93 us
[2024-04-23 15:46:33,427][root][INFO][Rank 11] - [Replay 0][Rank  11] [15552 / 15552] Replaying wait: NumElemsIn=None, NumElemsOut=None, Dtype=None with PG: default group
[2024-04-23 15:46:33,427][root][INFO][Rank  2] - [Replay 0][Rank   2] [15552 / 15552] Replaying wait: NumElemsIn=None, NumElemsOut=None, Dtype=None with PG: default group
[2024-04-23 15:46:33,427][root][INFO][Rank  9] - [Replay 0][Rank   9] [15552 / 15552] Replaying wait: NumElemsIn=None, NumElemsOut=None, Dtype=None with PG: default group
[2024-04-23 15:46:33,427][root][INFO][Rank  4] - [Replay 0][Rank   4] [15551 / 15552] Replaying barrier: NumElemsIn=None, NumElemsOut=None, Dtype=None with PG: default group
[2024-04-23 15:46:33,427][root][INFO][Rank 13] - [Replay 0][Rank  13] [15552 / 15552] Replaying wait: NumElemsIn=None, NumElemsOut=None, Dtype=None with PG: default group
[2024-04-23 15:46:33,427][root][INFO][Rank  7] - [Replay 0][Rank   7] [15552 / 15552] Replaying wait: NumElemsIn=None, NumElemsOut=None, Dtype=None with PG: default group
[2024-04-23 15:46:33,427][root][INFO][Rank 14] - [Replay 0][Rank  14] [15552 / 15552] Replaying wait: NumElemsIn=None, NumElemsOut=None, Dtype=None with PG: default group
[2024-04-23 15:46:33,427][root][INFO][Rank  0] - [Replay 0][Rank   0] [15552 / 15552] Replaying wait: NumElemsIn=None, NumElemsOut=None, Dtype=None with PG: default group
[2024-04-23 15:46:33,427][root][INFO][Rank  0] - [Replay 0][15552 / 15552] Replayed wait in block [Unamed/Unknown]... 64.35 us
[2024-04-23 15:46:33,427][root][INFO][Rank  4] - [Replay 0][Rank   4] [15552 / 15552] Replaying wait: NumElemsIn=None, NumElemsOut=None, Dtype=None with PG: default group
[2024-04-23 15:46:33,427][root][INFO][Rank 10] - [Replay 0][Rank  10] [15552 / 15552] Replaying wait: NumElemsIn=None, NumElemsOut=None, Dtype=None with PG: default group
[2024-04-23 15:46:33,580][root][INFO][Rank  0] - 
==================== Message size Statistcs ====================

stdout.txt

...
Hello from Rank 1: [Rank   1] host eos0031.eos.clusters.nvidia.com, device: cuda:1, local_rank: 1 world_size: 16, master_ip: eos0031
Hello from Rank 2: [Rank   2] host eos0031.eos.clusters.nvidia.com, device: cuda:2, local_rank: 2 world_size: 16, master_ip: eos0031
Hello from Rank 3: [Rank   3] host eos0031.eos.clusters.nvidia.com, device: cuda:3, local_rank: 3 world_size: 16, master_ip: eos0031
Hello from Rank 4: [Rank   4] host eos0031.eos.clusters.nvidia.com, device: cuda:4, local_rank: 4 world_size: 16, master_ip: eos0031
Hello from Rank 5: [Rank   5] host eos0031.eos.clusters.nvidia.com, device: cuda:5, local_rank: 5 world_size: 16, master_ip: eos0031
Hello from Rank 6: [Rank   6] host eos0031.eos.clusters.nvidia.com, device: cuda:6, local_rank: 6 world_size: 16, master_ip: eos0031
Hello from Rank 7: [Rank   7] host eos0031.eos.clusters.nvidia.com, device: cuda:7, local_rank: 7 world_size: 16, master_ip: eos0031
Hello from Rank 8: [Rank   8] host eos0032.eos.clusters.nvidia.com, device: cuda:0, local_rank: 0 world_size: 16, master_ip: eos0031
Hello from Rank 9: [Rank   9] host eos0032.eos.clusters.nvidia.com, device: cuda:1, local_rank: 1 world_size: 16, master_ip: eos0031
Hello from Rank 10: [Rank  10] host eos0032.eos.clusters.nvidia.com, device: cuda:2, local_rank: 2 world_size: 16, master_ip: eos0031
Hello from Rank 11: [Rank  11] host eos0032.eos.clusters.nvidia.com, device: cuda:3, local_rank: 3 world_size: 16, master_ip: eos0031
Hello from Rank 12: [Rank  12] host eos0032.eos.clusters.nvidia.com, device: cuda:4, local_rank: 4 world_size: 16, master_ip: eos0031
Hello from Rank 13: [Rank  13] host eos0032.eos.clusters.nvidia.com, device: cuda:5, local_rank: 5 world_size: 16, master_ip: eos0031
Hello from Rank 14: [Rank  14] host eos0032.eos.clusters.nvidia.com, device: cuda:6, local_rank: 6 world_size: 16, master_ip: eos0031
Hello from Rank 15: [Rank  15] host eos0032.eos.clusters.nvidia.com, device: cuda:7, local_rank: 7 world_size: 16, master_ip: eos0031

+++++ 15552 msgs recorded in /home/theo/scratch/meta-traces/ +++++

--------------------------------------------------
+ 3300 reduce_scatter_base
--------------------------------------------------
Size of Input tensors (bytes)
 Total (MB)            Max.       Min.       Average           p50           p95
2675307.81   2147500032.00 809535488.00    850079868.12    809535488.00    809535488.00
Size of Output tensors (bytes)
 Total (MB)            Max.       Min.       Average           p50           p95
 167206.74    134218752.00 50595968.00     53129991.76     50595968.00     50595968.00
--------------------------------------------------
+ 1224 all_reduce
--------------------------------------------------
Size of Input tensors (bytes)
 Total (MB)            Max.       Min.       Average           p50           p95
      0.04          192.00       1.00           30.92            4.00          192.00
Size of Output tensors (bytes)
 Total (MB)            Max.       Min.       Average           p50           p95
      0.04          192.00       1.00           30.92            4.00          192.00

==================== Performance of replayed comms ====================
--------------------------------------------------
 Total latency (us) of comms in trace 7346423.357: 
--------------------------------------------------
--------------------------------------------------
 Replayed 11024 wait (50.17%): 
--------------------------------------------------
Latency (us)
      Total       Max.       Min.    Average        p50        p95
 3278511.34 1624661.69       1.16     297.40       1.71       4.26
--------------------------------------------------
 Replayed 3300 reduce_scatter_base (30.86%): 
--------------------------------------------------
Latency (us)
      Total       Max.       Min.    Average        p50        p95
 2016983.79    5416.98      20.69     611.21      22.97    1982.82
--------------------------------------------------
 Replayed 1224 all_reduce (18.90%): 
--------------------------------------------------
Latency (us)
      Total       Max.       Min.    Average        p50        p95
 1235344.22    2733.59      19.40    1009.27    1279.10    2025.04
--------------------------------------------------
 Replayed 4 barrier (0.06%): 
--------------------------------------------------
Latency (us)
      Total       Max.       Min.    Average        p50        p95
    4237.59    2087.03      60.93    1059.40    1044.81    2077.98

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 23, 2024
@TaekyungHeo
Copy link
Contributor Author

Close based on the discussion with Srinivas and Brian.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kingchc kingchc Awaiting requested review from kingchc kingchc is a code owner

@louisfeng louisfeng Awaiting requested review from louisfeng louisfeng is a code owner

@sunghlin sunghlin Awaiting requested review from sunghlin sunghlin is a code owner

@wfu-fb wfu-fb Awaiting requested review from wfu-fb

@shengbao-zheng shengbao-zheng Awaiting requested review from shengbao-zheng shengbao-zheng is a code owner

@briancoutinho briancoutinho Awaiting requested review from briancoutinho briancoutinho is a code owner

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@TaekyungHeo @facebook-github-bot