How to debug Can't find an address, check slurm.conf errors when launching jobs?

[yaroslavvb]

Our .yaml file is here

Trying a job with --nodes=10 works if job is small enough to launch on static nodes, but fail when dynamics nodes are used.

Any tips how to troubleshoot this?

After waiting 10 mins for nodes to allocate, the failure looks like this

srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-28: Can't find an address, check slurm.conf

Things I checked:

1. Slurmd running on ghpc-28? Yes

÷23-07-15 00:44:32|1|alpha5-ultra-ghpc-28|~\÷ ps aux | grep slurmd
root        5545  0.0  0.0  79824  8160 ?        Ss   00:17   0:00 /usr/local/sbin/slurmd -D -s -N alpha5-ultra-ghpc-28 --conf-server=alpha5-controller:6820-6830

2. Can I launch jobs from ghpc-28? Yes

÷23-07-15 00:44:46|1|alpha5-ultra-ghpc-28|~\÷ srun hostname
alpha5-ultra-ghpc-4
÷23-07-15 00:45:04|1|alpha5-ultra-ghpc-28|~\÷ 

3. what happens I launch large job from login node?

run: error: fwd_tree_thread: can't find address for host alpha5-ultra-ghpc-43, check slurm.conf
srun: error: fwd_tree_thread: can't find address for host alpha5-ultra-ghpc-38, check slurm.conf
srun: error: fwd_tree_thread: can't find address for host alpha5-ultra-ghpc-45, check slurm.conf
srun: error: fwd_tree_thread: can't find address for host alpha5-ultra-ghpc-34, check slurm.conf
srun: error: fwd_tree_thread: can't find address for host alpha5-ultra-ghpc-40, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-40: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-34: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-45: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-38: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-43: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-42: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-41: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-47: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-46: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-44: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-35: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-37: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-36: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-39: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-33: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-28: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-32: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-29: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-30: Can't find an address, check slurm.conf
srun: error: Task launch for StepId=293.0 failed on node alpha5-ultra-ghpc-31: Can't find an address, check slurm.conf
srun: error: Application launch failed: Can't find an address, check slurm.conf
srun: Job step aborted

4. Controller logs don't seem to have anything informative

==> slurmctld.log <==                                                                                                                                 
[2023-07-15T00:17:41.269] Node alpha5-ultra-ghpc-28 now responding                                                                                    
[2023-07-15T00:17:49.324] Node alpha5-ultra-ghpc-29 now responding                                                                                    
[2023-07-15T00:18:05.231] job_time_limit: Configuration for JobId=272 complete                                                                        
[2023-07-15T00:18:05.231] Resetting JobId=272 start time for node power up                                                                            
[2023-07-15T00:18:06.285] _job_complete: JobId=272 WTERMSIG 116            
[2023-07-15T00:18:06.288] _job_complete: JobId=272 done                                                                                               
[2023-07-15T00:18:26.257] sched: _slurm_rpc_allocate_resources JobId=273 NodeList=alpha5-ultra-ghpc-[25-26] usec=5093                                 
[2023-07-15T00:19:16.686] cleanup_completing: JobId=272 completion process took 70 seconds                                                            
[2023-07-15T00:19:49.290] sched: _slurm_rpc_allocate_resources JobId=274 NodeList=alpha5-ultra-ghpc-[15-24,28-31] usec=4901                           
                                                                                                                                                      
==> resume.log <==                                                                                                                                    
2023-07-15 00:19:50,097 INFO: resume alpha5-ultra-ghpc-[30-31]                                                                                        
2023-07-15 00:22:29,320 INFO: created 2 instances: nodes=alpha5-ultra-ghpc-[30-31]                                                                    
2023-07-15 00:22:29,324 INFO: create 2 subscriptions (alpha5-ultra-ghpc-[30-31])                                                                      
2023-07-15 00:22:40,855 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-30                            
2023-07-15 00:22:45,070 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-31                            
                                                                                                                                                      
==> slurmctld.log <==                                                                                                                                 
[2023-07-15T00:23:59.039] Node alpha5-ultra-ghpc-31 now responding                                                                                    
[2023-07-15T00:24:06.445] Node alpha5-ultra-ghpc-30 now responding                                                                                    
[2023-07-15T00:24:35.557] job_time_limit: Configuration for JobId=274 complete                                                                        
[2023-07-15T00:24:35.557] Resetting JobId=274 start time for node power up                                                                            
[2023-07-15T00:24:42.783] _job_complete: JobId=274 WTERMSIG 116                                                                                       
[2023-07-15T00:24:42.786] _job_complete: JobId=274 done                                                                                               
[2023-07-15T00:25:48.781] sched: _slurm_rpc_allocate_resources JobId=275 NodeList=alpha5-ultra-ghpc-[16-23,28-53] usec=6059                           
                                                                           
==> resume.log <==                                                                                                                                    
2023-07-15 00:25:49,131 INFO: resume alpha5-ultra-ghpc-[32-53]                                                                                        
                                                                                                                                                      
==> slurmctld.log <==                                                                                                                                 
[2023-07-15T00:25:53.180] cleanup_completing: JobId=274 completion process took 71 seconds                                                            
                                                                                                                                                      
==> resume.log <==                                                                                                                                    
2023-07-15 00:29:05,617 INFO: created 22 instances: nodes=alpha5-ultra-ghpc-[32-53]                                                                   
2023-07-15 00:29:05,621 INFO: create 22 subscriptions (alpha5-ultra-ghpc-[32-53])                                                                     
2023-07-15 00:29:15,506 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-44                            
2023-07-15 00:29:15,814 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-32                            
2023-07-15 00:29:16,537 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-34                            
2023-07-15 00:29:16,755 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-39                            
2023-07-15 00:29:16,858 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-36                            
2023-07-15 00:29:17,364 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-49                            
2023-07-15 00:29:26,790 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-40                            
2023-07-15 00:29:26,916 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-46                            
2023-07-15 00:29:27,055 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-33                            
2023-07-15 00:29:28,445 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-45                            
2023-07-15 00:29:28,632 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-38                            
2023-07-15 00:29:28,705 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-51                            
2023-07-15 00:29:34,573 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-52                            
2023-07-15 00:29:35,808 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-47                            
2023-07-15 00:29:36,193 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-53                            
2023-07-15 00:29:37,792 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-42                            
2023-07-15 00:29:38,844 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-50                            
2023-07-15 00:29:40,418 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-37                            
2023-07-15 00:29:43,607 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-43                            
2023-07-15 00:29:43,997 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-41                            
                                                                                                                                                      
==> slurmctld.log <==                                                                                                                                 
[2023-07-15T00:29:44.823] Node alpha5-ultra-ghpc-36 now responding                                                                                    
[2023-07-15T00:29:45.140] Node alpha5-ultra-ghpc-51 now responding         
[2023-07-15T00:29:45.148] Node alpha5-ultra-ghpc-48 now responding                                                                                    
                                                                                                                                                      
==> resume.log <==                                                                                                                                    
2023-07-15 00:29:45,925 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-48                            
                                                                                                                                                      
==> slurmctld.log <==                                                      
[2023-07-15T00:29:46.501] Node alpha5-ultra-ghpc-44 now responding                                                                                    
[2023-07-15T00:29:52.442] Node alpha5-ultra-ghpc-53 now responding                                                                                    
[2023-07-15T00:29:52.552] Node alpha5-ultra-ghpc-46 now responding         
[2023-07-15T00:29:53.331] Node alpha5-ultra-ghpc-35 now responding                                                                                    
[2023-07-15T00:29:53.708] Node alpha5-ultra-ghpc-49 now responding                                                                                    
[2023-07-15T00:29:54.251] Node alpha5-ultra-ghpc-47 now responding                                                                                    
                                                                                                                                                      
==> resume.log <==                                                                                                                                    
2023-07-15 00:29:55,708 INFO: Subscription created: projects/contextual-research-common/subscriptions/alpha5-ultra-ghpc-35                            
                                                                                                                                                      
==> slurmctld.log <==                                                                                                                                 
[2023-07-15T00:29:56.504] Node alpha5-ultra-ghpc-39 now responding                                                                                    
[2023-07-15T00:29:59.860] Node alpha5-ultra-ghpc-52 now responding                                                                                    
[2023-07-15T00:30:00.009] Node alpha5-ultra-ghpc-34 now responding         
[2023-07-15T00:30:00.802] Node alpha5-ultra-ghpc-32 now responding         
[2023-07-15T00:30:02.283] Node alpha5-ultra-ghpc-45 now responding                                                                                    
[2023-07-15T00:30:07.322] Node alpha5-ultra-ghpc-50 now responding                                                                                    
[2023-07-15T00:30:08.097] Node alpha5-ultra-ghpc-40 now responding                                                                                    
[2023-07-15T00:30:33.832] Node alpha5-ultra-ghpc-37 now responding                                                                                  

[rohitramu]

Thank you for the detailed description!

I have been unable to reproduce the issue so far (I'm running into unrelated errors, e.g. quota limits). I will continue to look into this on Monday.

[mr0re1]

Not sure if your cluster is still around, but here is what could be done:
Could you please provide ?

scontrol show node alpha5-ultra-ghpc-28
sudo grep alpha5-ultra-ghpc /slurm/scripts/log/slurmsync.log

We could try to reset the node by manually killing the instance , waiting until its gone, then executing
scontrol update nodename=alpha5-ultra-ghpc-28 state=power_down_force
It's not debugging, but an unblocking attempt.

[mr0re1]

Closing the discussion, please reopen if new info will became available.