Différences

Ci-dessous, les différences entre deux révisions de la page.

--- tech:notes_ordonnanceur_cluster_grid_batch_scheduler_slurm [2025/03/24 15:06] – créée - modification externe 127.0.0.1
+++ tech:notes_ordonnanceur_cluster_grid_batch_scheduler_slurm [2026/06/07 19:12] (Version actuelle) – modification externe 127.0.0.1
@@ Ligne 1: / Ligne 1: @@
+<!DOCTYPE markdown>
+{{tag>Brouillon Cluster Grid Ressource}}
+# Notes ordonnanceur cluster grid batch scheduler slurm
+## Slurm
+Liens :
+* http://cascisdi.inra.fr/sites/cascisdi.inra.fr/files/slurm_0.txt
+* https://wiki.fysik.dtu.dk/niflheim/SLURM
+* https://www.glue.umd.edu/hpcc/help/slurm-vs-moab.html
+* https://www.crc.rice.edu/wp-content/uploads/2014/08/Torque-to-SLURM-cheatsheet.pdf
+* http://slurm.schedmd.com/rosetta.pdf
+* http://www.accre.vanderbilt.edu/wp-content/uploads/2012/04/Slurm.pdf
+* https://github.com/accre/SLURM
+* http://slurm.schedmd.com/quickstart.html
+* http://slurm.schedmd.com/slurm_ug_2011/Basic_Configuration_Usage.pdf
+* https://www.unila.edu.br/sites/default/files/files/user_guide_slurm.pdf
+* https://computing.llnl.gov/tutorials/slurm/slurm.pdf
+* https://computing.llnl.gov/tutorials/bgq/
+* https://computing.llnl.gov/linux/slurm/quickstart.html
+* https://computing.llnl.gov/linux/slurm/faq.html
+* https://rc.fas.harvard.edu/resources/running-jobs/
+* http://bap-alap.blogspot.fr/2012_09_01_archive.html
+* https://fortylines.com/blog/startingWithSLURM.blog.html
+* https://github.com/ciemat-tic/codec/wiki/Slurm-cluster
+* http://manx.classiccmp.org/mirror/techpubs.sgi.com/library/manuals/5000/007-5814-001/pdf/007-5814-001.pdf
+* http://wildflower.diablonet.net/~scaron/slurmsetup.html
+* http://wiki.sc3.uis.edu.co/index.php/Slurm_Installation
+* http://eniac.cyi.ac.cy/display/UserDoc/Copy+of+Slurm+notes
+* http://www.ibm.com/developerworks/library/l-slurm-utility/index.html
+* https://www.lrz.de/services/compute/linux-cluster/batch_parallel/
+* http://www.gmpcs.lumat.u-psud.fr/spip.php?rubrique35
+* https://services-numeriques.unistra.fr/hpc/applications-disponibles/systeme-de-files-dattente-slurm.html
+* http://www.brightcomputing.com/Blog/bid/174099/Slurm-101-Basic-Slurm-Usage-for-Linux-Clusters
+* https://dashboard.hpc.unimelb.edu.au/started/
+API
+* http://slurm.schedmd.com/slurm_ug_2012/pyslurm.pdf
+Voir aussi :
+* https://aws.amazon.com/fr/batch/use-cases
+* https://oar.imag.fr/
+## A faire
+MPI with Slurm
+* http://slurm.schedmd.com/mpi_guide.html
+* openmpi
+* https://www.hpc2n.umu.se/batchsystem/slurm_info
+* hwloc-nox (Portable Linux Processor Affinity (PLPA))
+* https://www.hpc2n.umu.se/batchsystem/slurm_info
+* https://computing.llnl.gov/linux/slurm/mpi_guide.html
+* https://computing.llnl.gov/tutorials/openMP/ProcessThreadAffinity.pdf
+* https://www.open-mpi.org/faq/?category=slurm
+* http://stackoverflow.com/questions/31848608/slurms-srun-slower-than-mpirun
+* https://www.rc.colorado.edu/support/examples-and-tutorials/parallel-mpi-jobs.html
+* http://www.brightcomputing.com/Blog/bid/149455/How-to-run-an-OpenMPI-job-in-Bright-Cluster-Manager-through-Slurm
+* http://www.hpc2n.umu.se/node/875
+## Install
+Slurm utilisant par défaut **munge** pour faire le lien entre les comptes des machines **il faut que toutes les machines aient l'horloge synchronisées**
+Manager :
+~~~bash
+apt-get install slurm-wlm
+~~~
+Nœuds :
+~~~bash
+apt-get install -y slurmd slurm-wlm-basic-plugins
+~~~
+Manager et Nœuds
+~~~bash
+systemctl enable munge.service
+~~~
+~~~bash
+zcat /usr/share/doc/slurm-client/examples/slurm.conf.simple.gz > /etc/slurm-llnl/slurm.conf
+~~~
+Il faut adapter slurm.conf, il peut-être généré à partir de :
+* /usr/share/doc/slurmctld/slurm-wlm-configurator.easy.html
+* /usr/share/doc/slurmctld/slurm-wlm-configurator.html
+* https://computing.llnl.gov/linux/slurm/configurator.html
+On copie le même fichier de conf sur les nœuds (le même fichier sur le manager que sur les nœuds)
+~~~bash
+scp -3 vmdeb1:/etc/slurm-llnl/slurm.conf vmdeb2:/etc/slurm-llnl/slurm.conf
+scp -3 vmdeb1:/etc/munge/munge.key vmdeb2:/etc/munge/munge.key
+~~~
+Lister les "daemons" démarrés
+~~~bash
+scontrol show daemons
+~~~
+Sur le maître (ControlMachine) : slurmctld slurmd \\
+Sur les nœuds : slurmd
+`/etc/slurm-llnl/slurm.conf`
+~~~ini
+# slurm.conf file generated by configurator easy.html.
+# Put this file on all nodes of your cluster.
+# See the slurm.conf man page for more information.
+#
+ControlMachine=vmdeb1
+#ControlAddr=127.0.0.1
+#
+#MailProg=/bin/mail
+#MpiDefault=none
+MpiDefault=openmpi
+MpiParams=ports=12000-12999
+#MpiParams=ports=#-#
+#ProctrackType=proctrack/pgid
+Proctracktype=proctrack/linuxproc
+SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
+#SlurmctldPort=6817
+SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
+#SlurmdPort=6818
+SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
+SlurmUser=slurm
+#SlurmdUser=root
+#UsePAM=1
+DisableRootJobs=YES
+EnforcePartLimits=YES
+JobRequeue=0
+ReturnToService=1
+#TopologyPlugin=topology/tree
+# Must be writable by user SlurmUser. The file must be accessible by the primary and backup control machines.
+# On NFS share !? See http://manx.classiccmp.org/mirror/techpubs.sgi.com/library/manuals/5000/007-5814-001/pdf/007-5814-001.pdf
+StateSaveLocation=/var/lib/slurm-llnl/slurmctld
+SwitchType=switch/none
+#TaskPlugin=task/none
+#TaskPlugin=task/cgroup
+TaskPlugin=task/affinity
+#
+#
+# TIMERS
+#KillWait=30
+#MinJobAge=300
+#SlurmctldTimeout=120
+#SlurmdTimeout=300
+Waittime=0
+#
+#
+# SCHEDULING
+FastSchedule=1
+SchedulerType=sched/backfill
+#SchedulerPort=7321
+SelectType=select/linear
+#
+#
+# LOGGING AND ACCOUNTING
+ClusterName=cluster1
+#JobAcctGatherFrequency=30
+JobAcctGatherType=jobacct_gather/linux
+#SlurmctldDebug=3
+SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
+#SlurmdDebug=3
+SlurmdLogFile=/var/log/slurm-llnl/slurmd.log
+SlurmSchedLogFile=/var/log/slurm-llnl/slurmSched.log
+#JobCompType=jobcomp/filetxt
+#JobCompType=jobcomp/mysql
+JobCompType=jobcomp/none
+JobCompLoc=/var/log/slurm-llnl/jobcomp
+#JobCheckpointDir=/var/lib/slurm-llnl/checkpoint
+#AccountingStorageType=jobacct_gather/linux
+#AccountingStorageType=accounting_storage/filetxt
+AccountingStorageType=accounting_storage/slurmdbd
+AccountingStoreJobComment=YES
+DefaultStorageType=accounting_storage/slurmdbd
+#AccountingStorageLoc=/var/log/slurm-llnl/accounting
+AccountingStoragePort=6819
+AccountingStorageEnforce=associations
+#
+#
+NodeName=vmdeb1
+# COMPUTE NODES
+NodeName=DEFAULT
+PartitionName=DEFAULT MaxTime=INFINITE State=UP
+NodeName=vmdeb2 CPUs=1 RealMemory=494 State=UNKNOWN
+NodeName=vmdeb3 CPUs=2 RealMemory=494 TmpDisk=8000 State=UNKNOWN
+PartitionName=debug Nodes=vmdeb[2-3] Default=YES MaxTime=INFINITE  Shared=YES State=UP
+~~~
+### Install de slurmdbd
+Il est recommandé d'utiliser MySQL (pas toutes les fonctionnalité avec PostgreSQL, dommage)
+Ici on part du principe que vous avez déjà une base de donnés MySQL et compte et droits crée.
+~~~bash
+apt-get install slurmdbd
+zcat /usr/share/doc/slurmdbd/examples/slurmdbd.conf.simple.gz > /etc/slurm-llnl/slurmdbd.conf
+~~~
+On adapte le fichier slurmdbd.conf
+Puis
+~~~bash
+service slurmdbd restart
+~~~
+On test
+~~~
+sacct
+       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode
+------------ ---------- ---------- ---------- ---------- ---------- --------
+~~~
+## Pb
+~~~bash
+munge -n | ssh vmdeb1  unmunge
+~~~
+~~~
+STATUS:           Expired credential (15)
+~~~
+Solution :
+~~~bash
+ntpdate -u pool.ntp.org
+~~~
+~~~
+sudo -u slurm -- /usr/sbin/slurmctld -Dcvvvv
+/usr/sbin/slurmd    -Dcvvvv
+-c : Clear : Efface l'etat précedent, purge les jobs...
+-D : Deamon : Lancement en arrière plan. Logs sur STDOUT
+-v : Verbose : Mode bavare. Mettre plusieurs "v" pour être très bavare
+slurmd -C
+Affiche la configuration de l'hôte courant
+Aide
+Le **man**
+et
+commande --help
+commande --usage
+Variables :
+SQUEUE_STATES=all for the squeue command to display jobs in any state. (y compris les job en COMPLETED et CANCELLED)
+Commande :
+sbatch
+salloc
+srun
+sattach
+srun -l --ntasks-per-core=1 --exclusive -n 2 hostname
+sinfo --Node
+scontrol show partition
+scancel --user=test --state=pending
+scontrol show config
+scontrol show job
+scancel -i --user=test
+# The Slurm -d singleton argument tells Slurm not to dispatch this job until all previous jobs with the same name have completed.
+sbatch -d singleton simple.sh
+scontrol ping
+sinfo -R
+# Afficher egalement les jobs terminés
+squeue -t all
+#A/I/O/T = "active(in use)/idle/other/total"
+sinfo -l
+#
+sinfo -Nle -o '%n %C %t'
+~~~
+### Astuce
+#### Lancer une commande **srun** sans attendre
+Normalement
+~~~
+$ srun -N2 -l hostname
+srun: job 219 queued and waiting for resources
+~~~
+Solution (compte root ou le "SlurmUser")
+~~~
+# sinfo --noheader -o %N
+vmdeb[2-3]
+~~~
+~~~
+# srun -N2 --no-allocate -w vmdeb[2-3] hostname
+~~~
+-------
+Cancel / terminate a job in "CG" state
+~~~bash
+scontrol update nodename=node4-blender state=down reason=hung
+scontrol update nodename=node4-blender state=idle
+~~~
+Il faudra aussi tuer le processus 'slurmstepd' sur les nœuds \\
+Problème de flux réseaux : Node => Manager:TCP6817
+PB
+</code ->
+"srun: error: Application launch failed: User not found on host"
+~~~
+Solution :
+Il faut que le même utilisateur ai le même UID sur les nœuds ainsi que sur le manager. Apparemment c'est lié à **munge**
+Il peut être intéressant d'utiliser LDAP