Jeff Squyres, profile picture
Uploaded byJeff Squyres
PPTX, PDF3,474 views

Open MPI Explorations in Process Affinity (EuroMPI'13 presentation)

The document discusses Open MPI's Locality-Aware Mapping Algorithm (LAMA) interface for controlling process placement on parallel machines. It describes how LAMA allows users to specify regular mapping patterns of processes to resources. It also outlines the three main steps in MPI process placement with LAMA: 1) mapping processes to resources, 2) ordering processes, and 3) binding processes during launch according to the mapping. The goal is to provide a mechanism for exploring different process placements to minimize communication costs.

Embed presentation

Downloaded 46 times
Advancing Application Process Affinity Experimentation: Open MPI's LAMA-Based Affinity Interface Jeff Squyres September 18, 2013 Joshua Hursey
Locality Matters • Multiple talks here at EuroMPI’13 about network locality • Goals:  Minimize data transfer distance  Reduce network congestion and contention • …this also matters inside the server, too!
Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI 8086:1521 eth0 PCI 8086:1521 eth1 PCI 8086:1521 eth2 PCI 8086:1521 eth3 PCI 1137:0043 eth4 PCI 1137:0043 eth5 PCI 102b:0522 NUMANode P#1 (64GB) Socket P#1 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#8 PU P#24 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#9 PU P#25 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#10 PU P#26 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#11 PU P#27 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#12 PU P#28 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#13 PU P#29 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#14 PU P#30 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#15 PU P#31 PCI 1000:005b sda sdb PCI 1137:0043 eth6 PCI 1137:0043 eth7 Intel Xeon E5-2690 (“Sandy Bridge”) 2 sockets, 8 cores, 64GB per socket 1G NICs 10G NICs 10G NICs L1 and L2 Shared L3 Hyperthreading enabled
The intent of this work is to provide a mechanism that allows users to explore the process-placement space within the scope of their own applications. A User’s Playground
LAMA • Locality-Aware Mapping Algorithm (LAMA)  Supports a wide range of regular mapping patterns. • Adapts at runtime to available hardware  Supports homogeneous and heterogeneous systems. • Extensible to any depth of server topology  Naturally supports potentially deeper topologies of future server architectures.
LAMA Inspiration • Drawn from much prior work • Most notably, heavily inspired by BlueGene/P and /Q mapping systems  LAMA’s mapping specification is similar
Launching MPI Applications • Three steps in MPI process placement 1. Mapping 2. Ordering 3. Binding • Let's discuss how these work in Open MPI
1. Mapping • Create a layout of processes-to-resources Server Server Server Server Server Server Server Server Server Server Server Server Server Server Server Server MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI
Mapping • MPI's runtime must create a map, pairing processes-to-processors (and memory). • Basic technique:  Gather hwloc topologies from allocated nodes.  Mapping agent then makes a plan for which resources are assigned to processes
Mapping Agent • Act of planning mappings:  Specify which process will be launched on each server  Identify if any hardware resource will be oversubscribed • Processes are mapped to the resolution of a single processing unit (PU)  Smallest unit of allocation: hardware thread  In HPC, usually the same as a processor core
Oversubscription • Common / usual definition:  When a single PU is assigned more than one process • Complicating the definition:  Some application may need more than one PU per process (multithreaded applications) • How can the user express what their application means by “oversubscription”?
2. Ordering: By “Slot” Assigning MCW ranks to mapped processes 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 36 40 44 48 49 50 51 64 65 66 67 80
2. Ordering: By Node Assigning MCW ranks to mapped processes 0 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 1 17 33 49 65 81 97 113 129 145 161 177 193 209 225 241 2 66 130 194 4 20 36 52 5 23 37 53 6
Ordering • Each process must be assigned a unique rank in MPI_COMM_WORLD • Two common types of ordering:  natural • The order in which processes are mapped determines their rank in MCW  sequential • The processes are sequentially numbered starting at the first processing unit, and continuing until the last processing unit
3. Binding • Launch processes and enforce the layout Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI 8086:1521 eth0 PCI 8086:1521 eth1 PCI 8086:1521 eth2 PCI 8086:1521 eth3 PCI 1137:0043 eth4 PCI 1137:0043 eth5 PCI 102b:0522 NUMANode P#1 (64GB) Socket P#1 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#8 PU P#24 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#9 PU P#25 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#10 PU P#26 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#11 PU P#27 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#12 PU P#28 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#13 PU P#29 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#14 PU P#30 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#15 PU P#31 PCI 1000:005b sda sdb PCI 1137:0043 eth6 PCI 1137:0043 eth7 Indexes: physical Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI 8086:1521 eth0 PCI 8086:1521 eth1 PCI 8086:1521 eth2 PCI 8086:1521 eth3 PCI 1137:0043 eth4 PCI 1137:0043 eth5 PCI 102b:0522 NUMANode P#1 (64GB) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI 8086:1521 eth0 PCI 8086:1521 eth1 PCI 8086:1521 eth2 PCI 8086:1521 eth3 PCI 1137: eth4 PCI 1137: eth5 PCI 102b:0522 NUMANode P#1 (64GB) Socket P#1 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#8 PU P#24 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#9 PU P#25 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#10 PU P#26 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#11 PU P#27 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#12 PU P#28 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#13 PU P#29 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#14 PU P#30 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#15 PU P#31 PCI 1000:005b sda sdb PCI 1137: eth6 PCI 1137: eth7 Indexes: physical Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI 8086:1521 eth0 PCI 8086:1521 eth1 PCI 8086:1521 eth2 PCI 8086:1521 eth3 PCI 1137: eth4 PCI 1137: eth5 PCI 102b:0522 NUMANode P#1 (64GB) 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (25 L1d (3 L1i (3 Core PU PU NUMANode P#1 (64GB) Socket P#1 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#8 PU P#24 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#9 PU P#25 L2 (25 L1d (3 L1i (3 Core PU PU Indexes: physical Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (25 L1d (3 L1i (3 Core PU PU NUMANode P#1 (64GB) 32 33 3 40 41 4 Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) PCI 8086:1521 eth0 PCI 8086:1521 Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) PCI 8086:1521 eth0 PCI 8086:1521 Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L2 (256KB) L2 (25
Binding • Process-launching agent working with the OS to limit where each process can run: 1. No restrictions 2. Limited set of restrictions 3. Specific resource restrictions • “Binding width”  The number of PUs to which a process is bound
Command Line Interface (CLI) • 4 levels of abstraction for the user  Level 1: None  Level 2: Simple, common patterns  Level 3: LAMA process layout regular patterns  Level 4: Irregular patterns
CLI: Level 1 (none) • No mapping or binding options specified  May or may not specify the number of processes to launch (-np)  If not specified, default to the number of cores available in the allocation  One process is mapped to each core in the system in a "by-core" style  Processes are not bound • …for backwards compatibility reasons 
CLI: Level 2 (common) • Simple, common patterns for mapping and binding  Specify mapping pattern with • --map-by X (e.g., --map-by socket)  Specify binding option with: • --bind-to Y (e.g., --bind-to core)  All of these options are translated to Level 3 options for processing by LAMA (full list of X / Y values shown later)
CLI: Level 3 (regular patterns) • LAMA process layout regular patterns  Power users wanting something unique for their application  Four MCA run-time parameters • rmaps_lama_map: Mapping process layout • rmaps_lama_bind: Binding width • rmaps_lama_order: Ordering of MCW ranks • rmaps_lama_mppr: Maximum allowable number of processes per resource (oversubscription)
rmaps_lama_map (map) • Takes as an argument the "process layout"  A series of nine tokens • allowing 9! (362,880) mapping permutation options.  Preferred iteration order for LAMA • innermost iteration specified first • outermost iteration specified last
Example system 2 servers (nodes), 4 sockets, 2 cores, 2 PUs
rmaps_lama_map (map) • map=scbnh (a.k.a., by socket, then by core)
rmaps_lama_map (map) • map=scbnh (a.k.a., by socket, then by core)
rmaps_lama_map (map) • map=scbnh (a.k.a., by socket, then by core)
rmaps_lama_map (map) • map=scbnh (a.k.a., by socket, then by core)
rmaps_lama_map (map) • map=scbnh (a.k.a., by socket, then by core)
rmaps_lama_bind (bind) • “Binding width" and layer • Example: bind=3c (3 cores)Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI e PCI e PCI e PCI e bind = 3c
Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 rmaps_lama_bind (bind) • “Binding width" and layer • Example: bind=2s (2 sockets) Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI 8086:1521 eth0 PCI 8086:1521 eth1 PCI 8086:1521 eth2 PCI 8086:1521 eth3 PCI 1137:0043 eth4 PCI 1137:0043 eth5 PCI 102b:0522 bind = 2s bind = 2s
rmaps_lama_bind (bind) • “Binding width" and layer • Example: bind=12 (all PUs in an L2) bind = 12
rmaps_lama_bind (bind) • “Binding width" and layer • Example: bind=1N (all PUs in NUMA locality) bind = 1N
rmaps_lama_order (order) • Select which ranks are assigned to processes in MCW • There are other possible orderings, but no one has asked for them yet… Natural order for map-by-node (default) Sequential order for any mapping
rmaps_lama_mppr (mppr) • mppr (mip-per) sets the Maximum number of allowable Processes Per Resource  User-specified definition of oversubscription • Comma-delimited list of <#:resource>  1:c  At most one process per core  1:c,2:s  At most one process per core, and at most two processes per socket
MPPR  1:c  At most one process per coreMachine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23
MPPR  1:c,2:s  At most one process per core and two processes per socket Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23
CLI: Level 4 (rankfile) • Complete specification of processor-to- resource mapping description  Bypasses LAMA • Not described in the paper
Level 2 to Level 3 Chart
Remember the prior example? • -np 24 -mppr 2:c -map scbnh
Same example, different mapping • -np 24 -mppr 2:c -map nbsch
• Displays prettyprint representation of the binding actually used for each process.  Visual feedback = quite helpful when exploring mpirun -np 4 --mca rmaps lama --mca rmaps_lama_bind 1c --mca rmaps_lama_map nbsch --mca rmaps_lama_mppr 1:c --report- bindings hello_world MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..] MCW rank 1 bound to socket 1[core 8[hwt 0-1]]: [../../../../../../../..][BB/../../../../../../..] MCW rank 2 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../../../..][../../../../../../../..] MCW rank 3 bound to socket 1[core 9[hwt 0-1]]: [../../../../../../../..][../BB/../../../../../..] Report Bindings
Future Work • Available in Open MPI v1.7.2 (and later) • Open questions to users:  Are more flexible ordering options useful?  What common mapping patterns are useful?  What additional features would you like to see?
Thank You

More Related Content

PDF
(Open) MPI, Parallel Computing, Life, the Universe, and Everything
byJeff Squyres
 
PPTX
2016 NCTU P4 Workshop
byYi Tseng
 
PDF
BPF: Next Generation of Programmable Datapath
byThomas Graf
 
PDF
Cilium - Fast IPv6 Container Networking with BPF and XDP
byThomas Graf
 
PDF
p4alu: Arithmetic Logic Unit in P4
byKentaro Ebisawa
 
PDF
20170925 onos and p4
byYi Tseng
 
PDF
The Next Generation Firewall for Red Hat Enterprise Linux 7 RC
byThomas Graf
 
PDF
2015 FOSDEM - OVS Stateful Services
byThomas Graf
 
(Open) MPI, Parallel Computing, Life, the Universe, and Everything
byJeff Squyres
 
2016 NCTU P4 Workshop
byYi Tseng
 
BPF: Next Generation of Programmable Datapath
byThomas Graf
 
Cilium - Fast IPv6 Container Networking with BPF and XDP
byThomas Graf
 
p4alu: Arithmetic Logic Unit in P4
byKentaro Ebisawa
 
20170925 onos and p4
byYi Tseng
 
The Next Generation Firewall for Red Hat Enterprise Linux 7 RC
byThomas Graf
 
2015 FOSDEM - OVS Stateful Services
byThomas Graf
 

What's hot

PDF
Using GTP on Linux with libgtpnl
byKentaro Ebisawa
 
PDF
Networking and Go: An Epic Journey
bySneha Inguva
 
PDF
Ebpf ovsconf-2016
byCheng-Chun William Tu
 
PDF
Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)
byThomas Graf
 
PDF
LinuxCon 2015 Stateful NAT with OVS
byThomas Graf
 
PDF
LF_DPDK17_Lagopus Router
byLF_DPDK
 
PDF
Open vSwitch - Stateful Connection Tracking & Stateful NAT
byThomas Graf
 
PPTX
Compiling P4 to XDP, IOVISOR Summit 2017
byCheng-Chun William Tu
 
PDF
LinuxCon 2015 Linux Kernel Networking Walkthrough
byThomas Graf
 
PDF
Protecting the Privacy of the Network – Using P4 to Prototype and Extend Netw...
byOpen-NFP
 
PDF
DevConf 2014 Kernel Networking Walkthrough
byThomas Graf
 
PDF
[Webinar Slides] Programming the Network Dataplane in P4
byOpen Networking Summits
 
PDF
P4-based VNF and Micro-VNF Chaining for Servers With Intelligent Server Adapters
byOpen-NFP
 
PDF
Programming Protocol-Independent Packet Processors
byOpen Networking Summits
 
PDF
BPF - All your packets belong to me
by_xhr_
 
PDF
Network Measurement with P4 and C on Netronome Agilio
byOpen-NFP
 
PDF
Linux Native, HTTP Aware Network Security
byThomas Graf
 
PDF
P4 for Custom Identification, Flow Tagging, Monitoring and Control
byOpen-NFP
 
PDF
BPF & Cilium - Turning Linux into a Microservices-aware Operating System
byThomas Graf
 
PDF
Consensus as a Network Service
byOpen-NFP
 
Using GTP on Linux with libgtpnl
byKentaro Ebisawa
 
Networking and Go: An Epic Journey
bySneha Inguva
 
Ebpf ovsconf-2016
byCheng-Chun William Tu
 
Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)
byThomas Graf
 
LinuxCon 2015 Stateful NAT with OVS
byThomas Graf
 
LF_DPDK17_Lagopus Router
byLF_DPDK
 
Open vSwitch - Stateful Connection Tracking & Stateful NAT
byThomas Graf
 
Compiling P4 to XDP, IOVISOR Summit 2017
byCheng-Chun William Tu
 
LinuxCon 2015 Linux Kernel Networking Walkthrough
byThomas Graf
 
Protecting the Privacy of the Network – Using P4 to Prototype and Extend Netw...
byOpen-NFP
 
DevConf 2014 Kernel Networking Walkthrough
byThomas Graf
 
[Webinar Slides] Programming the Network Dataplane in P4
byOpen Networking Summits
 
P4-based VNF and Micro-VNF Chaining for Servers With Intelligent Server Adapters
byOpen-NFP
 
Programming Protocol-Independent Packet Processors
byOpen Networking Summits
 
BPF - All your packets belong to me
by_xhr_
 
Network Measurement with P4 and C on Netronome Agilio
byOpen-NFP
 
Linux Native, HTTP Aware Network Security
byThomas Graf
 
P4 for Custom Identification, Flow Tagging, Monitoring and Control
byOpen-NFP
 
BPF & Cilium - Turning Linux into a Microservices-aware Operating System
byThomas Graf
 
Consensus as a Network Service
byOpen-NFP
 

Similar to Open MPI Explorations in Process Affinity (EuroMPI'13 presentation)

PDF
R&D work on pre exascale HPC systems
byJoshua Mora
 
PDF
Buiding a better Userspace - The current and future state of QEMU and KVM int...
byaliguori
 
PPTX
Porting NetBSD to the open source LatticeMico32 CPU
byYann Sionneau
 
PPT
Distributed systems
bySyed Zaid Irshad
 
PPTX
The n00bs guide to ovs dpdk
bymarkdgray
 
PPTX
DPDK layer for porting IPS-IDS
byVipin Varghese
 
PDF
L4 Microkernel :: Design Overview
byNational Cheng Kung University
 
PPTX
Understanding DPDK
byDenys Haryachyy
 
PDF
Network operating systems
byAnkit Kumar
 
PDF
Network operating systems
bySMK Informatika Wonosobo
 
PPT
Intro (Distributed computing)
bySri Prasanna
 
PDF
Flexible and Scalable Domain-Specific Architectures
byNetronome
 
PPTX
Slideshare - PCIe
byJin Wu
 
ODP
Time to rethink /proc
byKir Kolyshkin
 
PPT
Basic Linux Internals
bymukul bhardwaj
 
PDF
Deep Dive on Amazon EC2 Instances (March 2017)
byJulien SIMON
 
PDF
[OpenStack Days Korea 2016] Track3 - OpenStack on 64-bit ARM with X-Gene
byOpenStack Korea Community
 
PDF
1083 wang
byAndre Bueno
 
PDF
CC-4005, Performance analysis of 3D Finite Difference computational stencils ...
byAMD Developer Central
 
PDF
Bài tập lớn hệ điều hành HCMUT_HK232.pdf
bydanhnguyenthanh15
 
R&D work on pre exascale HPC systems
byJoshua Mora
 
Buiding a better Userspace - The current and future state of QEMU and KVM int...
byaliguori
 
Porting NetBSD to the open source LatticeMico32 CPU
byYann Sionneau
 
Distributed systems
bySyed Zaid Irshad
 
The n00bs guide to ovs dpdk
bymarkdgray
 
DPDK layer for porting IPS-IDS
byVipin Varghese
 
L4 Microkernel :: Design Overview
byNational Cheng Kung University
 
Understanding DPDK
byDenys Haryachyy
 
Network operating systems
byAnkit Kumar
 
Network operating systems
bySMK Informatika Wonosobo
 
Intro (Distributed computing)
bySri Prasanna
 
Flexible and Scalable Domain-Specific Architectures
byNetronome
 
Slideshare - PCIe
byJin Wu
 
Time to rethink /proc
byKir Kolyshkin
 
Basic Linux Internals
bymukul bhardwaj
 
Deep Dive on Amazon EC2 Instances (March 2017)
byJulien SIMON
 
[OpenStack Days Korea 2016] Track3 - OpenStack on 64-bit ARM with X-Gene
byOpenStack Korea Community
 
1083 wang
byAndre Bueno
 
CC-4005, Performance analysis of 3D Finite Difference computational stencils ...
byAMD Developer Central
 
Bài tập lớn hệ điều hành HCMUT_HK232.pdf
bydanhnguyenthanh15
 

More from Jeff Squyres

PDF
Open MPI State of the Union X SC'16 BOF
byJeff Squyres
 
PPTX
MPI Sessions: a proposal to the MPI Forum
byJeff Squyres
 
PDF
MPI Fourm SC'15 BOF
byJeff Squyres
 
PPTX
Open MPI SC'15 State of the Union BOF
byJeff Squyres
 
PDF
Cisco's journey from Verbs to Libfabric
byJeff Squyres
 
PPTX
(Very) Loose proposal to revamp MPI_INIT and MPI_FINALIZE
byJeff Squyres
 
PPTX
Fun with Github webhooks: verifying Signed-off-by
byJeff Squyres
 
PPTX
Open MPI new version number scheme and roadmap
byJeff Squyres
 
PDF
The State of libfabric in Open MPI
byJeff Squyres
 
PPTX
Cisco usNIC libfabric provider
byJeff Squyres
 
PPTX
2014 01-21-mpi-community-feedback
byJeff Squyres
 
PDF
Cisco usNIC: how it works, how it is used in Open MPI
byJeff Squyres
 
PPTX
Cisco EuroMPI'13 vendor session presentation
byJeff Squyres
 
PDF
MPI History
byJeff Squyres
 
PPTX
MOSSCon 2013, Cisco Open Source talk
byJeff Squyres
 
PPTX
Ethernet and TCP optimizations
byJeff Squyres
 
PPTX
Friends don't let friends leak MPI_Requests
byJeff Squyres
 
PPTX
MPI-3 Timer requests proposal
byJeff Squyres
 
PPTX
MPI_Mprobe is good for you
byJeff Squyres
 
PPTX
The Message Passing Interface (MPI) in Layman's Terms
byJeff Squyres
 
Open MPI State of the Union X SC'16 BOF
byJeff Squyres
 
MPI Sessions: a proposal to the MPI Forum
byJeff Squyres
 
MPI Fourm SC'15 BOF
byJeff Squyres
 
Open MPI SC'15 State of the Union BOF
byJeff Squyres
 
Cisco's journey from Verbs to Libfabric
byJeff Squyres
 
(Very) Loose proposal to revamp MPI_INIT and MPI_FINALIZE
byJeff Squyres
 
Fun with Github webhooks: verifying Signed-off-by
byJeff Squyres
 
Open MPI new version number scheme and roadmap
byJeff Squyres
 
The State of libfabric in Open MPI
byJeff Squyres
 
Cisco usNIC libfabric provider
byJeff Squyres
 
2014 01-21-mpi-community-feedback
byJeff Squyres
 
Cisco usNIC: how it works, how it is used in Open MPI
byJeff Squyres
 
Cisco EuroMPI'13 vendor session presentation
byJeff Squyres
 
MPI History
byJeff Squyres
 
MOSSCon 2013, Cisco Open Source talk
byJeff Squyres
 
Ethernet and TCP optimizations
byJeff Squyres
 
Friends don't let friends leak MPI_Requests
byJeff Squyres
 
MPI-3 Timer requests proposal
byJeff Squyres
 
MPI_Mprobe is good for you
byJeff Squyres
 
The Message Passing Interface (MPI) in Layman's Terms
byJeff Squyres
 

Recently uploaded

PDF
CompTIA Cybersecurity Analyst (CySA+) CS0-003: Unit 2
byVICTOR MAESTRE RAMIREZ
 
PPTX
"Feelings versus facts: why metrics are more important than intuition", Igor ...
byFwdays
 
PDF
PCCC25（設立25年記念PCクラスタシンポジウム）：富士通株式会社 テーマ２「AI Computing Broker: Make your GPUs ...
byPC Cluster Consortium
 
PDF
"DISC as GPS for team leaders: how to lead a team from storming to performing...
byFwdays
 
PDF
Cybersecurity Prevention and Detection: Unit 3
byVICTOR MAESTRE RAMIREZ
 
PDF
[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...
byWintari Yasmin
 
PDF
10 Best Automation QA Testing Software Tools in 2025.pdf
byRohitBhandari66
 
PDF
Lets Build a Serverless Function with Kiro
byChris Bingham
 
PDF
[BDD 2025 - Mobile Development] Mobile Engineer and Software Engineer: Are we...
bywintari3
 
PPTX
4_ii_Analysis of Indeterminate Structures (Slope-Deflection Method).pptx
byrohitpatel130020
 
PDF
Transforming Content Operations in the Age of AI
byEnterprise Knowledge
 
PDF
[BDD 2025 - Full-Stack Development] The Modern Stack: Building Web & AI Appli...
byWintari Yasmin
 
PDF
How Much Does It Cost to Build an eCommerce Website in 2025.pdf
byPixlogix Infotech
 
PDF
Cloud Infrastructure monitoring with Grafana Cloud
byImma Valls Bernaus
 
PPTX
UFCD 0797 - SISTEMAS OPERATIVOS_Unidade Completa.pptx
byscribddobruno
 
PPTX
kernel PPT (Explanation of Windows Kernal).pptx
byINFOBYTES1
 
PPTX
MuleSoft AI Series : Introduction to MCP
byMulesoftMunichMeetup
 
PDF
So You Want to Work at Google | DevFest Seattle 2025
byMargaret Maynard-Reid
 
PDF
Parallel Computing BCS702 Module notes of the vtu college 7th sem 4.pdf
byMatheshVishnu
 
PDF
MuleSoft Meetup: Dreamforce'25 Tour- Vibing With AI & Agents.pdf
byTintu Jacob Shaji
 
CompTIA Cybersecurity Analyst (CySA+) CS0-003: Unit 2
byVICTOR MAESTRE RAMIREZ
 
"Feelings versus facts: why metrics are more important than intuition", Igor ...
byFwdays
 
PCCC25（設立25年記念PCクラスタシンポジウム）：富士通株式会社 テーマ２「AI Computing Broker: Make your GPUs ...
byPC Cluster Consortium
 
"DISC as GPS for team leaders: how to lead a team from storming to performing...
byFwdays
 
Cybersecurity Prevention and Detection: Unit 3
byVICTOR MAESTRE RAMIREZ
 
[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...
byWintari Yasmin
 
10 Best Automation QA Testing Software Tools in 2025.pdf
byRohitBhandari66
 
Lets Build a Serverless Function with Kiro
byChris Bingham
 
[BDD 2025 - Mobile Development] Mobile Engineer and Software Engineer: Are we...
bywintari3
 
4_ii_Analysis of Indeterminate Structures (Slope-Deflection Method).pptx
byrohitpatel130020
 
Transforming Content Operations in the Age of AI
byEnterprise Knowledge
 
[BDD 2025 - Full-Stack Development] The Modern Stack: Building Web & AI Appli...
byWintari Yasmin
 
How Much Does It Cost to Build an eCommerce Website in 2025.pdf
byPixlogix Infotech
 
Cloud Infrastructure monitoring with Grafana Cloud
byImma Valls Bernaus
 
UFCD 0797 - SISTEMAS OPERATIVOS_Unidade Completa.pptx
byscribddobruno
 
kernel PPT (Explanation of Windows Kernal).pptx
byINFOBYTES1
 
MuleSoft AI Series : Introduction to MCP
byMulesoftMunichMeetup
 
So You Want to Work at Google | DevFest Seattle 2025
byMargaret Maynard-Reid
 
Parallel Computing BCS702 Module notes of the vtu college 7th sem 4.pdf
byMatheshVishnu
 
MuleSoft Meetup: Dreamforce'25 Tour- Vibing With AI & Agents.pdf
byTintu Jacob Shaji
 

Open MPI Explorations in Process Affinity (EuroMPI'13 presentation)

  • 1.
    Advancing Application ProcessAffinity Experimentation: Open MPI's LAMA-Based Affinity Interface Jeff Squyres September 18, 2013 Joshua Hursey
  • 2.
    Locality Matters • Multipletalks here at EuroMPI’13 about network locality • Goals:  Minimize data transfer distance  Reduce network congestion and contention • …this also matters inside the server, too!
  • 3.
    Machine (128GB) NUMANode P#0(64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI 8086:1521 eth0 PCI 8086:1521 eth1 PCI 8086:1521 eth2 PCI 8086:1521 eth3 PCI 1137:0043 eth4 PCI 1137:0043 eth5 PCI 102b:0522 NUMANode P#1 (64GB) Socket P#1 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#8 PU P#24 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#9 PU P#25 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#10 PU P#26 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#11 PU P#27 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#12 PU P#28 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#13 PU P#29 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#14 PU P#30 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#15 PU P#31 PCI 1000:005b sda sdb PCI 1137:0043 eth6 PCI 1137:0043 eth7 Intel Xeon E5-2690 (“Sandy Bridge”) 2 sockets, 8 cores, 64GB per socket 1G NICs 10G NICs 10G NICs L1 and L2 Shared L3 Hyperthreading enabled
  • 4.
    The intent ofthis work is to provide a mechanism that allows users to explore the process-placement space within the scope of their own applications. A User’s Playground
  • 5.
    LAMA • Locality-Aware MappingAlgorithm (LAMA)  Supports a wide range of regular mapping patterns. • Adapts at runtime to available hardware  Supports homogeneous and heterogeneous systems. • Extensible to any depth of server topology  Naturally supports potentially deeper topologies of future server architectures.
  • 6.
    LAMA Inspiration • Drawnfrom much prior work • Most notably, heavily inspired by BlueGene/P and /Q mapping systems  LAMA’s mapping specification is similar
  • 7.
    Launching MPI Applications •Three steps in MPI process placement 1. Mapping 2. Ordering 3. Binding • Let's discuss how these work in Open MPI
  • 8.
    1. Mapping • Createa layout of processes-to-resources Server Server Server Server Server Server Server Server Server Server Server Server Server Server Server Server MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI MPI
  • 9.
    Mapping • MPI's runtimemust create a map, pairing processes-to-processors (and memory). • Basic technique:  Gather hwloc topologies from allocated nodes.  Mapping agent then makes a plan for which resources are assigned to processes
  • 10.
    Mapping Agent • Actof planning mappings:  Specify which process will be launched on each server  Identify if any hardware resource will be oversubscribed • Processes are mapped to the resolution of a single processing unit (PU)  Smallest unit of allocation: hardware thread  In HPC, usually the same as a processor core
  • 11.
    Oversubscription • Common /usual definition:  When a single PU is assigned more than one process • Complicating the definition:  Some application may need more than one PU per process (multithreaded applications) • How can the user express what their application means by “oversubscription”?
  • 12.
    2. Ordering: By“Slot” Assigning MCW ranks to mapped processes 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 36 40 44 48 49 50 51 64 65 66 67 80
  • 13.
    2. Ordering: ByNode Assigning MCW ranks to mapped processes 0 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 1 17 33 49 65 81 97 113 129 145 161 177 193 209 225 241 2 66 130 194 4 20 36 52 5 23 37 53 6
  • 14.
    Ordering • Each processmust be assigned a unique rank in MPI_COMM_WORLD • Two common types of ordering:  natural • The order in which processes are mapped determines their rank in MCW  sequential • The processes are sequentially numbered starting at the first processing unit, and continuing until the last processing unit
  • 15.
    3. Binding • Launchprocesses and enforce the layout Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI 8086:1521 eth0 PCI 8086:1521 eth1 PCI 8086:1521 eth2 PCI 8086:1521 eth3 PCI 1137:0043 eth4 PCI 1137:0043 eth5 PCI 102b:0522 NUMANode P#1 (64GB) Socket P#1 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#8 PU P#24 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#9 PU P#25 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#10 PU P#26 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#11 PU P#27 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#12 PU P#28 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#13 PU P#29 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#14 PU P#30 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#15 PU P#31 PCI 1000:005b sda sdb PCI 1137:0043 eth6 PCI 1137:0043 eth7 Indexes: physical Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI 8086:1521 eth0 PCI 8086:1521 eth1 PCI 8086:1521 eth2 PCI 8086:1521 eth3 PCI 1137:0043 eth4 PCI 1137:0043 eth5 PCI 102b:0522 NUMANode P#1 (64GB) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI 8086:1521 eth0 PCI 8086:1521 eth1 PCI 8086:1521 eth2 PCI 8086:1521 eth3 PCI 1137: eth4 PCI 1137: eth5 PCI 102b:0522 NUMANode P#1 (64GB) Socket P#1 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#8 PU P#24 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#9 PU P#25 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#10 PU P#26 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#11 PU P#27 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#12 PU P#28 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#13 PU P#29 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#14 PU P#30 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#15 PU P#31 PCI 1000:005b sda sdb PCI 1137: eth6 PCI 1137: eth7 Indexes: physical Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI 8086:1521 eth0 PCI 8086:1521 eth1 PCI 8086:1521 eth2 PCI 8086:1521 eth3 PCI 1137: eth4 PCI 1137: eth5 PCI 102b:0522 NUMANode P#1 (64GB) 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (25 L1d (3 L1i (3 Core PU PU NUMANode P#1 (64GB) Socket P#1 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#8 PU P#24 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#9 PU P#25 L2 (25 L1d (3 L1i (3 Core PU PU Indexes: physical Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (25 L1d (3 L1i (3 Core PU PU NUMANode P#1 (64GB) 32 33 3 40 41 4 Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) PCI 8086:1521 eth0 PCI 8086:1521 Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) PCI 8086:1521 eth0 PCI 8086:1521 Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L2 (256KB) L2 (25
  • 16.
    Binding • Process-launching agentworking with the OS to limit where each process can run: 1. No restrictions 2. Limited set of restrictions 3. Specific resource restrictions • “Binding width”  The number of PUs to which a process is bound
  • 17.
    Command Line Interface(CLI) • 4 levels of abstraction for the user  Level 1: None  Level 2: Simple, common patterns  Level 3: LAMA process layout regular patterns  Level 4: Irregular patterns
  • 18.
    CLI: Level 1(none) • No mapping or binding options specified  May or may not specify the number of processes to launch (-np)  If not specified, default to the number of cores available in the allocation  One process is mapped to each core in the system in a "by-core" style  Processes are not bound • …for backwards compatibility reasons 
  • 19.
    CLI: Level 2(common) • Simple, common patterns for mapping and binding  Specify mapping pattern with • --map-by X (e.g., --map-by socket)  Specify binding option with: • --bind-to Y (e.g., --bind-to core)  All of these options are translated to Level 3 options for processing by LAMA (full list of X / Y values shown later)
  • 20.
    CLI: Level 3(regular patterns) • LAMA process layout regular patterns  Power users wanting something unique for their application  Four MCA run-time parameters • rmaps_lama_map: Mapping process layout • rmaps_lama_bind: Binding width • rmaps_lama_order: Ordering of MCW ranks • rmaps_lama_mppr: Maximum allowable number of processes per resource (oversubscription)
  • 21.
    rmaps_lama_map (map) • Takesas an argument the "process layout"  A series of nine tokens • allowing 9! (362,880) mapping permutation options.  Preferred iteration order for LAMA • innermost iteration specified first • outermost iteration specified last
  • 22.
    Example system 2 servers(nodes), 4 sockets, 2 cores, 2 PUs
  • 23.
    rmaps_lama_map (map) • map=scbnh(a.k.a., by socket, then by core)
  • 24.
    rmaps_lama_map (map) • map=scbnh(a.k.a., by socket, then by core)
  • 25.
    rmaps_lama_map (map) • map=scbnh(a.k.a., by socket, then by core)
  • 26.
    rmaps_lama_map (map) • map=scbnh(a.k.a., by socket, then by core)
  • 27.
    rmaps_lama_map (map) • map=scbnh(a.k.a., by socket, then by core)
  • 28.
    rmaps_lama_bind (bind) • “Bindingwidth" and layer • Example: bind=3c (3 cores)Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI e PCI e PCI e PCI e bind = 3c
  • 29.
    Machine (128GB) NUMANode P#0(64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 rmaps_lama_bind (bind) • “Binding width" and layer • Example: bind=2s (2 sockets) Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23 PCI 8086:1521 eth0 PCI 8086:1521 eth1 PCI 8086:1521 eth2 PCI 8086:1521 eth3 PCI 1137:0043 eth4 PCI 1137:0043 eth5 PCI 102b:0522 bind = 2s bind = 2s
  • 30.
    rmaps_lama_bind (bind) • “Bindingwidth" and layer • Example: bind=12 (all PUs in an L2) bind = 12
  • 31.
    rmaps_lama_bind (bind) • “Bindingwidth" and layer • Example: bind=1N (all PUs in NUMA locality) bind = 1N
  • 32.
    rmaps_lama_order (order) • Selectwhich ranks are assigned to processes in MCW • There are other possible orderings, but no one has asked for them yet… Natural order for map-by-node (default) Sequential order for any mapping
  • 33.
    rmaps_lama_mppr (mppr) • mppr(mip-per) sets the Maximum number of allowable Processes Per Resource  User-specified definition of oversubscription • Comma-delimited list of <#:resource>  1:c  At most one process per core  1:c,2:s  At most one process per core, and at most two processes per socket
  • 34.
    MPPR  1:c At most one process per coreMachine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23
  • 35.
    MPPR  1:c,2:s At most one process per core and two processes per socket Machine (128GB) NUMANode P#0 (64GB) Socket P#0 L3 (20MB) L2 (256KB) L1d (32KB) L1i (32KB) Core P#0 PU P#0 PU P#16 L2 (256KB) L1d (32KB) L1i (32KB) Core P#1 PU P#1 PU P#17 L2 (256KB) L1d (32KB) L1i (32KB) Core P#2 PU P#2 PU P#18 L2 (256KB) L1d (32KB) L1i (32KB) Core P#3 PU P#3 PU P#19 L2 (256KB) L1d (32KB) L1i (32KB) Core P#4 PU P#4 PU P#20 L2 (256KB) L1d (32KB) L1i (32KB) Core P#5 PU P#5 PU P#21 L2 (256KB) L1d (32KB) L1i (32KB) Core P#6 PU P#6 PU P#22 L2 (256KB) L1d (32KB) L1i (32KB) Core P#7 PU P#7 PU P#23
  • 36.
    CLI: Level 4(rankfile) • Complete specification of processor-to- resource mapping description  Bypasses LAMA • Not described in the paper
  • 37.
    Level 2 toLevel 3 Chart
  • 38.
    Remember the priorexample? • -np 24 -mppr 2:c -map scbnh
  • 39.
    Same example, differentmapping • -np 24 -mppr 2:c -map nbsch
  • 40.
    • Displays prettyprintrepresentation of the binding actually used for each process.  Visual feedback = quite helpful when exploring mpirun -np 4 --mca rmaps lama --mca rmaps_lama_bind 1c --mca rmaps_lama_map nbsch --mca rmaps_lama_mppr 1:c --report- bindings hello_world MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..] MCW rank 1 bound to socket 1[core 8[hwt 0-1]]: [../../../../../../../..][BB/../../../../../../..] MCW rank 2 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../../../..][../../../../../../../..] MCW rank 3 bound to socket 1[core 9[hwt 0-1]]: [../../../../../../../..][../BB/../../../../../..] Report Bindings
  • 41.
    Future Work • Availablein Open MPI v1.7.2 (and later) • Open questions to users:  Are more flexible ordering options useful?  What common mapping patterns are useful?  What additional features would you like to see?
  • 42.

Editor's Notes

  • #15 Can be done either before or after binding – MPI implementation choice.
  • #23 --map-by socket would have put rank 4 on Node 1 (not what we wanted)
  • #24 --map-by socket would have put rank 4 on Node 1 (not what we wanted)
  • #25 --map-by socket would have put rank 4 on Node 1 (not what we wanted)
  • #26 --map-by socket would have put rank 4 on Node 1 (not what we wanted)
  • #27 --map-by socket would have put rank 4 on Node 1 (not what we wanted)
  • #28 --map-by socket would have put rank 4 on Node 1 (not what we wanted)
  • #34 Oversubscription is an error by defaultuser can specify –oversubscribe.
  • #35 Oversubscription is an error by defaultuser can specify –oversubscribe.
  • #36 Oversubscription is an error by defaultuser can specify –oversubscribe.