Workshop: Scaling CUDA C++ Applications to Multiple Nodes
Friday, September 13, 2024 -
9:00 AM
Monday, September 9, 2024
Tuesday, September 10, 2024
Wednesday, September 11, 2024
Thursday, September 12, 2024
Friday, September 13, 2024
9:00 AM
Introduction (- Meet the instructor. - Get familiar with your GPU-accelerated interactive JupyterLab environment.)
9:00 AM - 9:30 AM
9:30 AM
Multi-GPU Programming Paradigms: (- Survey multiple techniques for programming CUDA C++ applications for multiple GPUs using a Monte Carlo approximation of a CUDA C++ program: – Use CUDA to utilize multiple GPUs. – Learn how to enable and use direct peer-to-peer memory communication. – Write an SPMD version with CUDA-aware MPI.)
9:30 AM - 11:30 AM
11:30 AM
Lunch break
11:30 AM - 12:30 PM
12:30 PM
Introduction to NVSHMEM (Learn how to write code with NVSHMEM and understand its symmetric memory model: – Use NVSHMEM to write SPMD code for multiple GPUs. – Utilize symmetric memory to let all GPUs access data on other GPUs. – Make GPU-initiated memory transfers.)
12:30 PM - 2:30 PM
2:30 PM
Coffee break
2:30 PM - 2:45 PM
2:45 PM
Halo Exchanges with NVSHMEM: (Practice common coding motifs like halo exchanges and domain decomposition using NVSHMEM, and work on the assessment: – Write an NVSHMEM implementation of a Laplace equation Jacobi solver. – Refactor a single GPU 1D wave equation solver with NVSHMEM.)
2:45 PM - 4:30 PM
4:30 PM
Final Review (– Complete the assessment and earn a certificate. – Review key learnings and wrap up questions. – Learn about application tradeoffs on GPU clusters. – Take the workshop survey.) Conveners: Domen Verber, Jani Dugonik
4:30 PM - 5:00 PM