Colfax International

I'm happy to share a new coding tutorial for doing fast matrix multiplication on #NVIDIA Hopper GPUs! This covers the warpgroup matrix-multiply-accumulate (WGMMA) instruction that specifically targets the Tensor Cores on Hopper GPUs. Using tools from the CUTLASS library, we go into detail on all aspects of correctly invoking WGMMA as a primitive for matmul when writing a CUDA kernel -- how tensor data should be laid out in memory for WGMMA, how to use CUTLASS to define these layouts of your data, and how to synchronize WGMMA as an async instruction to guard against race conditions and ensure correct behavior of your kernel. If you've read the blog post on FlashAttention-3, you'll know how heavily FA-3 exploits WGMMA -- both in terms of its higher throughput and asynchronous capabilities -- to achieve its impressive performance gains. Our hope is that this tutorial can help similarly unlock the potential of the Hopper architecture when coding up your own projects and research ideas! We're also planning at least two followups to this tutorial - one covering the overall structure of an efficient GEMM kernel with a focus on copy-compute overlapping techniques such as warp specialization, and another on persistent kernels and the Stream-K algorithm for GEMM. Work done in collaboration with my colleagues at Colfax and Hieu Pham. https://lnkd.in/g-tsnnha

𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗶𝗻𝗴 𝗖𝗼𝗹𝗳𝗮𝘅 𝗔𝗰𝗰𝗲𝘀𝘀 𝗛𝘂𝗯: 𝗦𝗲𝗰𝘂𝗿𝗲𝗹𝘆 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗲 𝗬𝗼𝘂𝗿 𝗖𝘂𝘀𝘁𝗼𝗺 𝗖𝗼𝗻𝗳𝗶𝗴𝘂𝗿𝗲𝗱 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 𝗣𝗿𝗶𝗼𝗿 𝘁𝗼 𝗦𝗵𝗶𝗽𝗽𝗶𝗻𝗴 We are excited to offer a new way for our customers to seamlessly integrate new workstations, servers, and storage systems into their computing infrastructure. With the Colfax Access Hub, you can now 𝘀𝗲𝗰𝘂𝗿𝗲𝗹𝘆 validate and apply 𝗰𝘂𝘀𝘁𝗼𝗺 configurations to your systems, 𝗶𝗻𝘀𝘁𝗮𝗹𝗹 specialized operating systems, software, and licenses, 𝘁𝗲𝘀𝘁 high-speed networking that your computing system may include — all before they are boxed and shipped to: — Your or your clients’ datacenter — Satellite offices — Employees working from their home offices or employees working remotely 👉 𝘓𝘦𝘢𝘳𝘯 𝘮𝘰𝘳𝘦 https://lnkd.in/g7Je5Hzt 𝗪𝗵𝗮𝘁 𝗶𝘀 𝘁𝗵𝗲 𝗖𝗼𝗹𝗳𝗮𝘅 𝗔𝗰𝗰𝗲𝘀𝘀 𝗛𝘂𝗯? The Colfax Access Hub comes in two flavors: 𝗘𝘅𝗽𝗿𝗲𝘀𝘀 and 𝗦𝗰𝗮𝗹𝗲. 𝗖𝗼𝗹𝗳𝗮𝘅 𝗔𝗰𝗰𝗲𝘀𝘀 𝗛𝘂𝗯 𝗘𝘅𝗽𝗿𝗲𝘀𝘀: • Connect to your systems in minutes. • Direct access to your server’s OS. 𝗖𝗼𝗹𝗳𝗮𝘅 𝗔𝗰𝗰𝗲𝘀𝘀 𝗛𝘂𝗯 𝗦𝗰𝗮𝗹𝗲: • Connect to a Linux-based Landing Host for server management, OS installation, and remote desktop sessions. • Ideal for preparing systems for integration into remote datacenters where direct hardware interaction isn’t possible. 𝗪𝗵𝘆 𝗖𝗵𝗼𝗼𝘀𝗲 𝗖𝗼𝗹𝗳𝗮𝘅 𝗔𝗰𝗰𝗲𝘀𝘀 𝗛𝘂𝗯? Both Express and Scale options ensure: • Quick Connection: Access your systems in minutes. • Top-notch Security: Industry-standard authentication, encryption, and network isolation measures to protect your systems and data. Ready to interact with your new systems before they ship? Contact us at [email protected]. The Colfax Access Hub service is free for all Colfax customers. 👉 𝘓𝘦𝘢𝘳𝘯 𝘮𝘰𝘳𝘦 https://lnkd.in/g7Je5Hzt

𝗙𝗿𝗲𝗲 𝗡𝗩𝗜𝗗𝗜𝗔® 𝗛𝟭𝟬𝟬 𝗡𝗩𝗟 𝗧𝗲𝘀𝘁 𝗗𝗿𝗶𝘃𝗲 𝘚𝘶𝘱𝘦𝘳𝘤𝘩𝘢𝘳𝘨𝘦 𝘓𝘓𝘔 𝘐𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦 Colfax Experience Center is offering a FREE test drive program that provides you remote access to a Colfax server with four (4) NVIDIA® H100 NVL Tensor Core GPU cards (based on the NVIDIA Hopper™ architecture) for evaluation. 👉 𝘓𝘦𝘢𝘳𝘯 𝘮𝘰𝘳𝘦 𝘢𝘯𝘥 𝘴𝘪𝘨𝘯 𝘶𝘱 𝘧𝘰𝘳 𝘢 𝘵𝘦𝘴𝘵 𝘥𝘳𝘪𝘷𝘦 https://lnkd.in/dUAUSuV2 𝗔𝗯𝗼𝘂𝘁 𝗡𝗩𝗜𝗗𝗜𝗔® 𝗛𝟭𝟬𝟬 𝗡𝗩𝗟 𝗧𝗲𝗻𝘀𝗼𝗿 𝗖𝗼𝗿𝗲 𝗚𝗣𝗨 Based on the NVIDIA Hopper™ architecture, NVIDIA® H100 NVL Tensor Core GPU is the most optimized platform for LLM Inference with its high compute density, high memory bandwidth, high energy efficiency, and unique NVLink architecture. It also delivers unprecedented acceleration to power the world’s highest-performing elastic data centers for AI, data analytics, and high-performance computing (HPC) applications. NVIDIA H100 NVL Tensor Core technology supports a broad range of math precisions, providing a single accelerator for every compute workload. The NVIDIA H100 NVL supports double precision (FP64), single precision (FP32), half precision (FP16), 8-bit floating point (FP8), and integer (INT8) compute tasks. The NVIDIA H100 NVL card features Multi-Instance GPU (MIG) capability. This can be used to partition the GPU into as many as seven hardware isolated GPU instances, providing a unified platform that enables elastic data centers to adjust dynamically to shifting workload demands. As well as it can allocate the right size of resources from the smallest to biggest multi-GPU jobs. NVIDIA H100 NVL versatility means that IT managers can maximize the utility of every graphics processing unit (GPU) in their data center. #colfax #nvidia #Hopper #AI #HPC #llm #generativeAI

The new technique developed by researchers at NVIDIA, Meta, Georgia Institute of Technology, Princeton University, Together AI, and Colfax International can significantly reduce the time it takes to train LLMs and allow them to handle longer-token context windows. https://lnkd.in/ePD_XBHx #ai #enterpriseai #gpus #nvidia #compute #computerscience

Check out the FlashAttention-3 release! It's specifically crafted to take advantage of the asynchronous execution capabilities present in NVIDIA Hopper™ architecture, and achieves up to 740 TFLOPS for FP16 (75% utilization!) and 1.2 PFLOPS for FP8. This was a massive effort done in collaboration with my colleague Ganesh Bikshandi, Ying Zhang (PyTorch), Vijay Thakkar and Pradeep Ramani (NVIDIA), and of course Tri Dao (Together AI). Blogpost and paper: https://lnkd.in/gWZFbccZ Code on Tri's repo: https://lnkd.in/gvUFUQDv

FlashAttention-3 is out! Over the last 6 months, I have been having a blast collaborating with a team of brilliant researchers on it. It's designed from the ground up to take advantage of the asynchronous execution capabilities of the Hopper tensor core architecture, and achieves up to 740 TFLOPS for FP16 and 1.2 PFLOPS for FP8. This project was a collaboration across companies with my colleague Pradeep Ramani (NVIDIA), Jay Shah and Ganesh Bikshandi (Colfax International), Ying Zhang (PyTorch), and Tri Dao (Together AI) being involved. All of this work is open source, and is built on fully open source components from CUTLASS 3. When FA2 released, I quote tweeted Tri expressing how happy I was that the community was able to create new algorithms and realize ideas using the tools we had shipped in CUTLASS. A year later, the trend continues with even deeper codesign and direct collaboration :D arXiv preprint: https://lnkd.in/eC5RyKjj Code: https://lnkd.in/eA7xiTJ2 CUTLASS: https://lnkd.in/gQWYs7fu PyTorch blog: https://lnkd.in/eAU4srzm Happy hacking!

📢 Announcing the release of FlashAttention-3. ✅ 1.5-2x faster on FP16 ✅ Up to 740 TFLOPS on NVIDIA H100 GPUs ✅ FP8 close to 1.2 PFLOPS Read our latest blog to see how we collaborated with Colfax International, Meta, Together AI, and Princeton University on this achievement: https://nvda.ws/4bJDEdU

We are thrilled to release FlashAttention-3 in partnership with Meta, NVIDIA Princeton University, and Colfax International The improvements from FlashAttention-3 will result in: • More efficient GPU Utilization - up to 75% from 35% • Better performance with lower precision while maintaining accuracy • Ability to use longer context in LLMs more efficiently https://lnkd.in/gx48DBuB

Introducing FlashAttention-3 🚀 It’s optimized for H100, 1.5-2.0x faster than FlashAttention-2 with FP16, up to 740 TFLOPS, i.e., 75% utilization of H100 theoretical max FLOPS. With FP8, FlashAttention-3 reaches close to 1.2 PFLOPS, with 2.6x smaller error than baseline FP8 attention. Thank you to Colfax International, AI at Meta, NVIDIA AI and Together AI for the collaboration here 🙌 Read more in our blog: https://hubs.la/Q02Gfjx-0

Excited to share the following coding tutorial on the #NVIDIA Tensor Memory Accelerator! TMA is essential to extracting performance on NVIDIA Hopper™ GPUs, but it's not the easiest feature to learn how to program for. This tutorial aims to change that state of affairs and impart an operational understanding of TMA by walking through a few fully worked-out examples. We cover TMA load, store, store reduce, and load multicast. This is the fruit of a collaboration with Hieu Pham and is part of an ongoing series of CUDA® tutorials with an emphasis on the CUTLASS library. https://lnkd.in/g8MRp8a2