Context - 2 years Joint project (PTC: Programme Transverceaux de Compétences) - Two labs, both CEA: L_Sim in Grenoble (the BigDFT team), Maison de la Simulation in Saclay (the Kokkos side) - This talk = year 1: a Kokkos backend in BigDFT's Poisson solver

BigDFT - "Big :" (left): BigDFT because it is Big - 500k to 1M lines, about 20 developers, up to 100 users (last year's numbers) - module graph: many modules; psolver in gold, with the new Kokkos library shipped in psolver 1.10 - "DFT :" (right): a quantum mechanics code, Density Functional Theory - predicts the behaviour of atoms and molecules - runs on supercomputers, optimised on GPUs, in use since 2008 - example: antibody of 22 000 atoms, 1.2 h on 32 nodes (Irene Rome) - Prix Bull - Joseph Fourier

The psolver module and its backends - psolver = BigDFT's Poisson solver: potential V from a charge density rho on a 3D grid - Generalized: various boundary conditions (free, wire, surface, periodic); - Orchestrated in Fortran, distributed with MPI, optimised on GPUs - Table = the backends that existed before this work - SGFFT: built-in Fortran FFT, any CPU - ONEMATHCPU / ONEMATHGPU: SYCL, mostly used on AMD - DBFFT: SYCL for Intel GPUs - CUDA: pure CUDA, NVIDIA only - Kokkos: one implementation for any CPU and any GPU - One sentence: one backend per vendor means duplicated code and uneven performance, which Kokkos could avoid

- Reveal 1: - A door that already existed: psolver already had CUDA and SYCL backends - The hourglass: wide Fortran side, narrow C layer in the middle, wide backend side - Reveal 2: - Fortran 3D array -> C layer as a C-pointer -> device pointer stored as integer - C: one extern "C" function, ISO binding C types only - C++: the device view is own by C++; the Fortran array is wrapped in an unmanaged host View, no copy no ref counting - Memory by hand: RAII cannot cross the language boundary, the owner View lives on the heap (kokkos_malloc / kokkos_free) - Reveal 3: - the data cross back once computation done - buffers are flat 1D; layout matters later, in the kernels - two worlds separate not constraint here bu solution

- Reveal 1: - A door that already existed: psolver already had CUDA and SYCL backends - The hourglass: wide Fortran side, narrow C layer in the middle, wide backend side - Reveal 2: - Fortran 3D array -> C layer as a C-pointer -> device pointer stored as integer - C: one extern "C" function, ISO binding C types only - C++: the device view is own by C++; the Fortran array is wrapped in an unmanaged host View, no copy no ref counting - Memory by hand: RAII cannot cross the language boundary, the owner View lives on the heap (kokkos_malloc / kokkos_free) - Reveal 3: - the data cross back once computation done - buffers are flat 1D; layout matters later, in the kernels - two worlds separate not constraint here bu solution

- Reveal 1: - A door that already existed: psolver already had CUDA and SYCL backends - The hourglass: wide Fortran side, narrow C layer in the middle, wide backend side - Reveal 2: - Fortran 3D array -> C layer as a C-pointer -> device pointer stored as integer - C: one extern "C" function, ISO binding C types only - C++: the device view is own by C++; the Fortran array is wrapped in an unmanaged host View, no copy no ref counting - Memory by hand: RAII cannot cross the language boundary, the owner View lives on the heap (kokkos_malloc / kokkos_free) - Reveal 3: - the data cross back once computation done - buffers are flat 1D; layout matters later, in the kernels - two worlds separate not constraint here bu solution

- Reveal 1: - A door that already existed: psolver already had CUDA and SYCL backends - The hourglass: wide Fortran side, narrow C layer in the middle, wide backend side - Reveal 2: - Fortran 3D array -> C layer as a C-pointer -> device pointer stored as integer - C: one extern "C" function, ISO binding C types only - C++: the device view is own by C++; the Fortran array is wrapped in an unmanaged host View, no copy no ref counting - Memory by hand: RAII cannot cross the language boundary, the owner View lives on the heap (kokkos_malloc / kokkos_free) - Reveal 3: - the data cross back once computation done - buffers are flat 1D; layout matters later, in the kernels - two worlds separate not constraint here bu solution

- Reveal 1: two separate worlds, CMake and Autotools - Kokkos : CMake (C++20, nvcc_wrapper / hipcc / icpx) - psolver : autotools - Reveal 2: - autotools : prod PSolver, need to link against the Kokkos code - CMake : prod C++ ext. lib psolverkokkos - pkg-config handshake - Reveal 3: the rest - pImpl header: psolver includes it without any Kokkos header or flag - library not found: stubs, the build still works

- Reveal 1: two separate worlds, CMake and Autotools - Kokkos : CMake (C++20, nvcc_wrapper / hipcc / icpx) - psolver : autotools - Reveal 2: - autotools : prod PSolver, need to link against the Kokkos code - CMake : prod C++ ext. lib psolverkokkos - pkg-config handshake - Reveal 3: the rest - pImpl header: psolver includes it without any Kokkos header or flag - library not found: stubs, the build still works

- Reveal 1: two separate worlds, CMake and Autotools - Kokkos : CMake (C++20, nvcc_wrapper / hipcc / icpx) - psolver : autotools - Reveal 2: - autotools : prod PSolver, need to link against the Kokkos code - CMake : prod C++ ext. lib psolverkokkos - pkg-config handshake - Reveal 3: the rest - pImpl header: psolver includes it without any Kokkos header or flag - library not found: stubs, the build still works

1.3 Validation - Message: tested AMAP - Library level: own test suites, separate from psolver - 24 tests through the public C interface (CTest): memory, transfers, pad/unpad, pre/post, accumulation, reductions, edge cases - 8 internal kernel tests (transposes, spread, fused vs separate, multiply), run by hand - psolver level: PS_Program - builds rho from solution V_exact and compare - every boundary condition, several grids, 1 to 8 MPI ranks, CPU and GPU - Application level: Fock computes Exact Exchange energy - one Poisson solve per orbital pair, energies summed - Fock is the exec for perf study - psolverkokkos integrated last August in BigDFT

- for every paig i,j, rho_ij = phi_i * phi_j and one Poisson solve - N(N+1)/2 solves - Three kernels = the three groups of the loop - 1: build rho_ij (pre_computation) - 2: Poisson solve (next slide) - 3: energy and gradient (reduction, accumulate, post_computation) - Order on a GPU: pre, pad, FFT chain, unpad, reduction, accumulate, post (twice)

- Poisson solution = convolution Green's func: go to Fourier, multiply, come back - A 3D FFT = three batches of 1D FFTs, one per axis - axis needs transpose OR spread ? do - FFT along axis - multiply Fourier space - come back - explain transpose & spread - Teal = the FFT, vendor library through KokkosFFT (cuFFT, rocFFT, a host FFT on CPU): the same in every backend - Gold = our Kokkos kernels - five kernels only move data; the only computation outside of FFT is multiplyKernel If asked: - zeros are drawn after the data; in the code they go first, so the result lands in the first half - the inverse transpose is the same kernel with the sizes swapped

- Software: Kokkos 5.2.2, CUDA 13.0.2, ROCm 6.4.3 - Two clusters, four platforms - Ruche: Xeon CPU + A100 GPU - Adastra: EPYC CPU + MI250X GPU (one GCD = half a card) - Four systems from BigDFT's continuous benchmarking: - H20 & UO2 periodic = no spread - Graphene & Aflatoxin non periodic = spread - Exclusive nodes; 5 repetitions on GPUs (A100: first one dropped as warm-up, say it orally), 3 on CPUs - Metric: Fock wall time ("Walltime since initialization") - on GPUs also profiled (nsys, rocprofv3) to get the GPU kernel time - CPU runs: whole node (4 ranks x 10 threads on Ruche, 12 x 16 on Adastra) - tables nice info sharing but visualisation

- Message: what we compare against decides what we can conclude - we have 5 VS - Each VS = one hardware; two opponents face each other, their FFT library below - gold = Kokkos, blue = existing backend, teal = FFT library - CPUs (top) - KOKKOSCPU vs SGFFT, on Xeon and on EPYC - KOKKOSCPU vs ONEMATHCPU, EPYC only (no SYCL CPU runtime on the Xeon machine) - GPUs (bottom) - A100: KOKKOSGPU<CUDA> vs CUDA, both call cuFFT - MI250X : KOKKOSGPU<HIP> vs ONEMATHGPU; hipFFT runs rocFFT's own kernels, so the FFT work is the same - The opponents differ in more than the programming model: the FFT library but also the implementation (ONEMATH =! CUDA =!!! SGFFT) - Dashed circles: x3 repetitions on CPUs, x5 on GPUs - "Best existing backend we measured on that platform" is the wording for the baselines If asked: - why no ONEMATH on Ruche: deployment gap (incomplete oneAPI Unified Runtime), not a limit of the code; never run on the A100 - ONEMATHGPU is our name for clarity (the accel string in the code is ONEMATH) - ONEMATHCPU = SYCL + oneMKL on Intel's OpenCL CPU runtime (TBB, not OpenMP); KOKKOSCPU = OpenMP with threaded FFTW; SGFFT = Fortran FFT with OpenMP

- Keep it short: - Lack of time - failed to reproduce known perf on Adastra on Aflatoxin - blue line is baseline, left psolverkokkos goes faster, right slower - Against SGFFT: competitive, no clear winner, expected. - Against ONEMATHCPU (EPYC): psolverkokkos faster everywhere, up to 3x on Aflatoxin - Honest limitation: the AMD CPU baseline - I could not reproduce the SGFFT performance published for Adastra from the documented build and launch (published ~2x faster on Aflatoxin) - so the EPYC comparison is indicative; it would lower the PP number If asked: - binding made ONEMATHCPU 10-20% faster, barely moved Kokkos or SGFFT; ranking depends on the launch configuration - cause of the 2x not tested (-O3 vs the group's -O2, thread placement, NUMA all plausible); test = rebuild with the group's flags - Xeon: no published SGFFT run to compare with

- Same reading as the CPU figure: wall time against the baseline in %, negative = psolverkokkos faster - What stands out : - Aflatoxin outlier - psolverkokkos significantly slower - deepdive into why later - MI250X against ONEMATHGPU: psolverkokkos faster on three systems - -6.8% (H2O-32), -13.6% (Graphene-H), -21.6% (UO2_2) - A100 against CUDA: - within a few % of hand-written CUDA on three systems If asked: - spread across repetitions is tiny (dots sit on the bars) - GPU time is only used for the profile analyses of 2.5 and 2.6

GPU results: One full Fock iteration (Graphene-H) - What the GPU does during one Poisson solve: one rectangle = one kernel, real start and real duration - Graphene-H because it has both periodic and non-periodic axes - Sequence: pre, memset + pad, three FFT stages with transposes, multiply, unpad, reduction, copy, accumulate, two post - "fused" = fused transposeSpread, on the non-periodic axis - Good things : - The FFT kernels are identical in every lane: KokkosFFT adds nothing - psolverkokkos GPU kernels/regions are well stacked - our psolverkokkos implementation ~maxed optim - Bad things : - ONEMATH idles ~10 µs between kernels: 239 vs 73 - fused transposeSpread not optimized - GOOD psolverkokkos reduction shorter; BAD psolverkokkos GPU idle longer (77 vs 22): fence + blocking scalar copy - a small GAP per iteration -> delay overall (next slide) - vendor FFT = 29-43% of the kernel time, same for both backends of a pair - Say "on the ONEMATHGPU build we used": a measurement of that stack, not a verdict on SYCL If asked: - cause of the ~10 microsecond gaps not established: needs a HIP API trace and an in-order SYCL queue microbenchmark - idle on 2.5 footnotes is the mean over the run (~270), here the typical solve (239): ~9% of ONEMATH solves have a long stall - H2O-32, UO2_2 and Aflatoxin versions are the extra slides after the end

- Reading: each bar = extra time of KOKKOSGPU over ONEMATHGPU for one kernel family, summed over the run - focus on how much of the walltime gap comes from the iteration ? - H2O-32: all kernels on par; the gain is ONEMATH idling between its kernels (about 10 microseconds each), slightly offset by a slower setup on the psolverkokkos side - UO2_2, Graphene-H: kernels on par, ONEMATH idles less than on H2O-32 but still more than psolverkokkos; most of the gain is the setup, ~1.1 s faster on the psolverkokkos side - Aflatoxin: the slow fused and spread kernels give +3990 ms of GPU work, and the setup is +3.5 s slower with psolverkokkos

- Left, H2O-32 and UO2_2: small wall gaps (+4.5%, +1.9%), entirely GPU idle time in the loop - psolverkokkos GPU work is actually 6% and 7% lower - cuFFT, pre/post, accumulate identical - psolverkokkos faster on reduction (one kernel, CUDA two), pad/unpad (copy kernel vs cudaMemcpy3D) and multiply - psolverkokkos slower on the plain transpose of the periodic systems (+7-10%), reason not investigated - the GPU idle cost : ~35ms/solve is the real slow down - Right, Graphene-H and Aflatoxin: - Aflatoxin +27.4%: unoptimised fused kernels = 78% of the GPU gap, real-data spread = 33% - GPU idle in the loop is only 2% of the Aflatoxin wall gap - the setup bar is small here: +1, +50, +99, +52 ms, against seconds on the MI250X; still about half of the small wall gaps of UO2_2 and Graphene-H - it is not the FFT plans: the plan and kernel creation timer is the same for CUDA and psolverkokkos (within 0.03 s); what the rest of the 50-100 ms is (before or after the loop) is not identified If asked: - fence + copy; test = keep the sum on the device, drop the fence - psolverkokkos beats CUDA on the reduction because of how the two are written (the CUDA one launches only 64 blocks and a second kernel), not because of better generated code - multiply and plain-transpose differences: cause not investigated, claim none

- Effective bandwidth of each data-movement kernel, as a share of the measured copy ceiling (1387 GB/s) - effective = bytes the algorithm must move / kernel time: a lower bound, same work in both backends - Kernels written the same way sit at the ceiling in both backends - multiply 91-98%, tiled plain transpose ~91% - pad / unpad: psolverkokkos 89% and 91% vs CUDA 82% and 58% (a copy kernel vs cudaMemcpy3D): psolverkokkos is the faster - bandwidth-limited: nothing left to tune, only moving fewer bytes - The fused kernels are not at the ceiling: 36-48% (psolverkokkos) vs 81-90% (CUDA) - Real-data spread (Aflatoxin): 21% and 26% vs 93% - Far below the ceiling = slow for another reason, not memory bound yet: no shared-memory tiling (inferred) - that is the gap of 2.5; fixing it would bring these kernels to the 85-90% of the CUDA ones (expected, not measured) - At the ceiling, speed only comes from fewer passes over memory: FFT callbacks (outlook, by analogy) If asked: - only Graphene-H and Aflatoxin drawn: H2O-32 and UO2_2 arrays fit in the 42 MB L2 cache, so "bandwidth" would be cache - MI250X not drawn: ONEMATHGPU itself only reaches 50-55% on its plain transposes, a weaker argument - only the non-periodic axes use the fused and spread kernels: why H2O-32 and UO2_2 are at parity and Graphene-H in between - cause is inferred from bandwidth and source, not from a hardware counter

2.7 Performance portability (Pennycook, application efficiency) - Metric: PP = harmonic mean over the platforms of e = t(best existing backend) / t(Kokkos), capped at 1; 0 if it does not run - the harmonic mean is dominated by the worst platform: that is the point of the metric - Baselines = best existing backend we measured: A100 CUDA; MI250X ONEMATHGPU (the only one there); Xeon SGFFT; EPYC SGFFT (faster than ONEMATHCPU on all four systems) - GPUs: PP 0.92 over the eight pairs - per system 0.98 / 0.77 / 0.98 / 0.99 (H2O-32, Aflatoxin, Graphene-H, UO2_2) - clean baselines - the one deficit is Aflatoxin: the fused and spread kernels of 2.5 - MI250X efficiencies of 1.00 = ONEMATH idling in the loop and its slower FFT plan setup, not faster kernels - CPUs: PP 0.96 over the eight pairs (1.00 / 0.93 / 0.91 / 1.00) - the soft number: the EPYC SGFFT baseline is our own build - say orally: flattered by our SGFFT, the real value is probably a little lower - All four platforms together: 0.94 If asked: - with the published SGFFT: EPYC Aflatoxin efficiency 0.52 instead of 1.00, Aflatoxin PP over four platforms 0.71 instead of 0.84 - no published reference for the Xeon or the other systems: unknown, not equal

- It works: a Kokkos library behind the same hourglass interface as the other backends - built separately from psolver (CMake, pImpl + pkg-config handshake with autotools) - validated at several levels, all backends give the same energies - WITH Kokkos debugging and profiling tools - It performs: PP 0.92 on the two GPUs, 0.96 on the two CPUs - CPU number is the soft one (our own SGFFT baseline) - GPU PP of at least 0.98 on three of four systems: a statement about wall time, not about the kernels (several are not on par) - The gap: Aflatoxin, PP 0.77 - where it lies: fused transposeSpread and spread kernels, far below the memory ceiling - plus GPU idle time from synchronisation (fence and blocking copy on the A100) - say "where the gap lies", never "tiling accounts for x%"; the effect of a fix is expected, not measured - Claim to land, said aloud, not on the slide: for all we can tell, Kokkos itself adds no measurable overhead; if any, it is crumbs, far behind the optimisations left in our own kernels. Writting your code in Kokkos is not more painful than in SYCL or CUDA, if ever it is less painful, Kokkos gives you the tools and the means to fully optimize your code and use the full power of the device.

- It works: a Kokkos library behind the same hourglass interface as the other backends - built separately from psolver (CMake, pImpl + pkg-config handshake with autotools) - validated at several levels, all backends give the same energies - WITH Kokkos debugging and profiling tools - It performs: PP 0.92 on the two GPUs, 0.96 on the two CPUs - CPU number is the soft one (our own SGFFT baseline) - GPU PP of at least 0.98 on three of four systems: a statement about wall time, not about the kernels (several are not on par) - The gap: Aflatoxin, PP 0.77 - where it lies: fused transposeSpread and spread kernels, far below the memory ceiling - plus GPU idle time from synchronisation (fence and blocking copy on the A100) - say "where the gap lies", never "tiling accounts for x%"; the effect of a fix is expected, not measured - Claim to land, said aloud, not on the slide: for all we can tell, Kokkos itself adds no measurable overhead; if any, it is crumbs, far behind the optimisations left in our own kernels. Writting your code in Kokkos is not more painful than in SYCL or CUDA, if ever it is less painful, Kokkos gives you the tools and the means to fully optimize your code and use the full power of the device.

- It works: a Kokkos library behind the same hourglass interface as the other backends - built separately from psolver (CMake, pImpl + pkg-config handshake with autotools) - validated at several levels, all backends give the same energies - WITH Kokkos debugging and profiling tools - It performs: PP 0.92 on the two GPUs, 0.96 on the two CPUs - CPU number is the soft one (our own SGFFT baseline) - GPU PP of at least 0.98 on three of four systems: a statement about wall time, not about the kernels (several are not on par) - The gap: Aflatoxin, PP 0.77 - where it lies: fused transposeSpread and spread kernels, far below the memory ceiling - plus GPU idle time from synchronisation (fence and blocking copy on the A100) - say "where the gap lies", never "tiling accounts for x%"; the effect of a fix is expected, not measured - Claim to land, said aloud, not on the slide: for all we can tell, Kokkos itself adds no measurable overhead; if any, it is crumbs, far behind the optimisations left in our own kernels. Writting your code in Kokkos is not more painful than in SYCL or CUDA, if ever it is less painful, Kokkos gives you the tools and the means to fully optimize your code and use the full power of the device.

- Close the gap: optimise the transposeSpread and spread kernels, and cut the GPU idle time (fence and blocking copy after the reduction) - shared-memory tiling already closed the gap on the older unfused transpose and spread kernels (merged in BigDFT devel) - expected to close the Aflatoxin and Graphene-H gaps by analogy, not measured on the fused kernel - Go beyond: FFT callbacks (KokkosFFT), my own feature, in progress - do the spread inside the FFT: those kernels would not be needed - candidate, by analogy with FFT libraries, never measured here, no prototype - "fewer passes over memory" wording only if asked - Clean up: split libpsolverkokkos into psolverkokkos (computation) + futilekokkos (memory management) - every View gets the same label: the shared Fortran/C interface cannot carry a label string - errors cannot be thrown through the interface, only a status code on malloc and memset - Not in the talk: the subject of year 2, the earlier optimisation campaign, any prototype of the fused-kernel fix

- Close the gap: optimise the transposeSpread and spread kernels, and cut the GPU idle time (fence and blocking copy after the reduction) - shared-memory tiling already closed the gap on the older unfused transpose and spread kernels (merged in BigDFT devel) - expected to close the Aflatoxin and Graphene-H gaps by analogy, not measured on the fused kernel - Go beyond: FFT callbacks (KokkosFFT), my own feature, in progress - do the spread inside the FFT: those kernels would not be needed - candidate, by analogy with FFT libraries, never measured here, no prototype - "fewer passes over memory" wording only if asked - Clean up: split libpsolverkokkos into psolverkokkos (computation) + futilekokkos (memory management) - every View gets the same label: the shared Fortran/C interface cannot carry a label string - errors cannot be thrown through the interface, only a status code on malloc and memset - Not in the talk: the subject of year 2, the earlier optimisation campaign, any prototype of the fused-kernel fix

- Close the gap: optimise the transposeSpread and spread kernels, and cut the GPU idle time (fence and blocking copy after the reduction) - shared-memory tiling already closed the gap on the older unfused transpose and spread kernels (merged in BigDFT devel) - expected to close the Aflatoxin and Graphene-H gaps by analogy, not measured on the fused kernel - Go beyond: FFT callbacks (KokkosFFT), my own feature, in progress - do the spread inside the FFT: those kernels would not be needed - candidate, by analogy with FFT libraries, never measured here, no prototype - "fewer passes over memory" wording only if asked - Clean up: split libpsolverkokkos into psolverkokkos (computation) + futilekokkos (memory management) - every View gets the same label: the shared Fortran/C interface cannot carry a label string - errors cannot be thrown through the interface, only a status code on malloc and memset - Not in the talk: the subject of year 2, the earlier optimisation campaign, any prototype of the fused-kernel fix

Thanks - Link 1: this Kokkos Tea Time page: the slides will be downloadable there soon - Link 2: the repository - build the deck (make pdf in slides/) - tidy benchmark and profiling data behind every figure, plus the analysis scripts - raw cluster dumps are not in it - Papers: the group's psolver papers, from free boundary conditions (2006) to hybrid functionals on GPU supercomputers (2018) - Questions If asked: - a fresh clone builds the deck, it needs network access (logos, Marp through npx)

Extra, H2O-32 (only if asked; periodic: no spread, no fused kernels) - Same figure as 2.4, typical (median) solve - MI250X: ONEMATHGPU 608 vs KOKKOSGPU 451 microseconds per solve - GPU busy for the same time (379 vs 376): the difference is the holes - ONEMATH ~10 microseconds between almost every pair of kernels: 230 idle (38%) - psolverkokkos back to back, idles only at the end of the solve: 75 (17%) - A100: CUDA 266 vs psolverkokkos 303 microseconds - CUDA idles 24; psolverkokkos 76 (fence + blocking 8-byte copy) - psolverkokkos is busy for less time (227 vs 243): a faster reduction (the CUDA one launches only 64 blocks and a second kernel) - FFT: 91 vs 92 microseconds (A100, cuFFT), 126 vs 124 (MI250X, rocFFT)

Extra, UO2_2 (only if asked; periodic, like H2O-32) - Same figure as 2.4, typical (median) solve - MI250X: ONEMATHGPU 751 vs KOKKOSGPU 615 microseconds - GPU busy 518 vs 539 - idle 233 (31%) vs 76 (12%) - A100: CUDA 321 vs psolverkokkos 345 microseconds - idle 23 vs 75 (fence + scalar copy) - psolverkokkos busy for less time (271 vs 298): shorter reduction and multiply - FFT: 78 vs 77 microseconds (A100), 151 vs 157 (MI250X)

Extra, Aflatoxin (only if asked; free in all directions: spread and fused kernels) - Same figure as 2.4, typical (median) solve; milliseconds long, so idle gaps are small in proportion - A100: CUDA 5137 vs psolverkokkos 6966 microseconds (+36%) - fused 2619 vs 1223 (+1396), spread 774 vs 188 (+586) - psolverkokkos reduction shorter (121 vs 226); FFT (2098 vs 2103) and multiply (709 vs 727) identical - idle 81 vs 24 - MI250X: ONEMATHGPU 7802 vs KOKKOSGPU 9941 microseconds (+27%) - fused 3943 vs 2442 (+1501), spread 1100 vs 222 (+878) - FFT and multiply identical - ONEMATH idles 276, psolverkokkos 78: a small gain against the extra kernel time - The gap of 2.5 and 2.6: fused and spread kernels, no shared-memory tiling yet (inferred)

Extra, only if asked: the setup difference behind the "setup" bar of 2.5 on the MI250X - What is measured: the Fock timer "PSolver Kernel Creation" = everything before the first Poisson solve (CPU Green function, start of the backend, FFT plans, copy of the kernel to the GPU) - MI250X: 2 to 12 s, and it differs between the backends: psolverkokkos is slower on H2O-32 (+0.5 s) and Aflatoxin (+3.5 s), faster on UO2_2 and Graphene-H (-1.1 s) - A100: 0.3 to 0.8 s, same for CUDA and psolverkokkos - Diamonds: hipFFT plan creation (6 plans) measured by Kokkos-Tools in the profiled run: 4.7, 1.4, 1.3, 10.3 s = most of the psolverkokkos setup - Suspect: rocFFT compiles its FFT kernels (the fft_rtc_* kernels) when a plan is created; cuFFT does not - GPU traces: pauses of about 1 s before the first solve, one per kernel built - the same 6 kernels (3 lengths, forward and backward) in both backends - Aflatoxin: 7 pauses with ONEMATH, 13 with psolverkokkos - Why psolverkokkos pays more, likely not proven (it is how our code creates its plans, not Kokkos itself): - ONEMATH: 3 oneMath descriptors, each used in both directions - psolverkokkos: 6 KokkosFFT plans (a plan has one direction), through hipFFT, which probably builds both directions for each complex plan - KokkosFFT also has a direct rocFFT backend (one plan = one direction), not tested - Once per run, not per solve: it does not scale with the number of Poisson solves If asked: - not tested: whether the rocFFT kernel cache persists between runs (the setup is steady over the 5 repetitions, so probably not) - profiled run and benchmark runs are different runs: the diamonds are an order of magnitude, not an exact part of the bars