x86: Optimizing memcpy for AMD Zen architecture.

Message ID e86e2da7a7f9dce8885de3ae40c9e7e191f57fef.1603870335.git.sajan.karumanchi@amd.com
State New
Headers show
Series
  • x86: Optimizing memcpy for AMD Zen architecture.
Related show

Commit Message

Karumanchi, Sajan Oct. 28, 2020, 7:35 a.m.
From: Sajan Karumanchi <sajan.karumanchi@amd.com>


Modifying the shareable cache '__x86_shared_cache_size', which is a
factor in computing the non-temporal threshold parameter
'__x86_shared_non_temporal_threshold' to optimize memcpy for AMD Zen
architectures.
In the existing implementation, the shareable cache is computed as 'L3
per thread, L2 per core'. Recomputing this shareable cache as 'L3 per
CCX(Core-Complex)' has brought in performance gains.
As per the large bench variant results, this patch also addresses the
regression problem on AMD Zen architectures.

Reviewed-by: Premachandra Mallappa <premachandra.mallappa@amd.com>

---
 sysdeps/x86/cacheinfo.h | 32 ++++++++++++++++++++++++++------
 1 file changed, 26 insertions(+), 6 deletions(-)

-- 
2.17.1

Comments

H.J. Lu via Libc-alpha Oct. 28, 2020, 9:22 a.m. | #1
* sajan karumanchi:

> From: Sajan Karumanchi <sajan.karumanchi@amd.com>

>

> Modifying the shareable cache '__x86_shared_cache_size', which is a

> factor in computing the non-temporal threshold parameter

> '__x86_shared_non_temporal_threshold' to optimize memcpy for AMD Zen

> architectures.

> In the existing implementation, the shareable cache is computed as 'L3

> per thread, L2 per core'. Recomputing this shareable cache as 'L3 per

> CCX(Core-Complex)' has brought in performance gains.

> As per the large bench variant results, this patch also addresses the

> regression problem on AMD Zen architectures.

>

> Reviewed-by: Premachandra Mallappa <premachandra.mallappa@amd.com>


Thanks, I've pushed this version for you.

Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

Patch

diff --git a/sysdeps/x86/cacheinfo.h b/sysdeps/x86/cacheinfo.h
index 7f342fdc23..1296c93b2b 100644
--- a/sysdeps/x86/cacheinfo.h
+++ b/sysdeps/x86/cacheinfo.h
@@ -320,7 +320,7 @@  init_cacheinfo (void)
 	      threads = 1 << ((ecx >> 12) & 0x0f);
 	    }
 
-	  if (threads == 0)
+	  if (threads == 0 || cpu_features->basic.family >= 0x17)
 	    {
 	      /* If APIC ID width is not available, use logical
 		 processor count.  */
@@ -335,13 +335,30 @@  init_cacheinfo (void)
 	  if (threads > 0)
 	    shared /= threads;
 
-	  /* Account for exclusive L2 and L3 caches.  */
-	  shared += core;
-	}
+	  /* Get shared cache per ccx for Zen architectures.  */
+	  if (cpu_features->basic.family >= 0x17)
+	    {
+	      unsigned int eax;
+
+	      /* Get number of threads share the L3 cache in CCX.  */
+	      __cpuid_count (0x8000001D, 0x3, eax, ebx, ecx, edx);
+
+	      unsigned int threads_per_ccx = ((eax >> 14) & 0xfff) + 1;
+	      shared *= threads_per_ccx;
+	    }
+	  else
+	    {
+	      /* Account for exclusive L2 and L3 caches.  */
+	      shared += core;
+            }
+      }
     }
 
   if (cpu_features->data_cache_size != 0)
-    data = cpu_features->data_cache_size;
+    {
+      if (data == 0 || cpu_features->basic.kind != arch_kind_amd)
+        data = cpu_features->data_cache_size;
+    }
 
   if (data > 0)
     {
@@ -354,7 +371,10 @@  init_cacheinfo (void)
     }
 
   if (cpu_features->shared_cache_size != 0)
-    shared = cpu_features->shared_cache_size;
+    {
+      if (shared == 0 || cpu_features->basic.kind != arch_kind_amd)
+        shared = cpu_features->shared_cache_size;
+    }
 
   if (shared > 0)
     {