Intel SNB/IVB/HSW/BDW : IMC extended timings quest

[cyring]

Reverse protocol

Specification

Missing specs from 0x4008 to 0x4024

Dump

Channel[0]
0x4000 [001b899a]
0x4004 [90248588]
0x4008 [0a022220]
0x400c [000058d7]
0x4010 [00000000]
0x4014 [00000000]
0x4018 [00000000]
0x401c [00100000]
0x4020 [00100005]
0x4024 [27272020]
0x4028 [000e2200]
0x402c [00000000]
Channel[1]
0x4000 [001b899a]
0x4004 [90248588]
0x4008 [0a022220]
0x400c [000058d7]
0x4010 [00000000]
0x4014 [00000000]
0x4018 [00000000]
0x401c [00100000]
0x4020 [00100005]
0x4024 [28282020]
0x4028 [000e3300]
0x402c [00000000]

BIOS

Source

void Query_SNB_IMC(void __iomem *mchmap, unsigned short mc)
{
           /* ... */
    for (cha = 0; cha < PUBLIC(RO(Proc))->Uncore.MC[mc].ChannelCount; cha++)
    {
        unsigned int    R4000=0, R4004=0, R4008=0, R400c=0, R4010=0, R4014=0,
                        R4018=0, R401c=0, R4020=0, R4024=0, R4028=0, R402c=0;

                R4000 = readl(mchmap + 0x4000 + 0x400 * cha);
                R4004 = readl(mchmap + 0x4004 + 0x400 * cha);
                R4008 = readl(mchmap + 0x4008 + 0x400 * cha);
                R400c = readl(mchmap + 0x400c + 0x400 * cha);
                R4010 = readl(mchmap + 0x4010 + 0x400 * cha);
                R4014 = readl(mchmap + 0x4014 + 0x400 * cha);
                R4018 = readl(mchmap + 0x4018 + 0x400 * cha);
                R401c = readl(mchmap + 0x401c + 0x400 * cha);
                R4020 = readl(mchmap + 0x4020 + 0x400 * cha);
                R4024 = readl(mchmap + 0x4024 + 0x400 * cha);
                R4028 = readl(mchmap + 0x4028 + 0x400 * cha);
                R402c = readl(mchmap + 0x402c + 0x400 * cha);

                printk( "Channel[%d]\n"         \
                        "0x4000 [%08x]\n"       \
                        "0x4004 [%08x]\n"       \
                        "0x4008 [%08x]\n"       \
                        "0x400c [%08x]\n"       \
                        "0x4010 [%08x]\n"       \
                        "0x4014 [%08x]\n"       \
                        "0x4018 [%08x]\n"       \
                        "0x401c [%08x]\n"       \
                        "0x4020 [%08x]\n"       \
                        "0x4024 [%08x]\n"       \
                        "0x4028 [%08x]\n"       \
                        "0x402c [%08x]\n",
                        cha,
                        R4000,
                        R4004,
                        R4008,
                        R400c,
                        R4010,
                        R4014,
                        R4018,
                        R401c,
                        R4020,
                        R4024,
                        R4028,
                        R402c);
    }
           /* ... */
}

[cyring]

Guess round #1

BIOS (Third Timings)

Dump

                              Channel[0]
                              0x4000 [001b899a]
                              0x4004 [90248588]
                              0x4008 [cc025761]
                              0x400c [000058d7]
                              0x4010 [00000000]
                              0x4014 [00000000]
                              0x4018 [00000000]
                              0x401c [00100000]
                              0x4020 [00100005]
                              0x4024 [27272020]
                              0x4028 [000e2200]
                              0x402c [00000000]
                              Channel[1]
                              0x4000 [001b899a]
                              0x4004 [90248588]
                              0x4008 [cc025761]
                              0x400c [000058d7]
                              0x4010 [00000000]
                              0x4014 [00000000]
                              0x4018 [00000000]
                              0x401c [00100000]
                              0x4020 [00100005]
                              0x4024 [28282020]
                              0x4028 [000e3300]
                              0x402c [00000000]

CoreFreq

[cyring]

Guess round #2

BIOS tRRSR

Dump

                              Channel[0]
                              0x4000 [001b899a]
                              0x4004 [90248588]
                              0x4008 [1a022220]
                              0x400c [000058d7]
                              0x4010 [00000000]
                              0x4014 [00000000]
                              0x4018 [00000000]
                              0x401c [00100000]
                              0x4020 [00100005]
                              0x4024 [27272020]
                              0x4028 [000e2200]
                              0x402c [00000000]
                              Channel[1]
                              0x4000 [001b899a]
                              0x4004 [90248588]
                              0x4008 [1a022220]
                              0x400c [000058d7]
                              0x4010 [00000000]
                              0x4014 [00000000]
                              0x4018 [00000000]
                              0x401c [00100000]
                              0x4020 [00100005]
                              0x4024 [28282020]
                              0x4028 [000e3300]
                              0x402c [00000000]

CoreFreq

[cyring]

Guess round #3

BIOS

Dump

                              Channel[0]
                              0x4000 [001b899a]
                              0x4004 [90248588]
                              0x4008 [0a022220]
                              0x400c [000058d7]
                              0x4010 [00000000]
                              0x4014 [00000000]
                              0x4018 [00000000]
                              0x401c [00100000]
                              0x4020 [00100005]
                              0x4024 [27272020]
                              0x4028 [000e2200]
                              0x402c [00000000]
                              Channel[1]
                              0x4000 [001b899a]
                              0x4004 [90248588]
                              0x4008 [0a022220]
                              0x400c [000058d7]
                              0x4010 [00000000]
                              0x4014 [00000000]
                              0x4018 [00000000]
                              0x401c [00100000]
                              0x4020 [00100005]
                              0x4024 [28282020]
                              0x4028 [000e3300]
                              0x402c [00000000]

[svmlegacy]

@cyring
Here's a new gist with the outputs from my Sandy Bridge Celeron G550.

Here.

(This is a response to this note) This is the same system, but I've been slowing working up the CPU performance...

[cyring]

Your outputs are really helpful. Thank you.

So what we have more now is B2B and most third timings.
To compare with, does this platform has a BIOS showing them ?

[cyring]

Need someone to confirm with screenshot the IMC result on Haswell/Broadwell.
Especially platform wirh host bridge of DID 0x0c00 or 0x0c04
Please pull latest commits from the develop branch; rebuild, reload CoreFreq and post results here.

Focus on MC timings and DIMM geometry.
Recently SNB, IVB and Atom are progressively being added. Feel free to show me your results.

EDIT: I also need those Xeon's architectures.
Please give a try and show me. But not on Production machine or during down time you can boot my all-in-one developer ISO with hybrid mbr/uefi ArchLinux and CoreFreq embedded.
Disks are not mounted; ramfs only.
Go to www.cyring.fr to download ISO.

[svmlegacy]

Haswell-DT Intel Core i7-4570 SR14E
Broadwell-H Intel Xeon E3-1270L v4 SR2AZ

Bios pictures will go into the comments of each respectively. (Along with the G550)

[cyring]

Awesome!

I will come back with an attempt to decode the Broadwell Xeon IMC.

And minor fixes among both processors.

[svmlegacy]

Sandy Bridge-EP Intel Xeon E5-2640 SR0KR
Ivy Bridge-EP Intel Xeon E5-1620 v2 SR1AR

[cyring]

@svmlegacy Hello

For IMC testings is available the commit 347dd79 for the Broadwell E3-1270L and the Xeon E5-2640 processors.

[cyring]

Thank you for your tests.

• DIMM count is improved but channels count is still an issue.
  Do you confirm the platform is quad channel and only one DIMM is populated every two slots ?

• Yes I would like a test with your Haswell-E
  Not sure where it will fall through. Here all the Haswell I have set in table:
  

  CoreFreq/corefreqk.h

  Line 1771 in cd4cdfb

  /* [Haswell] 06_3Ch, 06_3Fh, 06_45h, 06_46h */

  
  The IMC decoding is following the PCI ID. So result could be the same as EP
  If no IMC is showing up then I miss those IDs. lspci -nn of the Haswell-E will then be useful to me.

• Dual sockets testings: I will appreciate the complete overview from the CLI; especially the Core topology.
  Core & socket Temperature, Core voltage, Power consumed are also of a great interest.
  I mean If CoreFreq is able to monitor discrete values among both sockets ?
  Suggest to change their scope in the Settings menu if they don't show up in the UI.
  If possible w/o a cooler, you will have to stress socket differently to catch any sensor difference among both.

[cyring]

@svmlegacy
There's a new commit for QPI frequency of Xeon
BUS Rate of IMC should display 7200

Another thought is that you have a single processor socket populated, perhaps quad channels timings are set for its connected DIMMs only ?

[svmlegacy]

Sandy-E and Ivy-E are quad channel CPU's, so in 2P operation I'd actually expect to see eight total channels. The current config has all 4 DIMM's connected to the first CPU only.

[svmlegacy]

2x Intel Xeon E5-2640 SR0KR

DIMMs as follows:
CPU 0: B1, C1
CPU 1: E1, G1

Odd config as I need to move the dimms to clear coolers... 2 memory channels per CPU in this config.

[cyring]

Oh, a topology problem I have already encountered within previous architecture use cases:

• Two controllers assigned to the same DID.
  Bellow 3ca8 will be probbed twice as controller #0, rather than controllers #0 then #1

7f:0f.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers [8086:3ca8] (rev 07)

ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers [8086:3ca8] (rev 07)

• So I have to refactor the topology discovering; not relying on that kernel DID enumeration; and do a direct BDF addressing per processor socket. Similar to what I did for Jasper
  

  CoreFreq/corefreqk.c

  Line 4895 in c9cb78f

  static PCI_CALLBACK Jasper_Forest_IMC(struct pci_dev *pdev)

[svmlegacy]

Haswell-E Intel Core i7 5960X SR20Q
(Only two memory channels populated.)

[svmlegacy]

@cyring No regression, this motherboard is very picky on POST. Could only get it to light up with a single DIMM.

1x Corsair 8 GB DIMM in channel #0.

[cyring]

This might explain why I can not read DDR3 speed.

Do you have any other Haswell Destop and/or mobile, not Xeon, to confirm IMC decoding ?

[cyring]

This i5-4570 for IMC again, please.

[svmlegacy]

Haswell-DT, which is DDR3. Remember that Haswell-E is DDR4 (And I think Haswell-E/EP has DDR3 onboard like Skylake, but I can't test that...), so these are probably very different memory controllers.

[cyring]

i5-4570

Timings

Setting	CoreFreq	BIOS
tRRD	5	4
tWR	N/A	10
tFAW	N/A	20
tRTP	N/A	5
tWTP	N/A	21
drWR	4	5
tCKE	N/A	4

Geometry

( 8 x Bank x Rank x Rows x Cols ) / ( 1024 x 1024 )

( 8 x 8 x 2 x 16384 x 1024 ) / ( 1024 x 1024) = 2048 MB vs 8192 MB

[cyring]

@svmlegacy Hello

For the 2x Intel Xeon E5-2640 setup, can you please run the attached archive.
In the Kernel log should be printed some PCI CPUBUSNO register(s) as bellow:

CoreFreq: CPUBUSNO[0x00000000] @ BDF(0x0,0x0,0x108)

CoreFreq_develop.tar.gz

[svmlegacy]

As follows:

CoreFreq(0:12:-1): Processor [ 06_2D] Architecture [SandyBridge/EP/Romley] SMT [24/24]
CoreFreq: CPUBUSNO[0x17f00] @ BDF(0x0,0x3c28,0x108)

[cyring]

As follows:

CoreFreq(0:12:-1): Processor [ 06_2D] Architecture [SandyBridge/EP/Romley] SMT [24/24]
CoreFreq: CPUBUSNO[0x17f00] @ BDF(0x0,0x3c28,0x108)

Thank you.

Based on E5 datasheet, these are the two IMC I'm trying to map:

CPUBUSNO is specified as:

With a dump value of 0x17f00 we have:

• IIO/PCH bus number 0x00
• Uncore bus number 0x7f
• Valid bit is one

Refering to the lspci of two processor sockets populated we have found buses 0x7f (for sure) but the other one shows as 0x00 where I would expect 0xff or perhaps 0x80

Device 0x3c28 where CPUBUSNO (of register 0x108) appears twice:

00:05.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management [8086:3c28] (rev 07)

80:05.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management [8086:3c28] (rev 07)

We also have another CPUBUSNO register to query:

More than that; using E7, according to its datasheet, CPUBUSNO has to be queried from BDF(1, 11, 3)

[cyring]

@svmlegacy Hello,

Please find another archive to dump registers printed as CoreFreq: @ BDF() CPUBUSNO[]

CoreFreq_develop.tar.gz

[svmlegacy]

[  319.217850] CoreFreq(10:22:-1): Processor [ 06_2D] Architecture [SandyBridge/EP/Romley] SMT [24/24]
[  319.218051] CoreFreq: @ BDF(0,5,0) CPUBUSNO[17f00]

[cyring]

Thank you for this dump.
BDF(1, 11, 3) did not work apparently.
It's difficult to find a register value or function result which explains why the IMC devices of second socket are listed under bus 0xff (presuming 0x7f being first)
Can it be kernel or BIOS choice ?
Can it be reported differently by lspci ?

[cyring]

Thank you for this dump. BDF(1, 11, 3) did not work apparently. It's difficult to find a register value or function result which explains why the IMC devices of second socket are listed under bus 0xff (presuming 0x7f being first) Can it be kernel or BIOS choice ? Can it be reported differently by lspci ?

Got it; thanks to this 06_3E

CoreFreq: @ BDF(0,5,0) CPUBUSNO[13f00]
CoreFreq: @ BDF(40,5,0) CPUBUSNO[17f40]

1. At bus #0x0 you'll read offset 0x108 for register CPUBUSNO at BDF(0,5,0)
   We start at bus 0x0 because we probed Host Bridge at this BDF location
2. Bits [16-8] UNC = 0x3f
3. Next bus is UNC + 1 = 0x3f + 1 = 0x40
4. At bus #0x40 you'll read offset 0x108 for CPUBUSNO at BDF(0x40,5,0)
5. Bits [16-8] UNC = 0x7f

We thus have two memory controllers, one MC at bus 0x3f and a second at bus 0x7f

Not sure about the MC order and if IIO=0x40 matters in the topology ?

EDIT: using Device 11, Function 3, Offset 0xd0 just confirms the mapping

CoreFreq: @ BDF(0,5,0) CPUBUSNO[13f00]
CoreFreq: @ BDF(3f,11,3) CPUBUSNO[80003f00]
CoreFreq: @ BDF(40,5,0) CPUBUSNO[17f40]
CoreFreq: @ BDF(7f,11,3) CPUBUSNO[80007f40]

Datasheet does not mention an end stop, so I believe 2 IMC is hard coded in architecture

[cyring]

@svmlegacy

Can you please try this version ?
CoreFreq_develop.tar.gz

This time, no dump. The Memory Controllers will be decoded following the CPU socket topology.
This test should only show the DRAM primary timings, aka tCL, for any IMC, channels possible.

Ignore the unused functions warnings during compilation

[cyring]

@svmlegacy
This version will scan all buses for memory controllers
CoreFreq_develop_xeon_all_mc.tar.gz

[svmlegacy]

[cyring]

Here's a dmesg dump from making and running this package: https://gist.github.com/svmlegacy/9979f205743c89ab570b2e6fb27881b8

Since the archive, additional codes should improve the IMC decoding.
Can you pull from the latest develop branch and try again ?

Thank you

[svmlegacy]

[svmlegacy]

It's interesting to see that the memory controller has different values in the unpopulated channels for timings. Everything in the populated channels does look correct at a glace. (And yes, I'm expecting to see channels 1 & 3 on CPU0 and 0 & 2 on CPU1

[cyring]

It's interesting to see that the memory controller has different values in the unpopulated channels for timings. Everything in the populated channels does look correct at a glace. (And yes, I'm expecting to see channels 1 & 3 on CPU0 and 0 & 2 on CPU1

Great progress on the PCI topology.

QPI rate is also available.

Specs of some third timings are still unknown

I'm close to merge into Master ...

Thank you.

[cyring]

Results from the latest development commit: