[Xen-devel] MSR load lists on Harpertown

Discussion:

Andrew Cooper

2018-10-16 18:54:25 UTC

Hello,

I realise this is an old CPU, but I've I've encountered a weird failure
on it.

Specifically:

(XEN) CPU Vendor: Intel, Family 6 (0x6), Model 23 (0x17), Stepping 6
(raw 00010676)
[***@harpertown ~]# head /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           E5420 @ 2.50GHz
stepping    : 6
microcode    : 0x60f
cpu MHz        : 2493.756
cache size    : 6144 KB
physical id    : 0

In Xen, we use an MSR load list to update EFER on vmentry/exit, when
hardware doesn't support the EFER field in the VMCB itself. This is a
change I made in 4.11 to fix a bug with NX handling on context switching.

After some investigation, it turns out that after vmentry, while the
load list has the value 0xd01 (NXE, LMA, LME, SCE), the value loaded
into hardware is 0xd00 (NXE, LMA, LME).

I.e. when an MSR load list is used for EFER, we resume the guest with
SCE cleared. This is rather terminal for 64bit guests, as
syscall/sysret instructions take a #UD fault.

I can't see anything relevant in the Specification Update for this
processor.

I've confirmed that by not using a load list, the current value in EFER
is preserved once the vmentry is complete, and by disabling the EFER
intercept, I can re-set SCE in non-root context and have syscall/sysret
work correctly.

However, given this behaviour, I can't think of any way to context
switch NX properly, and leave 64bit guests in a working state.

Do you have any suggestions?

Thanks,

~Andrew

Ian Jackson

2018-11-05 16:25:05 UTC

Permalink

Post by Andrew Cooper
I realise this is an old CPU, but I've I've encountered a weird failure
on it.

...

Post by Andrew Cooper
However, given this behaviour, I can't think of any way to context
switch NX properly, and leave 64bit guests in a working state.
Do you have any suggestions?

ISTM that Xen should run on these old CPUs and also that it should
work, ideally, even if the microcode is buggy.

So presumaly the right answer is to simply disable NX for the guest
completely ?

Ian.

Andrew Cooper

2018-11-05 16:36:07 UTC

Permalink

Post by Ian Jackson

Post by Andrew Cooper
I realise this is an old CPU, but I've I've encountered a weird failure
on it.

...

Post by Andrew Cooper
However, given this behaviour, I can't think of any way to context
switch NX properly, and leave 64bit guests in a working state.
Do you have any suggestions?

ISTM that Xen should run on these old CPUs and also that it should
work, ideally, even if the microcode is buggy.
So presumaly the right answer is to simply disable NX for the guest
completely ?

As you've observed, when SCE doesn't work, everything is fully hosed.

However, disabling NX on that CPU won't fix the problem, because the
problem appears to be using the MSR load list.

Without a microcode fix (if indeed this is a microcode issue, but given
how much of vmentry is, I'd be surprised if it is something else), the
only fix I can see is to effectively revert the change to use MSR load
lists.

However, I really don't want to do this unilaterally, because it will
break NX handling on all Intel hardware, and reintroduce a wart into the
context switch code which I thought I'd excised properly.

~Andrew

Andrew Cooper

2018-12-10 17:12:59 UTC

Permalink

Ping Kevin / Jun.

Post by Andrew Cooper
Hello,
I realise this is an old CPU, but I've I've encountered a weird failure
on it.
(XEN) CPU Vendor: Intel, Family 6 (0x6), Model 23 (0x17), Stepping 6
(raw 00010676)
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
stepping    : 6
microcode    : 0x60f
cpu MHz        : 2493.756
cache size    : 6144 KB
physical id    : 0
In Xen, we use an MSR load list to update EFER on vmentry/exit, when
hardware doesn't support the EFER field in the VMCB itself. This is a
change I made in 4.11 to fix a bug with NX handling on context switching.
After some investigation, it turns out that after vmentry, while the
load list has the value 0xd01 (NXE, LMA, LME, SCE), the value loaded
into hardware is 0xd00 (NXE, LMA, LME).
I.e. when an MSR load list is used for EFER, we resume the guest with
SCE cleared. This is rather terminal for 64bit guests, as
syscall/sysret instructions take a #UD fault.
I can't see anything relevant in the Specification Update for this
processor.
I've confirmed that by not using a load list, the current value in EFER
is preserved once the vmentry is complete, and by disabling the EFER
intercept, I can re-set SCE in non-root context and have syscall/sysret
work correctly.
However, given this behaviour, I can't think of any way to context
switch NX properly, and leave 64bit guests in a working state.
Do you have any suggestions?
Thanks,
~Andrew
_______________________________________________
Xen-devel mailing list
https://lists.xenproject.org/mailman/listinfo/xen-devel

Tian, Kevin

2018-12-11 07:10:08 UTC

Permalink

Sent: Tuesday, December 11, 2018 1:13 AM
Ping Kevin / Jun.

can you point to the commit number of NX fix?

Post by Andrew Cooper
After some investigation, it turns out that after vmentry, while the
load list has the value 0xd01 (NXE, LMA, LME, SCE), the value loaded
into hardware is 0xd00 (NXE, LMA, LME).
I.e. when an MSR load list is used for EFER, we resume the guest with
SCE cleared. This is rather terminal for 64bit guests, as
syscall/sysret instructions take a #UD fault.
I can't see anything relevant in the Specification Update for this
processor.
I've confirmed that by not using a load list, the current value in EFER
is preserved once the vmentry is complete, and by disabling the EFER
intercept, I can re-set SCE in non-root context and have syscall/sysret
work correctly.
However, given this behaviour, I can't think of any way to context
switch NX properly, and leave 64bit guests in a working state.
Do you have any suggestions?

I'm checking internally whether it's a known issue.

btw did you try upgrading to a newer microcode?

Thanks
Kevin