Discussion:
XSA-60 solutions
(too old to reply)
Liu, Jinsong
2013-09-30 20:28:53 UTC
Permalink
Hi, All

This email provides 2 solutions for XSA-60 issue found by Konrad (refer attached email for XSA-60 please).

Basically it involves how to emulate guest setting cr0.cd. For shadow, as Jan pointed out in earlier email Xen drop all shadows so that any new ones would be created with UC memory type, _not_ involving iteration over the whole address space. For EPT, currently Xen traverse all ept entries via problematic set_uc_mode, resulting in DOS-like behavior, so this email focus on Intel EPT case.

Solution 1 is Dual-EPT tables: When guest setting cr0.cd trapped, stop using normal EPT, switch to a temp EPT table and populate new EPT entries w/ UC type on demand at later EPT violation. When guest clearing cr0.cd, switch back to normal EPT. In this way, _no_ unbounded loop involved and hence security hole avoided.

Some concerns for Dual-EPT: the 1st concern comes from cachablity confliction between guest and Xen memory type point of view, though it also exists in current implementation. The 2nd concern comes from Dual EPT tables inconsistency/sync issue: things become complicated when p2m modifying, PoD populating, and super page spliting, etc.

Solution 2 is via PAT emulation: For guest w/o VT-d, and for guest with VT-d but snooped, Xen need do nothing, just simply ignore guest setting cr0.cd, since hardware snoop mechanism has ensured cache coherency (under these cases currently Xen has set EPT iPAT bit, ignore guest's memory type opinion); For guest with VT-d but non-snooped, cache coherency can not be guaranteed by h/w snoop so guest's memory type opinion must be considered (under this case Xen set iPAT bit combining guest and host memory type opinion). Only under this case PAT emulation need set all IA32_PAT fields as UC so that guest memory type are all UC.

Concern for PAT solution still comes from cachablity confliction between guest and Xen.

Thoughts?
BTW, today is Chinese National day, I will have several days travel with no email access, but your comments/suggestions are highly appreciated and I will reply ASAP after I come back.

Thanks,
Jinsong
Liu, Jinsong
2013-09-30 20:42:26 UTC
Permalink
Post by Liu, Jinsong
Hi, All
This email provides 2 solutions for XSA-60 issue found by Konrad
(refer attached email for XSA-60 please).
Basically it involves how to emulate guest setting cr0.cd. For
shadow, as Jan pointed out in earlier email Xen drop all shadows so
that any new ones would be created with UC memory type, _not_
involving iteration over the whole address space. For EPT, currently
Xen traverse all ept entries via problematic set_uc_mode, resulting
in DOS-like behavior, so this email focus on Intel EPT case.
Solution 1 is Dual-EPT tables: When guest setting cr0.cd trapped,
stop using normal EPT, switch to a temp EPT table and populate new
EPT entries w/ UC type on demand at later EPT violation. When guest
clearing cr0.cd, switch back to normal EPT. In this way, _no_
unbounded loop involved and hence security hole avoided.
Some concerns for Dual-EPT: the 1st concern comes from cachablity
confliction between guest and Xen memory type point of view, though
it also exists in current implementation. The 2nd concern comes from
Dual EPT tables inconsistency/sync issue: things become complicated
when p2m modifying, PoD populating, and super page spliting, etc.
Solution 2 is via PAT emulation: For guest w/o VT-d, and for guest
with VT-d but snooped, Xen need do nothing, just simply ignore guest
setting cr0.cd, since hardware snoop mechanism has ensured cache
coherency (under these cases currently Xen has set EPT iPAT bit,
ignore guest's memory type opinion); For guest with VT-d but
non-snooped, cache coherency can not be guaranteed by h/w snoop so
guest's memory type opinion must be considered (under this case Xen
set iPAT bit combining guest and host memory type opinion). Only
Sorry, under this case Xen _clear_ iPAT, combining guest and host memory type opinion.

Thanks,
Jinsong
Post by Liu, Jinsong
under this case PAT emulation need set all IA32_PAT fields as UC so
that guest memory type are all UC.
Concern for PAT solution still comes from cachablity confliction
between guest and Xen.
Thoughts?
BTW, today is Chinese National day, I will have several days travel
with no email access, but your comments/suggestions are highly
appreciated and I will reply ASAP after I come back.
Thanks,
Jinsong
Liu, Jinsong
2013-10-07 14:29:33 UTC
Permalink
Hi, All

Any comments/suggestions?

Thanks,
Jinsong
Post by Liu, Jinsong
Post by Liu, Jinsong
Hi, All
This email provides 2 solutions for XSA-60 issue found by Konrad
(refer attached email for XSA-60 please).
Basically it involves how to emulate guest setting cr0.cd. For
shadow, as Jan pointed out in earlier email Xen drop all shadows so
that any new ones would be created with UC memory type, _not_
involving iteration over the whole address space. For EPT, currently
Xen traverse all ept entries via problematic set_uc_mode, resulting
in DOS-like behavior, so this email focus on Intel EPT case.
Solution 1 is Dual-EPT tables: When guest setting cr0.cd trapped,
stop using normal EPT, switch to a temp EPT table and populate new
EPT entries w/ UC type on demand at later EPT violation. When guest
clearing cr0.cd, switch back to normal EPT. In this way, _no_
unbounded loop involved and hence security hole avoided.
Some concerns for Dual-EPT: the 1st concern comes from cachablity
confliction between guest and Xen memory type point of view, though
it also exists in current implementation. The 2nd concern comes from
Dual EPT tables inconsistency/sync issue: things become complicated
when p2m modifying, PoD populating, and super page spliting, etc.
Solution 2 is via PAT emulation: For guest w/o VT-d, and for guest
with VT-d but snooped, Xen need do nothing, just simply ignore guest
setting cr0.cd, since hardware snoop mechanism has ensured cache
coherency (under these cases currently Xen has set EPT iPAT bit,
ignore guest's memory type opinion); For guest with VT-d but
non-snooped, cache coherency can not be guaranteed by h/w snoop so
guest's memory type opinion must be considered (under this case Xen
set iPAT bit combining guest and host memory type opinion). Only
Sorry, under this case Xen _clear_ iPAT, combining guest and host memory type opinion.
Thanks,
Jinsong
Post by Liu, Jinsong
under this case PAT emulation need set all IA32_PAT fields as UC so
that guest memory type are all UC.
Concern for PAT solution still comes from cachablity confliction
between guest and Xen.
Thoughts?
BTW, today is Chinese National day, I will have several days travel
with no email access, but your comments/suggestions are highly
appreciated and I will reply ASAP after I come back.
Thanks,
Jinsong
Jan Beulich
2013-10-07 15:04:48 UTC
Permalink
Post by Liu, Jinsong
Any comments/suggestions?
As pointed out in earlier private conversation, I think that the dual
table approach would be preferable if it can be made work.

I'm surprised no-one from Oracle responded so far, as it was them
originally having found the issue.

Jan
Post by Liu, Jinsong
Post by Liu, Jinsong
Post by Liu, Jinsong
Hi, All
This email provides 2 solutions for XSA-60 issue found by Konrad
(refer attached email for XSA-60 please).
Basically it involves how to emulate guest setting cr0.cd. For
shadow, as Jan pointed out in earlier email Xen drop all shadows so
that any new ones would be created with UC memory type, _not_
involving iteration over the whole address space. For EPT, currently
Xen traverse all ept entries via problematic set_uc_mode, resulting
in DOS-like behavior, so this email focus on Intel EPT case.
Solution 1 is Dual-EPT tables: When guest setting cr0.cd trapped,
stop using normal EPT, switch to a temp EPT table and populate new
EPT entries w/ UC type on demand at later EPT violation. When guest
clearing cr0.cd, switch back to normal EPT. In this way, _no_
unbounded loop involved and hence security hole avoided.
Some concerns for Dual-EPT: the 1st concern comes from cachablity
confliction between guest and Xen memory type point of view, though
it also exists in current implementation. The 2nd concern comes from
Dual EPT tables inconsistency/sync issue: things become complicated
when p2m modifying, PoD populating, and super page spliting, etc.
Solution 2 is via PAT emulation: For guest w/o VT-d, and for guest
with VT-d but snooped, Xen need do nothing, just simply ignore guest
setting cr0.cd, since hardware snoop mechanism has ensured cache
coherency (under these cases currently Xen has set EPT iPAT bit,
ignore guest's memory type opinion); For guest with VT-d but
non-snooped, cache coherency can not be guaranteed by h/w snoop so
guest's memory type opinion must be considered (under this case Xen
set iPAT bit combining guest and host memory type opinion). Only
Sorry, under this case Xen _clear_ iPAT, combining guest and host memory type opinion.
Thanks,
Jinsong
Post by Liu, Jinsong
under this case PAT emulation need set all IA32_PAT fields as UC so
that guest memory type are all UC.
Concern for PAT solution still comes from cachablity confliction
between guest and Xen.
Thoughts?
BTW, today is Chinese National day, I will have several days travel
with no email access, but your comments/suggestions are highly
appreciated and I will reply ASAP after I come back.
Thanks,
Jinsong
Andrew Cooper
2013-10-07 15:08:25 UTC
Permalink
Post by Jan Beulich
Post by Liu, Jinsong
Any comments/suggestions?
As pointed out in earlier private conversation, I think that the dual
table approach would be preferable if it can be made work.
I'm surprised no-one from Oracle responded so far, as it was them
originally having found the issue.
Jan
I cant remember whether I asked this before or not, but is there a
reason why this cant be done in the same way as hypercall continuations?

~Andrew
Jan Beulich
2013-10-07 15:23:37 UTC
Permalink
Post by Andrew Cooper
Post by Jan Beulich
Post by Liu, Jinsong
Any comments/suggestions?
As pointed out in earlier private conversation, I think that the dual
table approach would be preferable if it can be made work.
I'm surprised no-one from Oracle responded so far, as it was them
originally having found the issue.
I cant remember whether I asked this before or not, but is there a
reason why this cant be done in the same way as hypercall continuations?
I think you did; the reason is that we're in the middle of processing
a CR0 write here, and hence can't intermediately exit back to the
guest (which is what we'd do for hypercall continuations). However,
you mentioning this one makes me think whether we couldn't
leverage/extend/clone the MMIO retry logic, or at least deal with
this in ways similar to it (i.e. preventing VM re-entry until processing
is complete). But maybe that would result in even worse hackery...

Jan
Liu, Jinsong
2013-10-07 16:03:03 UTC
Permalink
Post by Jan Beulich
Post by Liu, Jinsong
wrote: Any comments/suggestions?
As pointed out in earlier private conversation, I think that the dual
table approach would be preferable if it can be made work.
No, solution 2 has an important update compared with our private conversation before:
The old solution 2 is, Xen do nothing for guest cr0.cd setting.
The new solution 2 in this email is, Xen set IA32_PAT fields as UC when needed (that means, only when guest work with vt-d but vt-d non-snooped, since under this case h/w can not guarantee cache coherency), so that all guest memory type are UC. For other cases (when guest work w/o vt-d, and when guest work with vt-d but vt-d snooped), Xen can still do nothing for guest cr0.cd setting (since h/w has guaranteed cache coherency, otherwise Xen doesn't dare to do what it does now -- it set iPAT bit as 1, totally ignoring guest memory type point of view).

Compared with Dual-EPT solution, it's much simpler (only need IA32_PAT MSR emulation under rare case) and avoid the inconsistency issue between 2 EPT tables.

Thanks,
Jinsong
Post by Jan Beulich
I'm surprised no-one from Oracle responded so far, as it was them
originally having found the issue.
Jan
Post by Liu, Jinsong
Hi, All
This email provides 2 solutions for XSA-60 issue found by Konrad
(refer attached email for XSA-60 please).
Basically it involves how to emulate guest setting cr0.cd. For
shadow, as Jan pointed out in earlier email Xen drop all shadows so
that any new ones would be created with UC memory type, _not_
involving iteration over the whole address space. For EPT,
currently Xen traverse all ept entries via problematic
set_uc_mode, resulting in DOS-like behavior, so this email focus
on Intel EPT case.
Solution 1 is Dual-EPT tables: When guest setting cr0.cd trapped,
stop using normal EPT, switch to a temp EPT table and populate new
EPT entries w/ UC type on demand at later EPT violation. When guest
clearing cr0.cd, switch back to normal EPT. In this way, _no_
unbounded loop involved and hence security hole avoided.
Some concerns for Dual-EPT: the 1st concern comes from cachablity
confliction between guest and Xen memory type point of view, though
it also exists in current implementation. The 2nd concern comes
from Dual EPT tables inconsistency/sync issue: things become
complicated when p2m modifying, PoD populating, and super page
spliting, etc.
Solution 2 is via PAT emulation: For guest w/o VT-d, and for guest
with VT-d but snooped, Xen need do nothing, just simply ignore
guest setting cr0.cd, since hardware snoop mechanism has ensured
cache coherency (under these cases currently Xen has set EPT iPAT
bit, ignore guest's memory type opinion); For guest with VT-d but
non-snooped, cache coherency can not be guaranteed by h/w snoop so
guest's memory type opinion must be considered (under this case Xen
set iPAT bit combining guest and host memory type opinion). Only
Sorry, under this case Xen _clear_ iPAT, combining guest and host memory type opinion.
Thanks,
Jinsong
under this case PAT emulation need set all IA32_PAT fields as UC so
that guest memory type are all UC.
Concern for PAT solution still comes from cachablity confliction
between guest and Xen.
Thoughts?
BTW, today is Chinese National day, I will have several days travel
with no email access, but your comments/suggestions are highly
appreciated and I will reply ASAP after I come back.
Thanks,
Jinsong
Jan Beulich
2013-10-08 07:40:43 UTC
Permalink
Post by Liu, Jinsong
Post by Jan Beulich
wrote: Any comments/suggestions?
As pointed out in earlier private conversation, I think that the dual
table approach would be preferable if it can be made work.
The old solution 2 is, Xen do nothing for guest cr0.cd setting.
The new solution 2 in this email is, Xen set IA32_PAT fields as UC when
needed (that means, only when guest work with vt-d but vt-d non-snooped, since
under this case h/w can not guarantee cache coherency), so that all guest
memory type are UC. For other cases (when guest work w/o vt-d, and when guest
work with vt-d but vt-d snooped), Xen can still do nothing for guest cr0.cd
setting (since h/w has guaranteed cache coherency, otherwise Xen doesn't dare
to do what it does now -- it set iPAT bit as 1, totally ignoring guest memory
type point of view).
Compared with Dual-EPT solution, it's much simpler (only need IA32_PAT MSR
emulation under rare case) and avoid the inconsistency issue between 2 EPT
tables.
Right - as long as PAT is guaranteed to only be used for guest
induced memory accesses, that sounds like a viable yet not too
intrusive solution. Short of anyone else having an opinion here,
why don't you start drafting a patch for this?

Jan
Liu, Jinsong
2013-10-09 10:41:00 UTC
Permalink
Post by Jan Beulich
Post by Liu, Jinsong
Post by Jan Beulich
wrote: Any comments/suggestions?
As pointed out in earlier private conversation, I think that the
dual table approach would be preferable if it can be made work.
The old solution 2 is, Xen do nothing for guest cr0.cd setting.
The new solution 2 in this email is, Xen set IA32_PAT fields as UC
when needed (that means, only when guest work with vt-d but vt-d
non-snooped, since under this case h/w can not guarantee cache
coherency), so that all guest memory type are UC. For other cases
(when guest work w/o vt-d, and when guest work with vt-d but vt-d
snooped), Xen can still do nothing for guest cr0.cd setting (since
h/w has guaranteed cache coherency, otherwise Xen doesn't dare to do
what it does now -- it set iPAT bit as 1, totally ignoring guest
memory type point of view).
Compared with Dual-EPT solution, it's much simpler (only need
IA32_PAT MSR emulation under rare case) and avoid the inconsistency
issue between 2 EPT tables.
Right - as long as PAT is guaranteed to only be used for guest
induced memory accesses, that sounds like a viable yet not too
intrusive solution. Short of anyone else having an opinion here,
why don't you start drafting a patch for this?
Jan
That's great! We will draft patch.

Another point is, for shadow, currently Xen drop all shadows so that any new ones would be created on demand with UC. It works safely for XSA-60, but seems not good enough. We can still use IA32_PAT to solve it, not touch shadow page table itself.

One thing I need more confirm from AMD side: iirc, AMD IOMMU always works with snooped so cache cohernecy has been guaranteed, so no need do anything for guest cr0.cd setting, no matter shadow or NPT, right? If yes, most code could be done w/ Intel specific.

Thanks,
Jinsong
Jan Beulich
2013-10-09 11:40:25 UTC
Permalink
Post by Liu, Jinsong
Another point is, for shadow, currently Xen drop all shadows so that any new
ones would be created on demand with UC. It works safely for XSA-60, but seems
not good enough. We can still use IA32_PAT to solve it, not touch shadow page
table itself.
Tim, any though on this?
Post by Liu, Jinsong
One thing I need more confirm from AMD side: iirc, AMD IOMMU always works
with snooped so cache cohernecy has been guaranteed, so no need do anything
for guest cr0.cd setting, no matter shadow or NPT, right? If yes, most code
could be done w/ Intel specific.
Iirc SVM allows the guest to run with CR0.CD set. Whether that's
entirely valid to do I'm not certain, but I suppose it's at least to some
degree due to the major inhibiting factor (hyperthreads) not existing
there.

Jan
Tim Deegan
2013-10-09 18:01:46 UTC
Permalink
Post by Jan Beulich
Post by Liu, Jinsong
Another point is, for shadow, currently Xen drop all shadows so that any new
ones would be created on demand with UC. It works safely for XSA-60, but seems
not good enough. We can still use IA32_PAT to solve it, not touch shadow page
table itself.
Tim, any though on this?
The existing shadow solution is OK, I think -- but if the same solution
that's used for EPT can be applied there, so much the better.

Tim.
Post by Jan Beulich
Post by Liu, Jinsong
One thing I need more confirm from AMD side: iirc, AMD IOMMU always works
with snooped so cache cohernecy has been guaranteed, so no need do anything
for guest cr0.cd setting, no matter shadow or NPT, right? If yes, most code
could be done w/ Intel specific.
Iirc SVM allows the guest to run with CR0.CD set. Whether that's
entirely valid to do I'm not certain, but I suppose it's at least to some
degree due to the major inhibiting factor (hyperthreads) not existing
there.
Jan
_______________________________________________
Xen-devel mailing list
http://lists.xen.org/xen-devel
DuanZhenzhong
2013-10-08 02:25:38 UTC
Permalink
Post by Jan Beulich
Post by Liu, Jinsong
Any comments/suggestions?
As pointed out in earlier private conversation, I think that the dual
table approach would be preferable if it can be made work.
I'm surprised no-one from Oracle responded so far, as it was them
originally having found the issue.
Just back from vacation, we could help test if there is any patch out.

zduan
Jan Beulich
2013-10-08 07:37:42 UTC
Permalink
Post by DuanZhenzhong
Post by Jan Beulich
Post by Liu, Jinsong
Any comments/suggestions?
As pointed out in earlier private conversation, I think that the dual
table approach would be preferable if it can be made work.
I'm surprised no-one from Oracle responded so far, as it was them
originally having found the issue.
Just back from vacation, we could help test if there is any patch out.
Before getting to test anything, we first need to sort out how
we want to address the issue - that's where input would be
highly appreciated.

Jan
Konrad Rzeszutek Wilk
2013-10-01 13:44:19 UTC
Permalink
Post by Liu, Jinsong
Hi, All
This email provides 2 solutions for XSA-60 issue found by Konrad (refer attached email for XSA-60 please).
Basically it involves how to emulate guest setting cr0.cd. For shadow, as Jan pointed out in earlier email Xen drop all shadows so that any new ones would be created with UC memory type, _not_ involving iteration over the whole address space. For EPT, currently Xen traverse all ept entries via problematic set_uc_mode, resulting in DOS-like behavior, so this email focus on Intel EPT case.
Solution 1 is Dual-EPT tables: When guest setting cr0.cd trapped, stop using normal EPT, switch to a temp EPT table and populate new EPT entries w/ UC type on demand at later EPT violation. When guest clearing cr0.cd, switch back to normal EPT. In this way, _no_ unbounded loop involved and hence security hole avoided.
Some concerns for Dual-EPT: the 1st concern comes from cachablity confliction between guest and Xen memory type point of view, though it also exists in current implementation. The 2nd concern comes from Dual EPT tables inconsistency/sync issue: things become complicated when p2m modifying, PoD populating, and super page spliting, etc.
Solution 2 is via PAT emulation: For guest w/o VT-d, and for guest with VT-d but snooped, Xen need do nothing, just simply ignore guest setting cr0.cd, since hardware snoop mechanism has ensured cache coherency (under these cases currently Xen has set EPT iPAT bit, ignore guest's memory type opinion); For guest with VT-d but non-snooped, cache coherency can not be guaranteed by h/w snoop so guest's memory type opinion must be considered (under this case Xen set iPAT bit combining guest and host memory type opinion). Only under this case PAT emulation need set all IA32_PAT fields as UC so that guest memory type are all UC.
Concern for PAT solution still comes from cachablity confliction between guest and Xen.
Thoughts?
BTW, today is Chinese National day, I will have several days travel with no email access, but your comments/suggestions are highly appreciated and I will reply ASAP after I come back.
Thanks,
Jinsong
Date: Wed, 24 Jul 2013 11:36:55 +0000
Subject: [Xen-devel] Xen Security Advisory 60 (CVE-2013-2212) - Excessive
time to disable caching with HVM guests with PCI passthrough
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Xen Security Advisory CVE-2013-2212 / XSA-60
version 4
Excessive time to disable caching with HVM guests with PCI passthrough
UPDATES IN VERSION 4
====================
Public release.
ISSUE DESCRIPTION
=================
HVM guests are able to manipulate their physical address space such that
processing a subsequent request by that guest to disable caches takes an
extended amount of time changing the cachability of the memory pages assigned
to this guest. This applies only when the guest has been granted access to
some memory mapped I/O region (typically by way of assigning a passthrough
PCI device).
This can cause the CPU which processes the request to become unavailable,
possibly causing the hypervisor or a guest kernel (including the domain 0 one)
to halt itself ("panic").
For reference, as long as no patch implementing an approved alternative
solution is available (there's only a draft violating certain requirements
set by Intel's documentation), the problematic code is the function
vmx_set_uc_mode() (in that it calls ept_change_entry_emt_with_range() with
the full guest GFN range, which the guest has control over, but which also
would be a problem with sufficiently large but not malicious guests).
IMPACT
======
A malicious domain, given access to a device with memory mapped I/O
regions, can cause the host to become unresponsive for a period of
time, potentially leading to a DoS affecting the whole system.
VULNERABLE SYSTEMS
==================
Xen version 3.3 onwards is vulnerable.
Only systems using the Intel variant of Hardware Assisted Paging (aka EPT) are
vulnerable.
MITIGATION
==========
This issue can be avoided by not assigning PCI devices to untrusted guests, or
by running HVM guests with shadow mode paging (through adding "hap=0" to the
domain configuration file).
CREDITS
=======
Konrad Wilk found the issue as a bug, which on examination by the
Xenproject.org Security Team turned out to be a security problem.
RESOLUTION
==========
There is currently no resolution to this issue.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iQEcBAEBAgAGBQJR77wrAAoJEIP+FMlX6CvZB5MH/ibfpjHuoGOIo7mWukld4NM5
UVIKC+rTrnkYhbF2f+xIM833+WAUjPuXZKZ6/EirDAPAAQCut2DouNvVdVnZ5cBx
rq0N8l9wy0/dq/7kCyI3kAGFlJ3VYz7aM5+TTPFGfO7Yq3ohUNu2EE4vv/t5KVjD
H4reh8UaA5QuRbdh3evCM9Vdt2syqi8JQwB5D2CJqrgAuFPwEVle8MLKSXWWb/+V
KUy+mRAb1tN3jbWIev0TZ7Hm3x61yO60/WFzsQzkmkd+qWvC5btkWDg05K5DHC+Q
yvFU3Y5u7J/ub00ZO4e9wjNDG5+ItQUK4xp8y5s65qx27P/eK9VLi8dvnHVMk04=
=HUbY
-----END PGP SIGNATURE-----
_______________________________________________
Xen-devel mailing list
http://lists.xen.org/xen-devel
Continue reading on narkive:
Loading...