vt-d p7p55d evo

[Mark Hurenkamp]

Hi,

I'm not a subscriber to xen-devel yet, but did see your post about
the evo and iommu not working. Within the last week I have
installed a xen 4.0rc8 and xen/stable clone kernel 2.6.32.10 and
boot ok with iommu=1 and have also successfully passed through a tv
tuner card tonight which modules did load for. I did have one weird
quirk though. I haven't tested the card in the domu yet. Anyway if
you want to discuss this further then reply back.

Are you sure you are using VT-d here (are you using hvm loader)? I'd
be very interested to know what Bios version you are using, what your
'xm dmesg' and 'xm info' output is (after boot), and the xen
configuration file for your VM.
Cause although my system can boot fine with enabled iommu now, it is
simply switched off by xen at boottime... Did you do something special
to force it to on? Do you have any patches installed?
Btw, passthrough of a tuner device to a pvm domU is not a problem
here, but that does not require VT-d (nor VT-x).
Did you manage to passthrough a device to Windows?
My ultimate goal is to passthrough a (secondary) usb card, and a
(secondary) graphics card, to a Windows hvm.


Mark.

[Mark Hurenkamp]

I'm still pretty new to Xen so it could very well be that I've
misunderstood the requirements for certain capabilities. I thought
one of the wiki's had stated that to pass through to an hvm guest
you have to have VT-d though. Is there something specific that I
which can indicate its status after boot?
bios: 1207
cpu: i7 860
grub2: http://pastebin.ca/1856369
xm dmesg: http://pastebin.ca/1856370
xm info: http://pastebin.ca/1856372
hvm cfg: http://pastebin.ca/1856376
I think I compiled the linux kernel at xen/stable git commit
221c4dbf86 given the timestamp in xm info (all done same time frame)

From your earlier mail, it was not clear wether you were indeed using
hvm, and that xen enabled vt-d properly.
From your xm dmesg, and xm info, it is clear that iommu is indeed
enabled for you, something which i've not been able to achieve with my
board.
Could the CPU make a difference here? I'm on a i5 750, and notice you
are using an i7 860....

I'll see if i can do some more experiments with my setup, perhaps
switching back a few changesets to 4.0-rc8.
I'm not sure if the kernel makes a difference, but i'll see if i can
match your configuration as close as possible.

Thanks for your input!


Mark.

[Mark Hurenkamp]

according to this chart (and thread) the i5 doesn't support VT-d, Loading Image...
link source: http://www.tomshardware.com/forum/274342-28-tomshardware

Ok... i didn't know that... it would surely explain why I can't get VT-
d to work on my system...
I guess i'll have to upgrade the CPU then, anyone interested in a used
i5 750 ;-)


Mark.

[Sander Eikelenboom]

Hi Mark,

I guess that's your problem, your cpu doesn't seem to support VT-D, see
http://ark.intel.com/Product.aspx?id=42915

Both CPU, chipset and bios most support it ...

--
Sander

Post by Mark Hurenkamp

I'm still pretty new to Xen so it could very well be that I've
misunderstood the requirements for certain capabilities. I thought
one of the wiki's had stated that to pass through to an hvm guest
you have to have VT-d though. Is there something specific that I
which can indicate its status after boot?
bios: 1207
cpu: i7 860
grub2: http://pastebin.ca/1856369
xm dmesg: http://pastebin.ca/1856370
xm info: http://pastebin.ca/1856372
hvm cfg: http://pastebin.ca/1856376
I think I compiled the linux kernel at xen/stable git commit
221c4dbf86 given the timestamp in xm info (all done same time frame)

From your earlier mail, it was not clear wether you were indeed using
hvm, and that xen enabled vt-d properly.
From your xm dmesg, and xm info, it is clear that iommu is indeed
enabled for you, something which i've not been able to achieve with my
board.
Could the CPU make a difference here? I'm on a i5 750, and notice you
are using an i7 860....
I'll see if i can do some more experiments with my setup, perhaps
switching back a few changesets to 4.0-rc8.
I'm not sure if the kernel makes a difference, but i'll see if i can
match your configuration as close as possible.
Thanks for your input!
Mark.

[Mark Hurenkamp]

Hi,

I replaced the i5 with an i7 CPU, and now xen seems to enable vtd properly,
i have yet to test the feature with my board, but xm info now shows the
hvm_directio flag:
host : coruscant
release : 2.6.32.11mh18
version : #10 SMP PREEMPT Wed Apr 7 23:57:39 CEST 2010
machine : x86_64
nr_cpus : 8
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 3660
hw_caps :
bfebfbff:28100800:00000000:00001b40:0098e3fd:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 7975
free_memory : 2425
node_to_cpu : node0:0-7
node_to_memory : node0:2425
node_to_dma32_mem : node0:1310
max_node_id : 0
xen_major : 4
xen_minor : 0
xen_extra : .0
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : Wed Apr 07 12:38:28 2010 +0100 21091:f28f1ee587c8
xen_commandline : console_to_ring com1=115200,8n1 console=com1
cc_compiler : gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu9)
cc_compile_by : root
cc_compile_domain : karpeer.net
cc_compile_date : Wed Apr 7 23:16:01 CEST 2010
xend_config_format : 4


Regards,
Mark.

Post by Sander Eikelenboom
Hi Mark,
I guess that's your problem, your cpu doesn't seem to support VT-D, see
http://ark.intel.com/Product.aspx?id=42915
Both CPU, chipset and bios most support it ...
--
Sander

[Mark Hurenkamp]

Hi,


The last few days i played some more with the vt-d feature enabled on
my system,
I'm now able to passthrough an usb controller to a windows xp hvm.
When i plug
a usb drive into a slot controlled by that controller, windows behaves
as expected.
Passing through a hda-intel device (from my graphics card) seems to at
least be
recognised properly by windows, and windows is able to install the
driver (can't
verify that it is actually playing audio, since the graphics card
itself can't be passed
through yet, and thus the connected monitor is in standby).
So as far as i can tell, vt-d is working fine now on this system.

Passthrough of the secondary graphics card to windows, doesn't seem to
work
(well, i guess that's to be expected at this stage, since i've only
tried passing
it through as a normal pci card, and i think it needs an extra flag
set in the config
and some patches in xen, as well as passing along a bios file).
Strange thing i noticed though, when i try to boot windows with the
secondary graphics
card listed in its pci section, my primary X session will reset, and
i'm presented the gdm
login screen... (primary & secondary graphics card are both hd4350).
Another strange thing i noticed, is that when i kill windows using xm
shutdown (since
it won't boot with the graphics card listed), i am not able to start
it again later, not even
after removing the pci devices from the config.
Only after a machine reboot, things will work normal again.

Next on my list: try passthrough of the primary graphics card...


Regards,
Mark

Post by Mark Hurenkamp
Hi,
I replaced the i5 with an i7 CPU, and now xen seems to enable vtd properly,
i have yet to test the feature with my board, but xm info now shows the
host : coruscant
release : 2.6.32.11mh18
version : #10 SMP PREEMPT Wed Apr 7 23:57:39 CEST 2010
machine : x86_64
nr_cpus : 8
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 3660
00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 7975
free_memory : 2425
node_to_cpu : node0:0-7
node_to_memory : node0:2425
node_to_dma32_mem : node0:1310
max_node_id : 0
xen_major : 4
xen_minor : 0
xen_extra : .0
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : Wed Apr 07 12:38:28 2010 +0100
21091:f28f1ee587c8
xen_commandline : console_to_ring com1=115200,8n1 console=com1
cc_compiler : gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu9)
cc_compile_by : root
cc_compile_domain : karpeer.net
cc_compile_date : Wed Apr 7 23:16:01 CEST 2010
xend_config_format : 4
Regards,
Mark.

Post by Sander Eikelenboom
Hi Mark,
I guess that's your problem, your cpu doesn't seem to support VT-D, see
http://ark.intel.com/Product.aspx?id=42915
Both CPU, chipset and bios most support it ...
--
Sander

_______________________________________________
Xen-devel mailing list
http://lists.xensource.com/xen-devel

[listmail]

I had no problems passing through my Nvidia 8800gts 512. The nv driver
on linux works fine (as good as it can). There were a few quirks to
note, but I imagine your experience will vary using ATI.

- It takes a bit before you see VGA output
- The cursor was positioned a higher than normal (sitting at over the
halfway mark for the font)
- Shutdown of Xorg can leave you with a blank console
- Shutdown then Restart of domU without restarting dom0 may lead to no
VGA output, however you can still ssh in and startx (Xorg) and it will
produce output


This is my specific dom0 grub2 entry for VGA passthrough

menuentry "Xen 4.0.0-rc8 / Ubuntu 10.4 kernel 2.6.32.10-xen2 pvops IOMMU
SERIAL" {
insmod ext2
search --no-floppy --fs-uuid --set 321dedfa-be48-4b4d-bff0-0e5ecce240fd
multiboot /boot/xen-4.0.0-rc8.gz dummy=dummy iommu=1 dom0_mem=1024M
com1=19200,8n1,0xe480,17 console=com1
module /boot/vmlinuz-2.6.32.10-xen2 dummy=dummy root=/dev/sdc2 ro
xen-pciback.permissive xen-pciback.hide=(01:00.0)(00:1a.0)(00:1d.0)
console=hvc0
module /boot/initrd.img-2.6.32.10-xen2
}


A couple of those addresses are the usb controllers so I make use of
mouse and keyboard. I never could get qemu passthrough to work. Epic
fail on trying to use the nvidia drivers =) The new version tells you
specifically that it does not support the card @ PCI xxxx

All I needed in the hvm config was:

gfx_passthru=1
pci=['01:00.0','00:1a.0','00:1d.0']


I can't think of anything else right now :)

Post by Mark Hurenkamp
Next on my list: try passthrough of the primary graphics card...
Regards,
Mark

Post by Mark Hurenkamp
Hi,
I replaced the i5 with an i7 CPU, and now xen seems to enable vtd properly,
i have yet to test the feature with my board, but xm info now shows the
host : coruscant
release : 2.6.32.11mh18
version : #10 SMP PREEMPT Wed Apr 7 23:57:39 CEST 2010
machine : x86_64
nr_cpus : 8
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 3660
bfebfbff:28100800:00000000:00001b40:0098e3fd:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 7975
free_memory : 2425
node_to_cpu : node0:0-7
node_to_memory : node0:2425
node_to_dma32_mem : node0:1310
max_node_id : 0
xen_major : 4
xen_minor : 0
xen_extra : .0
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : Wed Apr 07 12:38:28 2010 +0100
21091:f28f1ee587c8
xen_commandline : console_to_ring com1=115200,8n1 console=com1
cc_compiler : gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu9)
cc_compile_by : root
cc_compile_domain : karpeer.net
cc_compile_date : Wed Apr 7 23:16:01 CEST 2010
xend_config_format : 4
Regards,
Mark.

Post by Sander Eikelenboom
Hi Mark,
I guess that's your problem, your cpu doesn't seem to support VT-D, see
http://ark.intel.com/Product.aspx?id=42915
Both CPU, chipset and bios most support it ...
--
Sander

_______________________________________________
Xen-devel mailing list
http://lists.xensource.com/xen-devel

[Pasi Kärkkäinen]

Post by listmail
I had no problems passing through my Nvidia 8800gts 512. The nv driver
on linux works fine (as good as it can). There were a few quirks to
note, but I imagine your experience will vary using ATI.

Was this primary or secondary adapter?

-- Pasi

Post by listmail
- It takes a bit before you see VGA output
- The cursor was positioned a higher than normal (sitting at over the
halfway mark for the font)
- Shutdown of Xorg can leave you with a blank console
- Shutdown then Restart of domU without restarting dom0 may lead to no
VGA output, however you can still ssh in and startx (Xorg) and it will
produce output
This is my specific dom0 grub2 entry for VGA passthrough
menuentry "Xen 4.0.0-rc8 / Ubuntu 10.4 kernel 2.6.32.10-xen2 pvops IOMMU
SERIAL" {
insmod ext2
search --no-floppy --fs-uuid --set 321dedfa-be48-4b4d-bff0-0e5ecce240fd
multiboot /boot/xen-4.0.0-rc8.gz dummy=dummy iommu=1 dom0_mem=1024M
com1=19200,8n1,0xe480,17 console=com1
module /boot/vmlinuz-2.6.32.10-xen2 dummy=dummy root=/dev/sdc2 ro
xen-pciback.permissive xen-pciback.hide=(01:00.0)(00:1a.0)(00:1d.0)
console=hvc0
module /boot/initrd.img-2.6.32.10-xen2
}
A couple of those addresses are the usb controllers so I make use of
mouse and keyboard. I never could get qemu passthrough to work. Epic
fail on trying to use the nvidia drivers =) The new version tells you
gfx_passthru=1
pci=['01:00.0','00:1a.0','00:1d.0']
I can't think of anything else right now :)

Post by Mark Hurenkamp
Next on my list: try passthrough of the primary graphics card...
Regards,
Mark

Post by Mark Hurenkamp
Hi,
I replaced the i5 with an i7 CPU, and now xen seems to enable vtd properly,
i have yet to test the feature with my board, but xm info now shows the
host : coruscant
release : 2.6.32.11mh18
version : #10 SMP PREEMPT Wed Apr 7 23:57:39 CEST 2010
machine : x86_64
nr_cpus : 8
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 3660
bfebfbff:28100800:00000000:00001b40:0098e3fd:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 7975
free_memory : 2425
node_to_cpu : node0:0-7
node_to_memory : node0:2425
node_to_dma32_mem : node0:1310
max_node_id : 0
xen_major : 4
xen_minor : 0
xen_extra : .0
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : Wed Apr 07 12:38:28 2010 +0100
21091:f28f1ee587c8
xen_commandline : console_to_ring com1=115200,8n1 console=com1
cc_compiler : gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu9)
cc_compile_by : root
cc_compile_domain : karpeer.net
cc_compile_date : Wed Apr 7 23:16:01 CEST 2010
xend_config_format : 4
Regards,
Mark.

Post by Sander Eikelenboom
Hi Mark,
I guess that's your problem, your cpu doesn't seem to support VT-D, see
http://ark.intel.com/Product.aspx?id=42915
Both CPU, chipset and bios most support it ...
--
Sander

_______________________________________________
Xen-devel mailing list
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
http://lists.xensource.com/xen-devel

[Pasi Kärkkäinen]

Post by Pasi Kärkkäinen

Post by listmail
I had no problems passing through my Nvidia 8800gts 512. The nv driver
on linux works fine (as good as it can). There were a few quirks to
note, but I imagine your experience will vary using ATI.

Was this primary or secondary adapter?

Also please refresh the XenVGAPassthrough wiki page and send all
the information asked there.. so I can add your entry there.

Thanks!

-- Pasi

Post by Pasi Kärkkäinen

Post by listmail
- It takes a bit before you see VGA output
- The cursor was positioned a higher than normal (sitting at over the
halfway mark for the font)
- Shutdown of Xorg can leave you with a blank console
- Shutdown then Restart of domU without restarting dom0 may lead to no
VGA output, however you can still ssh in and startx (Xorg) and it will
produce output
This is my specific dom0 grub2 entry for VGA passthrough
menuentry "Xen 4.0.0-rc8 / Ubuntu 10.4 kernel 2.6.32.10-xen2 pvops IOMMU
SERIAL" {
insmod ext2
search --no-floppy --fs-uuid --set 321dedfa-be48-4b4d-bff0-0e5ecce240fd
multiboot /boot/xen-4.0.0-rc8.gz dummy=dummy iommu=1 dom0_mem=1024M
com1=19200,8n1,0xe480,17 console=com1
module /boot/vmlinuz-2.6.32.10-xen2 dummy=dummy root=/dev/sdc2 ro
xen-pciback.permissive xen-pciback.hide=(01:00.0)(00:1a.0)(00:1d.0)
console=hvc0
module /boot/initrd.img-2.6.32.10-xen2
}
A couple of those addresses are the usb controllers so I make use of
mouse and keyboard. I never could get qemu passthrough to work. Epic
fail on trying to use the nvidia drivers =) The new version tells you
gfx_passthru=1
pci=['01:00.0','00:1a.0','00:1d.0']
I can't think of anything else right now :)

Post by Mark Hurenkamp
Next on my list: try passthrough of the primary graphics card...
Regards,
Mark

Post by Mark Hurenkamp
Hi,
I replaced the i5 with an i7 CPU, and now xen seems to enable vtd properly,
i have yet to test the feature with my board, but xm info now shows the
host : coruscant
release : 2.6.32.11mh18
version : #10 SMP PREEMPT Wed Apr 7 23:57:39 CEST 2010
machine : x86_64
nr_cpus : 8
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 3660
bfebfbff:28100800:00000000:00001b40:0098e3fd:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 7975
free_memory : 2425
node_to_cpu : node0:0-7
node_to_memory : node0:2425
node_to_dma32_mem : node0:1310
max_node_id : 0
xen_major : 4
xen_minor : 0
xen_extra : .0
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : Wed Apr 07 12:38:28 2010 +0100
21091:f28f1ee587c8
xen_commandline : console_to_ring com1=115200,8n1 console=com1
cc_compiler : gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu9)
cc_compile_by : root
cc_compile_domain : karpeer.net
cc_compile_date : Wed Apr 7 23:16:01 CEST 2010
xend_config_format : 4
Regards,
Mark.

Post by Sander Eikelenboom
Hi Mark,
I guess that's your problem, your cpu doesn't seem to support VT-D, see
http://ark.intel.com/Product.aspx?id=42915
Both CPU, chipset and bios most support it ...
--
Sander

_______________________________________________
Xen-devel mailing list
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
http://lists.xensource.com/xen-devel

[listmail]

- Xen version
4.0-rc8

- Dom0 kernel version, and if it's pvops or xenlinux based / Guest OS
and driver version
same kernel for both, via git clone of xen/stable-2.6.32 @ commit
062aaac18de6d875fcc646359179449218f486c5
packaged for install via debian's make-kpkg
used the .config attached here:
http://lists.xensource.com/archives/html/xen-users/2010-03/msg00878.html

-- "lspci" output for the graphics card to get the PCI IDs and model
information
dom0 lspci -vv
<snip>
01:00.0 VGA compatible controller: nVidia Corporation G92 [GeForce 8800
GTS 512] (rev a2)
Subsystem: XFX Pine Group Inc. Device 2332
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 10
Region 0: Memory at f2000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at c0000000 (64-bit, prefetchable) [size=256M]
Region 3: Memory at f0000000 (64-bit, non-prefetchable) [size=32M]
Region 5: I/O ports at cc00 [size=128]
Expansion ROM at f3ee0000 [disabled] [size=128K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<512ns, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency
L0 <256ns, L1 <1us
ClockPM- Suprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Capabilities: [100] Virtual Channel <?>
Capabilities: [128] Power Budgeting <?>
Capabilities: [600] Vendor Specific Information <?>
Kernel modules: nvidiafb
<snip>

- Did you passthru all the PCI IDs or just one?
I'm only aware of one ID -> xen-pciback.hide=(01:00.0)

- Was it primary or secondary graphics adapter?
primary

- Motherboard vendor/model and BIOS version/CPU version
Asus P7P55D Evo
bios 1207 (old)
core i7 860

- How did it work.. did you have some issues?
the nv driver on linux works fine (as good as it can)
it takes a bit before you see VGA output on boot
the console cursor was positioned higher than normal (sitting at over
the halfway mark for the font)
shutdown of Xorg can leave you with a blank console
shutdown then Restart of domU without restarting dom0 may lead to no VGA
output, however you can still ssh in and startx (Xorg) and it will
produce output

Post by Pasi Kärkkäinen

Post by Pasi Kärkkäinen

Post by listmail
I had no problems passing through my Nvidia 8800gts 512. The nv driver
on linux works fine (as good as it can). There were a few quirks to
note, but I imagine your experience will vary using ATI.

Was this primary or secondary adapter?

Also please refresh the XenVGAPassthrough wiki page and send all
the information asked there.. so I can add your entry there.
Thanks!
-- Pasi

[Dietmar Hahn]

Please see my comments inside.

Post by listmail
- Xen version
4.0-rc8
- Dom0 kernel version, and if it's pvops or xenlinux based / Guest OS
and driver version
062aaac18de6d875fcc646359179449218f486c5
packaged for install via debian's make-kpkg
http://lists.xensource.com/archives/html/xen-users/2010-03/msg00878.html
-- "lspci" output for the graphics card to get the PCI IDs and model
information
dom0 lspci -vv
<snip>
01:00.0 VGA compatible controller: nVidia Corporation G92 [GeForce 8800
GTS 512] (rev a2)
Subsystem: XFX Pine Group Inc. Device 2332
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 10
Region 0: Memory at f2000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at c0000000 (64-bit, prefetchable) [size=256M]
Region 3: Memory at f0000000 (64-bit, non-prefetchable) [size=32M]
Region 5: I/O ports at cc00 [size=128]
Expansion ROM at f3ee0000 [disabled] [size=128K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<512ns, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency
L0 <256ns, L1 <1us
ClockPM- Suprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Capabilities: [100] Virtual Channel <?>
Capabilities: [128] Power Budgeting <?>
Capabilities: [600] Vendor Specific Information <?>
Kernel modules: nvidiafb
<snip>
- Did you passthru all the PCI IDs or just one?
I'm only aware of one ID -> xen-pciback.hide=(01:00.0)
- Was it primary or secondary graphics adapter?
primary
- Motherboard vendor/model and BIOS version/CPU version
Asus P7P55D Evo
bios 1207 (old)
core i7 860
- How did it work.. did you have some issues?
the nv driver on linux works fine (as good as it can)
it takes a bit before you see VGA output on boot

Did you see any boot messages of the BIOS / grub
on the console device (with this gfx adapter) before the linux
kernel boot messages and the X11 screen?

Post by listmail
the console cursor was positioned higher than normal (sitting at over
the halfway mark for the font)
shutdown of Xorg can leave you with a blank console
shutdown then Restart of domU without restarting dom0 may lead to no VGA
output, however you can still ssh in and startx (Xorg) and it will
produce output

This is the same behavior as on my laptop where the copied VGA-BIOS not
works in the domU (beside no FLR - see flag FLReset-).

Dietmar.

Post by listmail

Post by Pasi Kärkkäinen

Post by Pasi Kärkkäinen

Post by listmail
I had no problems passing through my Nvidia 8800gts 512. The nv driver
on linux works fine (as good as it can). There were a few quirks to
note, but I imagine your experience will vary using ATI.

Was this primary or secondary adapter?

Also please refresh the XenVGAPassthrough wiki page and send all
the information asked there.. so I can add your entry there.
Thanks!
-- Pasi

_______________________________________________
Xen-devel mailing list
http://lists.xensource.com/xen-devel

[Pasi Kärkkäinen]

Post by Dietmar Hahn

Post by listmail
Capabilities: [78] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<512ns, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-

This is the same behavior as on my laptop where the copied VGA-BIOS not
works in the domU (beside no FLR - see flag FLReset-).

So hmm.. the device will have "FLReset-" in the DevCap section when FLR is supported?
Or does that "-" mean it's not supported?

-- Pasi

[Dietmar Hahn]

Post by Pasi Kärkkäinen

Post by Dietmar Hahn

Post by listmail
Capabilities: [78] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<512ns, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-

This is the same behavior as on my laptop where the copied VGA-BIOS not
works in the domU (beside no FLR - see flag FLReset-).

So hmm.. the device will have "FLReset-" in the DevCap section when FLR is supported?
Or does that "-" mean it's not supported?

As far as I understand the code in lspci right "-" means not supported.

Post by Pasi Kärkkäinen
shutdown of Xorg can leave you with a blank console
shutdown then Restart of domU without restarting dom0 may lead to no VGA
output, however you can still ssh in and startx (Xorg) and it will
produce output
it takes a bit before you see VGA output on boot

I had the same effect. I first saw the linux kernel boot messages, nothing from
BIOS or grub but X11 works fine.
I think the cause is that BIOS and grub are using INT10 (but VGA-BIOS is not
working right). Later linux kernel and X11 are using direct card access.

X11 works but when I start Windows XP no graphic output!

Dietmar.

Post by Pasi Kärkkäinen
-- Pasi

--
Dietmar Hahn
TSP ES&S SWE OS
FUJITSU
Fujitsu Technology Solutions
Domagkstraße 28, D-80807 München, Germany
Tel: [phone]
Email: ***@ts.fujitsu.com
Web: http://ts.fujitsu.com
Company details: http://ts.fujitsu.com/imprint.html

[listmail]

Post by Dietmar Hahn

Post by listmail
- How did it work.. did you have some issues?
the nv driver on linux works fine (as good as it can)
it takes a bit before you see VGA output on boot

Did you see any boot messages of the BIOS / grub
on the console device (with this gfx adapter) before the linux
kernel boot messages and the X11 screen?

No output only appeared some time after the kernel had already started.
I wasn't sure what to expect here. Would there be any more useful
information to provide?

Post by Dietmar Hahn

Post by listmail
the console cursor was positioned higher than normal (sitting at over
the halfway mark for the font)
shutdown of Xorg can leave you with a blank console
shutdown then Restart of domU without restarting dom0 may lead to no VGA
output, however you can still ssh in and startx (Xorg) and it will
produce output

This is the same behavior as on my laptop where the copied VGA-BIOS not
works in the domU (beside no FLR - see flag FLReset-).
Dietmar.

It is copied directly from the card just before domU starts right? I
read something about that the card's bios can change after it first
initialized when the computer starts, is there an option to test using a
file based copy of the bios?

[Dietmar Hahn]

Post by listmail

Post by Dietmar Hahn

Post by listmail
- How did it work.. did you have some issues?
the nv driver on linux works fine (as good as it can)
it takes a bit before you see VGA output on boot

Did you see any boot messages of the BIOS / grub
on the console device (with this gfx adapter) before the linux
kernel boot messages and the X11 screen?

No output only appeared some time after the kernel had already started.
I wasn't sure what to expect here. Would there be any more useful
information to provide?

I only wanted to know whether your vga-bios works well, but it didn't seem
so, otherwise you would see some messages from the bios and if you have
the grub bootloader maybe to select a kernel to boot in the domU.

Post by listmail

Post by Dietmar Hahn

Post by listmail
the console cursor was positioned higher than normal (sitting at over
the halfway mark for the font)
shutdown of Xorg can leave you with a blank console
shutdown then Restart of domU without restarting dom0 may lead to no VGA
output, however you can still ssh in and startx (Xorg) and it will
produce output

This is the same behavior as on my laptop where the copied VGA-BIOS not
works in the domU (beside no FLR - see flag FLReset-).
Dietmar.

It is copied directly from the card just before domU starts right?

No it's copied from the dom0 kernel memory to the domU physical memory.

Post by listmail
I read something about that the card's bios can change after it first
initialized when the computer starts, is there an option to test using a
file based copy of the bios?

To load the vga-bios from a file you have to change the hvmloader (there is
already some code about this) and must compile it on your own.

Dietmar.

[Pasi Kärkkäinen]

Post by listmail
- Xen version
4.0-rc8
- Dom0 kernel version, and if it's pvops or xenlinux based / Guest OS
and driver version
062aaac18de6d875fcc646359179449218f486c5
packaged for install via debian's make-kpkg
http://lists.xensource.com/archives/html/xen-users/2010-03/msg00878.html
-- "lspci" output for the graphics card to get the PCI IDs and model
information
dom0 lspci -vv
<snip>
- Did you passthru all the PCI IDs or just one?
I'm only aware of one ID -> xen-pciback.hide=(01:00.0)

Btw did you try without hiding from dom0?

I've added your info to the wiki page now.

Thanks!

-- Pasi

Post by listmail
- Was it primary or secondary graphics adapter?
primary
- Motherboard vendor/model and BIOS version/CPU version
Asus P7P55D Evo
bios 1207 (old)
core i7 860
- How did it work.. did you have some issues?
the nv driver on linux works fine (as good as it can)
it takes a bit before you see VGA output on boot
the console cursor was positioned higher than normal (sitting at over
the halfway mark for the font)
shutdown of Xorg can leave you with a blank console
shutdown then Restart of domU without restarting dom0 may lead to no VGA
output, however you can still ssh in and startx (Xorg) and it will
produce output

Post by Pasi Kärkkäinen

Post by Pasi Kärkkäinen

Post by listmail
I had no problems passing through my Nvidia 8800gts 512. The nv
driver on linux works fine (as good as it can). There were a few
quirks to note, but I imagine your experience will vary using ATI.

Was this primary or secondary adapter?

Also please refresh the XenVGAPassthrough wiki page and send all
the information asked there.. so I can add your entry there.
Thanks!
-- Pasi

[listmail]

Post by Pasi Kärkkäinen

Post by listmail
- Xen version
4.0-rc8
- Dom0 kernel version, and if it's pvops or xenlinux based / Guest OS
and driver version
062aaac18de6d875fcc646359179449218f486c5
packaged for install via debian's make-kpkg
http://lists.xensource.com/archives/html/xen-users/2010-03/msg00878.html
-- "lspci" output for the graphics card to get the PCI IDs and model
information
dom0 lspci -vv
<snip>
- Did you passthru all the PCI IDs or just one?
I'm only aware of one ID -> xen-pciback.hide=(01:00.0)

Btw did you try without hiding from dom0?
I've added your info to the wiki page now.
Thanks!
-- Pasi

No, I did not try this because I'm not sure of what the process would
be. I suppose I would need a script that gives the VGA card to pciback
first then starts the domU.

[listmail]

Post by Pasi Kärkkäinen

Post by listmail
- Xen version
4.0-rc8
- Dom0 kernel version, and if it's pvops or xenlinux based / Guest OS
and driver version
062aaac18de6d875fcc646359179449218f486c5
packaged for install via debian's make-kpkg
http://lists.xensource.com/archives/html/xen-users/2010-03/msg00878.html
-- "lspci" output for the graphics card to get the PCI IDs and model
information
dom0 lspci -vv
<snip>
- Did you passthru all the PCI IDs or just one?
I'm only aware of one ID -> xen-pciback.hide=(01:00.0)

Btw did you try without hiding from dom0?
I've added your info to the wiki page now.
Thanks!
-- Pasi

Just a small follow up

late binding
# VGA
#01:00.0 0300: 10de:0600 (rev a2)
echo "10de 0600" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo "0000:01:00.0" > /sys/bus/pci/drivers/pci-stub/bind

# USB
#00:1a.0 0c03: 8086:3b3c (rev 05)
echo "8086 3b3c" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:00:1a.0" > /sys/bus/pci/devices/0000:00:1a.0/driver/unbind
echo "0000:00:1a.0" > /sys/bus/pci/drivers/pci-stub/bind

# USB
#00:1d.0 0c03: 8086:3b34 (rev 05)
echo "8086 3b34" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:00:1d.0" > /sys/bus/pci/devices/0000:00:1d.0/driver/unbind
echo "0000:00:1d.0" > /sys/bus/pci/drivers/pci-stub/bind

Results were the same. I noticed a message:
pciback pci-1-0: 22 Couldn't locate PCI device (0000:01:00.0)! perhaps
already in-use?

Maybe this is because I'm doing all of this from script and immediately
starting the hvm. Could use a delay?


I also patched in load BIOS from file and results were the same

The only thing *new* that I've noticed is that when power saving/screen
blank occurred on console, mouse/keyboard (usb controllers passed
through per above) movement from domU would not wake it up. I thought
perhaps it was actually coming in from setterm on dom0. If this is the
case I assume setterm -blank 0 -powersave off -powerdown 0 may handle that.

[Mark Hurenkamp]

Hi,


Since vt-d seems to work ok now on my system, i have been tinkering with
the VGA passthrough today, and alas, got it to work for my windows hvm!

- Xen version used:
4.0.0

- Dom0 kernel version:
pvops from xen/stable-2.6.32.x branch

- lspci for the graphcis card:
01:00.0 VGA compatible controller: ATI Technologies Inc RV710 [Radeon HD
4350] (prog-if 00 [VGA controller])
Subsystem: ASUSTeK Computer Inc. Device 032e
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at b0000000 (64-bit, prefetchable) [size=256M]
Region 2: Memory at fb8e0000 (64-bit, non-prefetchable) [size=64K]
Region 4: I/O ports at a000 [size=256]
Expansion ROM at fb8c0000 [disabled] [size=128K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1
unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0
<64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive-
BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-,
Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance-
ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1
Len=010 <?>
Kernel modules: radeon


- Did you passthru all the PCI IDs or just one?
I passed the graphics, and the audio pci ids both to the hvm.
I also passed an usb card pci id.

- Was it primary or secondary graphics adapter?
It was the primary adapter. I tried the secondary as well, but could not
get it to work.

- Motherboard vendor/model and BIOS version/CPU version
Asus P7P55D-Evo
Bios 1307 (latest)
Core i7 860

- How did it work.. did you have some issues?
After i installed the catalyst driver, and XP rebooted, the graphics
login screen for XP appeared on the monitor (after some time).
I did not see any boot messages though...

[listmail]

Were the patches for secondary passthrough applied to xen-unstable? Are
they available so as to be applied manually to xen-unstable tip?

Post by Pasi Kärkkäinen

Post by listmail
I had no problems passing through my Nvidia 8800gts 512. The nv driver
on linux works fine (as good as it can). There were a few quirks to
note, but I imagine your experience will vary using ATI.

Was this primary or secondary adapter?
-- Pasi

[Pasi Kärkkäinen]

Post by listmail
Were the patches for secondary passthrough applied to xen-unstable? Are
they available so as to be applied manually to xen-unstable tip?

I don't think so.

The patches needed some thinking/designing about how to do it properly.
I'll open another thread about that.

-- Pasi

Post by listmail

Post by Pasi Kärkkäinen

Post by listmail
I had no problems passing through my Nvidia 8800gts 512. The nv
driver on linux works fine (as good as it can). There were a few
quirks to note, but I imagine your experience will vary using ATI.

Was this primary or secondary adapter?
-- Pasi

[listmail]

Post by Mark Hurenkamp

I'm still pretty new to Xen so it could very well be that I've
misunderstood the requirements for certain capabilities. I thought
one of the wiki's had stated that to pass through to an hvm guest you
have to have VT-d though. Is there something specific that I which
can indicate its status after boot?
bios: 1207
cpu: i7 860
grub2: http://pastebin.ca/1856369
xm dmesg: http://pastebin.ca/1856370
xm info: http://pastebin.ca/1856372
hvm cfg: http://pastebin.ca/1856376
I think I compiled the linux kernel at xen/stable git commit
221c4dbf86 given the timestamp in xm info (all done same time frame)

From your earlier mail, it was not clear wether you were indeed using
hvm, and that xen enabled vt-d properly.
From your xm dmesg, and xm info, it is clear that iommu is indeed
enabled for you, something which i've not been able to achieve with my
board.
Could the CPU make a difference here? I'm on a i5 750, and notice you
are using an i7 860....
I'll see if i can do some more experiments with my setup, perhaps
switching back a few changesets to 4.0-rc8.
I'm not sure if the kernel makes a difference, but i'll see if i can
match your configuration as close as possible.
Thanks for your input!
Mark.

according to this chart (and thread) the i5 doesn't support VT-d,
http://i683.photobucket.com/albums/vv199/OvrClkr/i5_chart.png
link source: http://www.tomshardware.com/forum/274342-28-tomshardware

[listmail]

Post by Mark Hurenkamp
Hi,

I'm not a subscriber to xen-devel yet, but did see your post about
the evo and iommu not working. Within the last week I have
installed a xen 4.0rc8 and xen/stable clone kernel 2.6.32.10 and boot
ok with iommu=1 and have also successfully passed through a tv tuner
card tonight which modules did load for. I did have one weird quirk
though. I haven't tested the card in the domu yet. Anyway if you
want to discuss this further then reply back.

Are you sure you are using VT-d here (are you using hvm loader)? I'd
be very interested to know what Bios version you are using, what your
'xm dmesg' and 'xm info' output is (after boot), and the xen
configuration file for your VM.
Cause although my system can boot fine with enabled iommu now, it is
simply switched off by xen at boottime... Did you do something special
to force it to on? Do you have any patches installed?
Btw, passthrough of a tuner device to a pvm domU is not a problem
here, but that does not require VT-d (nor VT-x).
Did you manage to passthrough a device to Windows?
My ultimate goal is to passthrough a (secondary) usb card, and a
(secondary) graphics card, to a Windows hvm.
Mark.

I'm still pretty new to Xen so it could very well be that I've
misunderstood the requirements for certain capabilities. I thought one
of the wiki's had stated that to pass through to an hvm guest you have
to have VT-d though. Is there something specific that I which can
indicate its status after boot?

bios: 1207
cpu: i7 860
grub2: http://pastebin.ca/1856369
xm dmesg: http://pastebin.ca/1856370
xm info: http://pastebin.ca/1856372
hvm cfg: http://pastebin.ca/1856376
I think I compiled the linux kernel at xen/stable git commit 221c4dbf86
given the timestamp in xm info (all done same time frame)