[net-next-2.6.git] / Documentation / kvm / ppc-pv.txt

The PPC KVM paravirtual interface
=================================

The basic execution principle by which KVM on PowerPC works is to run all kernel
space code in PR=1 which is user space. This way we trap all privileged
instructions and can emulate them accordingly.

Unfortunately that is also the downfall. There are quite some privileged
instructions that needlessly return us to the hypervisor even though they
could be handled differently.

This is what the PPC PV interface helps with. It takes privileged instructions
and transforms them into unprivileged ones with some help from the hypervisor.
This cuts down virtualization costs by about 50% on some of my benchmarks.

The code for that interface can be found in arch/powerpc/kernel/kvm*

Querying for existence
======================

To find out if we're running on KVM or not, we leverage the device tree. When
Linux is running on KVM, a node /hypervisor exists. That node contains a
compatible property with the value "linux,kvm".

Once you determined you're running under a PV capable KVM, you can now use
hypercalls as described below.

KVM hypercalls
==============

Inside the device tree's /hypervisor node there's a property called
'hypercall-instructions'. This property contains at most 4 opcodes that make
up the hypercall. To call a hypercall, just call these instructions.

The parameters are as follows:

	Register	IN			OUT

	r0		-			volatile
	r3		1st parameter		Return code
	r4		2nd parameter		1st output value
	r5		3rd parameter		2nd output value
	r6		4th parameter		3rd output value
	r7		5th parameter		4th output value
	r8		6th parameter		5th output value
	r9		7th parameter		6th output value
	r10		8th parameter		7th output value
	r11		hypercall number	8th output value
	r12		-			volatile

Hypercall definitions are shared in generic code, so the same hypercall numbers
apply for x86 and powerpc alike with the exception that each KVM hypercall
also needs to be ORed with the KVM vendor code which is (42 << 16).

Return codes can be as follows:

	Code		Meaning

	0		Success
	12		Hypercall not implemented
	<0		Error

The magic page
==============

To enable communication between the hypervisor and guest there is a new shared
page that contains parts of supervisor visible register state. The guest can
map this shared page using the KVM hypercall KVM_HC_PPC_MAP_MAGIC_PAGE.

With this hypercall issued the guest always gets the magic page mapped at the
desired location in effective and physical address space. For now, we always
map the page to -4096. This way we can access it using absolute load and store
functions. The following instruction reads the first field of the magic page:

	ld	rX, -4096(0)

The interface is designed to be extensible should there be need later to add
additional registers to the magic page. If you add fields to the magic page,
also define a new hypercall feature to indicate that the host can give you more
registers. Only if the host supports the additional features, make use of them.

The magic page has the following layout as described in
arch/powerpc/include/asm/kvm_para.h:

struct kvm_vcpu_arch_shared {
	__u64 scratch1;
	__u64 scratch2;
	__u64 scratch3;
	__u64 critical;		/* Guest may not get interrupts if == r1 */
	__u64 sprg0;
	__u64 sprg1;
	__u64 sprg2;
	__u64 sprg3;
	__u64 srr0;
	__u64 srr1;
	__u64 dar;
	__u64 msr;
	__u32 dsisr;
	__u32 int_pending;	/* Tells the guest if we have an interrupt */
};

Additions to the page must only occur at the end. Struct fields are always 32
or 64 bit aligned, depending on them being 32 or 64 bit wide respectively.

Magic page features
===================

When mapping the magic page using the KVM hypercall KVM_HC_PPC_MAP_MAGIC_PAGE,
a second return value is passed to the guest. This second return value contains
a bitmap of available features inside the magic page.

The following enhancements to the magic page are currently available:

  KVM_MAGIC_FEAT_SR		Maps SR registers r/w in the magic page

For enhanced features in the magic page, please check for the existence of the
feature before using them!

MSR bits
========

The MSR contains bits that require hypervisor intervention and bits that do
not require direct hypervisor intervention because they only get interpreted
when entering the guest or don't have any impact on the hypervisor's behavior.

The following bits are safe to be set inside the guest:

  MSR_EE
  MSR_RI
  MSR_CR
  MSR_ME

If any other bit changes in the MSR, please still use mtmsr(d).

Patched instructions
====================

The "ld" and "std" instructions are transormed to "lwz" and "stw" instructions
respectively on 32 bit systems with an added offset of 4 to accomodate for big
endianness.

The following is a list of mapping the Linux kernel performs when running as
guest. Implementing any of those mappings is optional, as the instruction traps
also act on the shared page. So calling privileged instructions still works as
before.

From			To
====			==

mfmsr	rX		ld	rX, magic_page->msr
mfsprg	rX, 0		ld	rX, magic_page->sprg0
mfsprg	rX, 1		ld	rX, magic_page->sprg1
mfsprg	rX, 2		ld	rX, magic_page->sprg2
mfsprg	rX, 3		ld	rX, magic_page->sprg3
mfsrr0	rX		ld	rX, magic_page->srr0
mfsrr1	rX		ld	rX, magic_page->srr1
mfdar	rX		ld	rX, magic_page->dar
mfdsisr	rX		lwz	rX, magic_page->dsisr

mtmsr	rX		std	rX, magic_page->msr
mtsprg	0, rX		std	rX, magic_page->sprg0
mtsprg	1, rX		std	rX, magic_page->sprg1
mtsprg	2, rX		std	rX, magic_page->sprg2
mtsprg	3, rX		std	rX, magic_page->sprg3
mtsrr0	rX		std	rX, magic_page->srr0
mtsrr1	rX		std	rX, magic_page->srr1
mtdar	rX		std	rX, magic_page->dar
mtdsisr	rX		stw	rX, magic_page->dsisr

tlbsync			nop

mtmsrd	rX, 0		b	<special mtmsr section>
mtmsr	rX		b	<special mtmsr section>

mtmsrd	rX, 1		b	<special mtmsrd section>

[Book3S only]
mtsrin	rX, rY		b	<special mtsrin section>

[BookE only]
wrteei	[0|1]		b	<special wrteei section>


Some instructions require more logic to determine what's going on than a load
or store instruction can deliver. To enable patching of those, we keep some
RAM around where we can live translate instructions to. What happens is the
following:

	1) copy emulation code to memory
	2) patch that code to fit the emulated instruction
	3) patch that code to return to the original pc + 4
	4) patch the original instruction to branch to the new code

That way we can inject an arbitrary amount of code as replacement for a single
instruction. This allows us to check for pending interrupts when setting EE=1
for example.
Commit	Line	Data
d7d3c2ea AG	1	The PPC KVM paravirtual interface
	2	=================================
	3
	4	The basic execution principle by which KVM on PowerPC works is to run all kernel
	5	space code in PR=1 which is user space. This way we trap all privileged
	6	instructions and can emulate them accordingly.
	7
	8	Unfortunately that is also the downfall. There are quite some privileged
	9	instructions that needlessly return us to the hypervisor even though they
	10	could be handled differently.
	11
	12	This is what the PPC PV interface helps with. It takes privileged instructions
	13	and transforms them into unprivileged ones with some help from the hypervisor.
	14	This cuts down virtualization costs by about 50% on some of my benchmarks.
	15
	16	The code for that interface can be found in arch/powerpc/kernel/kvm*
	17
	18	Querying for existence
	19	======================
	20
	21	To find out if we're running on KVM or not, we leverage the device tree. When
	22	Linux is running on KVM, a node /hypervisor exists. That node contains a
	23	compatible property with the value "linux,kvm".
	24
	25	Once you determined you're running under a PV capable KVM, you can now use
	26	hypercalls as described below.
	27
	28	KVM hypercalls
	29	==============
	30
	31	Inside the device tree's /hypervisor node there's a property called
	32	'hypercall-instructions'. This property contains at most 4 opcodes that make
	33	up the hypercall. To call a hypercall, just call these instructions.
	34
	35	The parameters are as follows:
	36
	37	Register IN OUT
	38
	39	r0 - volatile
	40	r3 1st parameter Return code
	41	r4 2nd parameter 1st output value
	42	r5 3rd parameter 2nd output value
	43	r6 4th parameter 3rd output value
	44	r7 5th parameter 4th output value
	45	r8 6th parameter 5th output value
	46	r9 7th parameter 6th output value
	47	r10 8th parameter 7th output value
	48	r11 hypercall number 8th output value
	49	r12 - volatile
	50
	51	Hypercall definitions are shared in generic code, so the same hypercall numbers
	52	apply for x86 and powerpc alike with the exception that each KVM hypercall
	53	also needs to be ORed with the KVM vendor code which is (42 << 16).
	54
	55	Return codes can be as follows:
	56
	57	Code Meaning
	58
	59	0 Success
	60	12 Hypercall not implemented
	61	<0 Error
	62
	63	The magic page
	64	==============
65
66	To enable communication between the hypervisor and guest there is a new shared
67	page that contains parts of supervisor visible register state. The guest can
68	map this shared page using the KVM hypercall KVM_HC_PPC_MAP_MAGIC_PAGE.
69
70	With this hypercall issued the guest always gets the magic page mapped at the
71	desired location in effective and physical address space. For now, we always
72	map the page to -4096. This way we can access it using absolute load and store
73	functions. The following instruction reads the first field of the magic page:
74
75	ld rX, -4096(0)
76
77	The interface is designed to be extensible should there be need later to add
78	additional registers to the magic page. If you add fields to the magic page,
79	also define a new hypercall feature to indicate that the host can give you more
80	registers. Only if the host supports the additional features, make use of them.
81
82	The magic page has the following layout as described in
83	arch/powerpc/include/asm/kvm_para.h:
84
85	struct kvm_vcpu_arch_shared {
86	__u64 scratch1;
87	__u64 scratch2;
88	__u64 scratch3;
89	__u64 critical; /* Guest may not get interrupts if == r1 */
90	__u64 sprg0;
91	__u64 sprg1;
92	__u64 sprg2;
93	__u64 sprg3;
94	__u64 srr0;
95	__u64 srr1;
96	__u64 dar;
97	__u64 msr;
98	__u32 dsisr;
99	__u32 int_pending; /* Tells the guest if we have an interrupt */
100	};
101
102	Additions to the page must only occur at the end. Struct fields are always 32
103	or 64 bit aligned, depending on them being 32 or 64 bit wide respectively.
104
d1e87c7e AG	105	Magic page features
	106	===================
	107
	108	When mapping the magic page using the KVM hypercall KVM_HC_PPC_MAP_MAGIC_PAGE,
	109	a second return value is passed to the guest. This second return value contains
	110	a bitmap of available features inside the magic page.
	111
	112	The following enhancements to the magic page are currently available:
	113
	114	KVM_MAGIC_FEAT_SR Maps SR registers r/w in the magic page
	115
	116	For enhanced features in the magic page, please check for the existence of the
	117	feature before using them!
	118
d7d3c2ea AG	119	MSR bits
	120	========
	121
	122	The MSR contains bits that require hypervisor intervention and bits that do
	123	not require direct hypervisor intervention because they only get interpreted
	124	when entering the guest or don't have any impact on the hypervisor's behavior.
	125
	126	The following bits are safe to be set inside the guest:
	127
	128	MSR_EE
	129	MSR_RI
	130	MSR_CR
	131	MSR_ME
	132
	133	If any other bit changes in the MSR, please still use mtmsr(d).
	134
	135	Patched instructions
	136	====================
	137
	138	The "ld" and "std" instructions are transormed to "lwz" and "stw" instructions
	139	respectively on 32 bit systems with an added offset of 4 to accomodate for big
	140	endianness.
	141
	142	The following is a list of mapping the Linux kernel performs when running as
	143	guest. Implementing any of those mappings is optional, as the instruction traps
	144	also act on the shared page. So calling privileged instructions still works as
	145	before.
	146
	147	From To
	148	==== ==
	149
	150	mfmsr rX ld rX, magic_page->msr
	151	mfsprg rX, 0 ld rX, magic_page->sprg0
	152	mfsprg rX, 1 ld rX, magic_page->sprg1
	153	mfsprg rX, 2 ld rX, magic_page->sprg2
	154	mfsprg rX, 3 ld rX, magic_page->sprg3
	155	mfsrr0 rX ld rX, magic_page->srr0
	156	mfsrr1 rX ld rX, magic_page->srr1
	157	mfdar rX ld rX, magic_page->dar
	158	mfdsisr rX lwz rX, magic_page->dsisr
	159
	160	mtmsr rX std rX, magic_page->msr
	161	mtsprg 0, rX std rX, magic_page->sprg0
	162	mtsprg 1, rX std rX, magic_page->sprg1
	163	mtsprg 2, rX std rX, magic_page->sprg2
	164	mtsprg 3, rX std rX, magic_page->sprg3
	165	mtsrr0 rX std rX, magic_page->srr0
	166	mtsrr1 rX std rX, magic_page->srr1
	167	mtdar rX std rX, magic_page->dar
	168	mtdsisr rX stw rX, magic_page->dsisr
	169
	170	tlbsync nop
	171
	172	mtmsrd rX, 0 b <special mtmsr section>
	173	mtmsr rX b <special mtmsr section>
	174
	175	mtmsrd rX, 1 b <special mtmsrd section>
	176
cbe487fa AG	177	[Book3S only]
	178	mtsrin rX, rY b <special mtsrin section>
	179
d7d3c2ea AG	180	[BookE only]
	181	wrteei [0\|1] b <special wrteei section>
	182
	183
	184	Some instructions require more logic to determine what's going on than a load
	185	or store instruction can deliver. To enable patching of those, we keep some
	186	RAM around where we can live translate instructions to. What happens is the
	187	following:
	188
	189	1) copy emulation code to memory
	190	2) patch that code to fit the emulated instruction
	191	3) patch that code to return to the original pc + 4
	192	4) patch the original instruction to branch to the new code
	193
	194	That way we can inject an arbitrary amount of code as replacement for a single
	195	instruction. This allows us to check for pending interrupts when setting EE=1
	196	for example.