Mirela Simonovic
2017-12-22 17:41:37 UTC
This document contains our design specification for "suspend to RAM"
support for ARM in Xen. It covers the basic suspend to RAM mechanism
based on ARM PSCI standard, that would allow individual guests and
Xen itself to suspend/resume.
We would appreciate your feedback.
Signed-off-by: Mirela Simonovic <***@aggios.com>
---
v2:
-Improved specification according to comments
-Added more implementation details
-Incremented revision number to 1.1
---
docs/misc/arm/suspend-to-ram.txt | 266 +++++++++++++++++++++++++++++++++++++++
1 file changed, 266 insertions(+)
create mode 100644 docs/misc/arm/suspend-to-ram.txt
diff --git a/docs/misc/arm/suspend-to-ram.txt b/docs/misc/arm/suspend-to-ram.txt
new file mode 100644
index 0000000000..6e8f10d1ce
--- /dev/null
+++ b/docs/misc/arm/suspend-to-ram.txt
@@ -0,0 +1,266 @@
+% Suspend to RAM Support in Xen for ARM
+% Revision 1.1
+
+========
+Overview
+========
+
+Suspend to RAM (in the following text 'suspend') for ARM in Xen should be
+coordinated using ARM PSCI standard [1].
+
+Ideally, EL1/2 should suspend in the following order:
+1) Unprivileged guests (DomUs) suspend
+2) Privileged guest (Dom0) suspends
+3) Xen suspends
+
+However, suspending unprivileged guests is not mandatory for suspending
+Dom0 and Xen. System suspend initiated by Dom0 (step 2) is considered to be an
+ultimate decision to suspend the physical machine. Suspending of Xen (step 3)
+is triggered whenever Dom0 completes suspend. Xen suspend leads to the full
+suspend of EL2.
+
+If an unprivileged guest is not suspended at the moment when Dom0 initiates
+its own suspend, the guest will be paused on Xen's suspend and unpaused on
+Xen's resume. That way, a guest which doesn't have power management support
+cannot prevent the physical system from suspending when the decision to suspend
+is made by privileged software (Dom0).
+
+Each guest in the system is responsible for suspending the devices it owns.
+If a guest does not suspend a device, the device will continue to operate as
+it is configured at the moment when the system suspends. If a device triggers
+an interrupt while the physical system is suspended, the system will resume.
+
+It is recommended for an unprivileged guest to participate in power management
+in the following scenario:
+Assume unprivileged guest owns a device which will trigger interrupt at some
+point. This interrupt will wake-up the system. If such a behavior is not wanted,
+coordination between Dom0 and the guest is required in order to inform the guest
+about the intended physical system suspend. Then, the guest will have a chance
+to suspend the device or respond to the request in an abort fashion.
+
+Since this proposal is focused on implementing PSCI-based suspend mechanisms in
+Xen, communication with or among the guests is not covered by this document.
+The order of suspending the guests is assumed to be guaranteed by the software
+running in EL1.
+
+This document covers the following:
+1) Mechanism for suspending/resuming a guest:
+ 1.1) Suspend is initiated by the guest
+ 1.2) Resume is initiated by a device interrupt
+2) Mechanism for pausing/unpausing running guests when Dom0 suspends
+3) Mechanism for suspending/resuming Xen when Dom0 completes suspend
+4) Resuming from any state on a wake-up event (device interrupt):
+ 4.1) Resume DomU on wake-up event when Dom0 is still running
+ 4.2) Resume DomU on wake-up event when Xen is suspended
+ 4.3) Resume Dom0 on wake-up event
+
+Mechanisms enumerated above will allow different kind of policies and
+coordination among guests to be implemented in EL1. That is out of the scope of
+this document.
+
+-----------------
+Suspending Guests
+-----------------
+
+Suspend procedure for a guest consists of the following:
+1) Suspending devices
+2) Suspending non-boot CPUs (based on hotplug/PSCI)
+3) System suspend, performed by the boot CPU
+
+Each guest should suspend the devices it owns just like it would when running
+without Xen.
+
+Guests should suspend their non-boot vCPUs using the hotplug mechanism.
+Virtual CPUs should be put offline using the already implemented PSCI vCPU_OFF
+call (prefix 'v' is added to distinguish PSCI calls made by guests to Xen, which
+affect virtual machines; as opposed to PSCI calls made by Xen to the EL3, which
+can affect power state of the physical machine).
+
+After suspending its non-boot vCPUs a guest should finalize the suspend by
+making the vSYSTEM_SUSPEND PSCI call. The resume address is specified by the
+guest via the vSYSTEM_SUSPEND entry_point_address argument. The vSYSTEM_SUSPEND
+call is currently not implemented in Xen.
+
+It is expected that a guest leaves enabled all interrupts that should wake it
+up. Other interrupts should be disabled by the guest prior to calling
+vSYSTEM_SUSPEND.
+
+After an unprivileged guest suspends, Xen will not suspend. Xen would suspend
+only after the Dom0 completes the system suspend.
+
+--------------
+Suspending Xen
+--------------
+
+Xen should start suspending itself upon receiving the vSYSTEM_SUSPEND call
+from the last running guest (Dom0). At that moment all physical CPUs are still
+online (taking offline a vCPU or suspending a VM does not affect physical CPUs).
+Xen shall now put offline the non-boot pCPUs by making the CPU_OFF PSCI call
+to EL3. The CPU_OFF PSCI function is currently not implemented in Xen.
+
+After putting offline the non-boot cores Xen must save the context and finalize
+suspend by invoking SYSTEM_SUSPEND PSCI call, which is passed to EL3.
+The resume point of Xen is specified by the entry_point_address argument of the
+SYSTEM_SUSPEND call. The SYSTEM_SUSPEND function and context saving is not
+implemented in Xen for ARM today.
+
+------------
+Resuming Xen
+------------
+
+Xen must be resumed prior to any software running in EL1. Starting from the
+resume point, Xen should restore the context and resume Dom0. Dom0 shall always
+be resumed whenever Xen resumes.
+
+---------------
+Resuming Guests
+---------------
+
+Resume of the privileged guest (Dom0) is always following the Xen resume.
+
+An unprivileged guest shall resume once a device it owns triggers a wake-up
+interrupt, regardless of whether Xen was suspended when the wake-up interrupt
+was triggered. If Xen was suspended, it is assumed that Dom0 will be running
+before the DomU guest starts to resume. The synchronization mechanism to
+enforce the assumed condition is TBD.
+
+If the ARM's GIC was powered down after the ARM subsystem suspended, it is
+assumed that Xen needs to restore the GIC interface for a VM prior to handing
+over control to the guest. However, the guest should restore its own context
+upon entering the resume point, just like it would when running without Xen.
+
+===============
+Implementation
+===============
+
+--------
+Overview
+--------
+
+In order to enable the suspend/resume of VMs and Xen itself, the following PSCI
+calls have to be implemented and integrated in Xen:
+1) vSYSTEM_SUSPEND
+2) CPU_OFF
+3) SYSTEM_SUSPEND
+
+In addition, the following have to be implemented:
+* Suspend/resume vCPU (triggered by vSYSTEM_SUSPEND call)
+* Suspend/resume Xen (triggered by vSYSTEM_SUSPEND called by Dom0), including:
+ * Disable wathdog on suspend, enable it on resume
+ * Pause domains on suspend, unpause them on resume
+ * Disable non-boot pCPUs on suspend, enable them on resume
+ * Save/restore of GIC configuration
+ * Suspend/resume timer
+ * Save/restore of EL2 context
+ * Implement resume entry point in Xen, including MMU configuration
+
+Implementation details are provided in the sections below. Function names and
+paths used below are consistent within the document but may not always match the
+names used in future implementation. Existing functions and paths are named as
+in Xen source tree.
+
+-------------------------------------
+Suspend/Resume Implementation Details
+-------------------------------------
+
+PSCI Implementation and Integration
+-----------------------------------
+vSYSTEM_SUSPEND
+---------------
+vSYSTEM_SUSPEND shall be implemented in
+* do_psci_system_suspend() in arch/arm/vpsci.c
+* Code independent from PSCI interface will be added in arch/arm/suspend.c
+
+The implementation shall include the following steps:
+* Suspend the current (calling) vCPU. Consists of 2 major steps:
+1) Reset context of vCPU and save entry point into PC and context ID into X0
+(entry point and context ID are provided via vSYSTEM_SUSPEND arguments)
+2) Block vCPU to ensure that it is not scheduled until it is unblocked by an
+interrupt.
+In step 1) above, the context is reset in order to prepare the vCPU for resume,
+i.e. to save vCPU context that matches reset values as expected by software on
+resume. This doesn't hold for PC and X0, since the PC contains resume entry
+point and X0 contains context ID, as defined by PSCI.
+* If the hardware domain made the call trigger Xen suspend, i.e.
+ call machine_suspend() which will be implemented in arch/arm/suspend.c
+ (similar as the machine_restart() is implemented in arch/arm/shutdown.c)
+
+The function do_psci_system_suspend() shall be called from
+* do_trap_psci() in arch/arm/traps.c
+
+CPU_OFF (physical CPUs)
+-----------------------
+The CPU_OFF function shall be implemented in
+* call_psci_cpu_off() in arch/arm/psci.c
+
+The implementation shall consist just of making the SMC call to EL3.
+
+This function needs to be called when Xen generic code disables a non-boot CPU.
+When a CPU is disabled it will loop forever in while loop (stop_cpu() function
+which is already implemented in xen/arch/arm/smpboot.c). Call to
+call_psci_cpu_off() shall be made before the CPU enters infinite loop.
+
+SYSTEM_SUSPEND (physical)
+-------------------------
+The SYSTEM_SUSPEND function shall be implemented in
+* call_psci_system_suspend() in arch/arm/psci.c
+
+The implementation shall consist just of making the SMC call to EL3. The
+entry_point_address argument of the SMC call needs to be an ARM architecture
+resume address, which shall be implemented, e.g. as hyp_resume() in
+arch/arm/arm64/entry.S. The call_psci_system_suspend() function does not return.
+On the resume, the execution flow continues from hyp_resume.
+
+The function needs to be called from machine_suspend() to finalize the suspend
+procedure.
+
+------------------
+Additional Changes
+------------------
+
+Suspend Flow
+------------
+The suspend procedure shall be implemented in
+* machine_suspend() in arch/arm/suspend.c
+
+The implementation shall include the following steps:
+* Move the execution to boot pCPU
+* Set the system_state variable to SYS_STATE_suspend
+* Disable watchdog
+* Freeze domains by calling domain_pause() for each domain
+* Disable non-boot CPUs by calling disable_nonboot_cpus()
+* Disable interrupts
+* Suspend timer
+* Save GIC context. Shall be implemented in arch/arm/gic.c,
+ include/asm-arm/gic.h and arch/arm/gic-v2.c (only GICv2 will be supported).
+* Save CPU context. This shall be implemented in assembly, in hyp_suspend()
+ in arch/arm/arm64/entry.S. The context consists of callee-saved general
+ purpose registers, as well as few system registers. Context of registers shall
+ be saved in a statically allocated structure.
+* Finalize the suspend by calling call_psci_system_suspend()
+
+Resume Flow
+------------
+The resume entry point shall be implemented in
+* hyp_resume() in arch/arm/arm64/entry.S
+The very beginning of the resume procedure has to be implemented in assembly.
+It shall contain the following:
+* Enable the MMU so that the structure containing CPU context which was saved on
+suspend can be accessed
+* Restore CPU context (to match the values saved on suspend) and return into C
+* Set the system_state variable to SYS_STATE_resume
+* Restore GIC context
+* Resume timer
+* Enable interrupts
+* Enable non-boot CPUs by calling enable_nonboot_cpus()
+* Thaw domains by calling domain_unpause() for each domain
+* Enable watchdog
+* Set the system_state variable to SYS_STATE_active
+* Resume Dom0
+
+==========
+References
+==========
+
+[1] Power State Coordination Interface (ARM):
+http://infocenter.arm.com/help/topic/com.arm.doc.den0022d/Power_State_Coordination_Interface_PDD_v1_1_DEN0022D.pdf
support for ARM in Xen. It covers the basic suspend to RAM mechanism
based on ARM PSCI standard, that would allow individual guests and
Xen itself to suspend/resume.
We would appreciate your feedback.
Signed-off-by: Mirela Simonovic <***@aggios.com>
---
v2:
-Improved specification according to comments
-Added more implementation details
-Incremented revision number to 1.1
---
docs/misc/arm/suspend-to-ram.txt | 266 +++++++++++++++++++++++++++++++++++++++
1 file changed, 266 insertions(+)
create mode 100644 docs/misc/arm/suspend-to-ram.txt
diff --git a/docs/misc/arm/suspend-to-ram.txt b/docs/misc/arm/suspend-to-ram.txt
new file mode 100644
index 0000000000..6e8f10d1ce
--- /dev/null
+++ b/docs/misc/arm/suspend-to-ram.txt
@@ -0,0 +1,266 @@
+% Suspend to RAM Support in Xen for ARM
+% Revision 1.1
+
+========
+Overview
+========
+
+Suspend to RAM (in the following text 'suspend') for ARM in Xen should be
+coordinated using ARM PSCI standard [1].
+
+Ideally, EL1/2 should suspend in the following order:
+1) Unprivileged guests (DomUs) suspend
+2) Privileged guest (Dom0) suspends
+3) Xen suspends
+
+However, suspending unprivileged guests is not mandatory for suspending
+Dom0 and Xen. System suspend initiated by Dom0 (step 2) is considered to be an
+ultimate decision to suspend the physical machine. Suspending of Xen (step 3)
+is triggered whenever Dom0 completes suspend. Xen suspend leads to the full
+suspend of EL2.
+
+If an unprivileged guest is not suspended at the moment when Dom0 initiates
+its own suspend, the guest will be paused on Xen's suspend and unpaused on
+Xen's resume. That way, a guest which doesn't have power management support
+cannot prevent the physical system from suspending when the decision to suspend
+is made by privileged software (Dom0).
+
+Each guest in the system is responsible for suspending the devices it owns.
+If a guest does not suspend a device, the device will continue to operate as
+it is configured at the moment when the system suspends. If a device triggers
+an interrupt while the physical system is suspended, the system will resume.
+
+It is recommended for an unprivileged guest to participate in power management
+in the following scenario:
+Assume unprivileged guest owns a device which will trigger interrupt at some
+point. This interrupt will wake-up the system. If such a behavior is not wanted,
+coordination between Dom0 and the guest is required in order to inform the guest
+about the intended physical system suspend. Then, the guest will have a chance
+to suspend the device or respond to the request in an abort fashion.
+
+Since this proposal is focused on implementing PSCI-based suspend mechanisms in
+Xen, communication with or among the guests is not covered by this document.
+The order of suspending the guests is assumed to be guaranteed by the software
+running in EL1.
+
+This document covers the following:
+1) Mechanism for suspending/resuming a guest:
+ 1.1) Suspend is initiated by the guest
+ 1.2) Resume is initiated by a device interrupt
+2) Mechanism for pausing/unpausing running guests when Dom0 suspends
+3) Mechanism for suspending/resuming Xen when Dom0 completes suspend
+4) Resuming from any state on a wake-up event (device interrupt):
+ 4.1) Resume DomU on wake-up event when Dom0 is still running
+ 4.2) Resume DomU on wake-up event when Xen is suspended
+ 4.3) Resume Dom0 on wake-up event
+
+Mechanisms enumerated above will allow different kind of policies and
+coordination among guests to be implemented in EL1. That is out of the scope of
+this document.
+
+-----------------
+Suspending Guests
+-----------------
+
+Suspend procedure for a guest consists of the following:
+1) Suspending devices
+2) Suspending non-boot CPUs (based on hotplug/PSCI)
+3) System suspend, performed by the boot CPU
+
+Each guest should suspend the devices it owns just like it would when running
+without Xen.
+
+Guests should suspend their non-boot vCPUs using the hotplug mechanism.
+Virtual CPUs should be put offline using the already implemented PSCI vCPU_OFF
+call (prefix 'v' is added to distinguish PSCI calls made by guests to Xen, which
+affect virtual machines; as opposed to PSCI calls made by Xen to the EL3, which
+can affect power state of the physical machine).
+
+After suspending its non-boot vCPUs a guest should finalize the suspend by
+making the vSYSTEM_SUSPEND PSCI call. The resume address is specified by the
+guest via the vSYSTEM_SUSPEND entry_point_address argument. The vSYSTEM_SUSPEND
+call is currently not implemented in Xen.
+
+It is expected that a guest leaves enabled all interrupts that should wake it
+up. Other interrupts should be disabled by the guest prior to calling
+vSYSTEM_SUSPEND.
+
+After an unprivileged guest suspends, Xen will not suspend. Xen would suspend
+only after the Dom0 completes the system suspend.
+
+--------------
+Suspending Xen
+--------------
+
+Xen should start suspending itself upon receiving the vSYSTEM_SUSPEND call
+from the last running guest (Dom0). At that moment all physical CPUs are still
+online (taking offline a vCPU or suspending a VM does not affect physical CPUs).
+Xen shall now put offline the non-boot pCPUs by making the CPU_OFF PSCI call
+to EL3. The CPU_OFF PSCI function is currently not implemented in Xen.
+
+After putting offline the non-boot cores Xen must save the context and finalize
+suspend by invoking SYSTEM_SUSPEND PSCI call, which is passed to EL3.
+The resume point of Xen is specified by the entry_point_address argument of the
+SYSTEM_SUSPEND call. The SYSTEM_SUSPEND function and context saving is not
+implemented in Xen for ARM today.
+
+------------
+Resuming Xen
+------------
+
+Xen must be resumed prior to any software running in EL1. Starting from the
+resume point, Xen should restore the context and resume Dom0. Dom0 shall always
+be resumed whenever Xen resumes.
+
+---------------
+Resuming Guests
+---------------
+
+Resume of the privileged guest (Dom0) is always following the Xen resume.
+
+An unprivileged guest shall resume once a device it owns triggers a wake-up
+interrupt, regardless of whether Xen was suspended when the wake-up interrupt
+was triggered. If Xen was suspended, it is assumed that Dom0 will be running
+before the DomU guest starts to resume. The synchronization mechanism to
+enforce the assumed condition is TBD.
+
+If the ARM's GIC was powered down after the ARM subsystem suspended, it is
+assumed that Xen needs to restore the GIC interface for a VM prior to handing
+over control to the guest. However, the guest should restore its own context
+upon entering the resume point, just like it would when running without Xen.
+
+===============
+Implementation
+===============
+
+--------
+Overview
+--------
+
+In order to enable the suspend/resume of VMs and Xen itself, the following PSCI
+calls have to be implemented and integrated in Xen:
+1) vSYSTEM_SUSPEND
+2) CPU_OFF
+3) SYSTEM_SUSPEND
+
+In addition, the following have to be implemented:
+* Suspend/resume vCPU (triggered by vSYSTEM_SUSPEND call)
+* Suspend/resume Xen (triggered by vSYSTEM_SUSPEND called by Dom0), including:
+ * Disable wathdog on suspend, enable it on resume
+ * Pause domains on suspend, unpause them on resume
+ * Disable non-boot pCPUs on suspend, enable them on resume
+ * Save/restore of GIC configuration
+ * Suspend/resume timer
+ * Save/restore of EL2 context
+ * Implement resume entry point in Xen, including MMU configuration
+
+Implementation details are provided in the sections below. Function names and
+paths used below are consistent within the document but may not always match the
+names used in future implementation. Existing functions and paths are named as
+in Xen source tree.
+
+-------------------------------------
+Suspend/Resume Implementation Details
+-------------------------------------
+
+PSCI Implementation and Integration
+-----------------------------------
+vSYSTEM_SUSPEND
+---------------
+vSYSTEM_SUSPEND shall be implemented in
+* do_psci_system_suspend() in arch/arm/vpsci.c
+* Code independent from PSCI interface will be added in arch/arm/suspend.c
+
+The implementation shall include the following steps:
+* Suspend the current (calling) vCPU. Consists of 2 major steps:
+1) Reset context of vCPU and save entry point into PC and context ID into X0
+(entry point and context ID are provided via vSYSTEM_SUSPEND arguments)
+2) Block vCPU to ensure that it is not scheduled until it is unblocked by an
+interrupt.
+In step 1) above, the context is reset in order to prepare the vCPU for resume,
+i.e. to save vCPU context that matches reset values as expected by software on
+resume. This doesn't hold for PC and X0, since the PC contains resume entry
+point and X0 contains context ID, as defined by PSCI.
+* If the hardware domain made the call trigger Xen suspend, i.e.
+ call machine_suspend() which will be implemented in arch/arm/suspend.c
+ (similar as the machine_restart() is implemented in arch/arm/shutdown.c)
+
+The function do_psci_system_suspend() shall be called from
+* do_trap_psci() in arch/arm/traps.c
+
+CPU_OFF (physical CPUs)
+-----------------------
+The CPU_OFF function shall be implemented in
+* call_psci_cpu_off() in arch/arm/psci.c
+
+The implementation shall consist just of making the SMC call to EL3.
+
+This function needs to be called when Xen generic code disables a non-boot CPU.
+When a CPU is disabled it will loop forever in while loop (stop_cpu() function
+which is already implemented in xen/arch/arm/smpboot.c). Call to
+call_psci_cpu_off() shall be made before the CPU enters infinite loop.
+
+SYSTEM_SUSPEND (physical)
+-------------------------
+The SYSTEM_SUSPEND function shall be implemented in
+* call_psci_system_suspend() in arch/arm/psci.c
+
+The implementation shall consist just of making the SMC call to EL3. The
+entry_point_address argument of the SMC call needs to be an ARM architecture
+resume address, which shall be implemented, e.g. as hyp_resume() in
+arch/arm/arm64/entry.S. The call_psci_system_suspend() function does not return.
+On the resume, the execution flow continues from hyp_resume.
+
+The function needs to be called from machine_suspend() to finalize the suspend
+procedure.
+
+------------------
+Additional Changes
+------------------
+
+Suspend Flow
+------------
+The suspend procedure shall be implemented in
+* machine_suspend() in arch/arm/suspend.c
+
+The implementation shall include the following steps:
+* Move the execution to boot pCPU
+* Set the system_state variable to SYS_STATE_suspend
+* Disable watchdog
+* Freeze domains by calling domain_pause() for each domain
+* Disable non-boot CPUs by calling disable_nonboot_cpus()
+* Disable interrupts
+* Suspend timer
+* Save GIC context. Shall be implemented in arch/arm/gic.c,
+ include/asm-arm/gic.h and arch/arm/gic-v2.c (only GICv2 will be supported).
+* Save CPU context. This shall be implemented in assembly, in hyp_suspend()
+ in arch/arm/arm64/entry.S. The context consists of callee-saved general
+ purpose registers, as well as few system registers. Context of registers shall
+ be saved in a statically allocated structure.
+* Finalize the suspend by calling call_psci_system_suspend()
+
+Resume Flow
+------------
+The resume entry point shall be implemented in
+* hyp_resume() in arch/arm/arm64/entry.S
+The very beginning of the resume procedure has to be implemented in assembly.
+It shall contain the following:
+* Enable the MMU so that the structure containing CPU context which was saved on
+suspend can be accessed
+* Restore CPU context (to match the values saved on suspend) and return into C
+* Set the system_state variable to SYS_STATE_resume
+* Restore GIC context
+* Resume timer
+* Enable interrupts
+* Enable non-boot CPUs by calling enable_nonboot_cpus()
+* Thaw domains by calling domain_unpause() for each domain
+* Enable watchdog
+* Set the system_state variable to SYS_STATE_active
+* Resume Dom0
+
+==========
+References
+==========
+
+[1] Power State Coordination Interface (ARM):
+http://infocenter.arm.com/help/topic/com.arm.doc.den0022d/Power_State_Coordination_Interface_PDD_v1_1_DEN0022D.pdf
--
2.13.0
2.13.0