Freezer: make kernel threads nonfreezable by default

Freezing of tasks

	(C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL

I. What is the freezing of tasks?

The freezing of tasks is a mechanism by which user space processes and some

kernel threads are controlled during hibernation or system-wide suspend (on some

architectures).

II. How does it work?

There are four per-task flags used for that, PF_NOFREEZE, PF_FROZEN, TIF_FREEZE

and PF_FREEZER_SKIP (the last one is auxiliary).  The tasks that have

PF_NOFREEZE unset (all user space processes and some kernel threads) are

regarded as 'freezable' and treated in a special way before the system enters a

suspend state as well as before a hibernation image is created (in what follows

we only consider hibernation, but the description also applies to suspend).

Namely, as the first step of the hibernation procedure the function

freeze_processes() (defined in kernel/power/process.c) is called.  It executes

try_to_freeze_tasks() that sets TIF_FREEZE for all of the freezable tasks and

sends a fake signal to each of them.  A task that receives such a signal and has

TIF_FREEZE set, should react to it by calling the refrigerator() function

(defined in kernel/power/process.c), which sets the task's PF_FROZEN flag,

changes its state to TASK_UNINTERRUPTIBLE and makes it loop until PF_FROZEN is

cleared for it.  Then, we say that the task is 'frozen' and therefore the set of

functions handling this mechanism is called 'the freezer' (these functions are

defined in kernel/power/process.c and include/linux/freezer.h).  User space

processes are generally frozen before kernel threads.

It is not recommended to call refrigerator() directly.  Instead, it is

recommended to use the try_to_freeze() function (defined in

include/linux/freezer.h), that checks the task's TIF_FREEZE flag and makes the

task enter refrigerator() if the flag is set.

For user space processes try_to_freeze() is called automatically from the

signal-handling code, but the freezable kernel threads need to call it

explicitly in suitable places.  The code to do this may look like the following:

	do {

		hub_events();

		wait_event_interruptible(khubd_wait,

					!list_empty(&hub_event_list));

		try_to_freeze();

	} while (!signal_pending(current));

(from drivers/usb/core/hub.c::hub_thread()).

If a freezable kernel thread fails to call try_to_freeze() after the freezer has

set TIF_FREEZE for it, the freezing of tasks will fail and the entire

hibernation operation will be cancelled.  For this reason, freezable kernel

threads must call try_to_freeze() somewhere.

After the system memory state has been restored from a hibernation image and

devices have been reinitialized, the function thaw_processes() is called in

order to clear the PF_FROZEN flag for each frozen task.  Then, the tasks that

have been frozen leave refrigerator() and continue running.

III. Which kernel threads are freezable?

Kernel threads are not freezable by default.  However, a kernel thread may clear

PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE

directly is strongly discouraged).  From this point it is regarded as freezable

and must call try_to_freeze() in a suitable place.

IV. Why do we do that?

Generally speaking, there is a couple of reasons to use the freezing of tasks:

1. The principal reason is to prevent filesystems from being damaged after

hibernation.  At the moment we have no simple means of checkpointing

filesystems, so if there are any modifications made to filesystem data and/or

metadata on disks, we cannot bring them back to the state from before the

modifications.  At the same time each hibernation image contains some

filesystem-related information that must be consistent with the state of the

on-disk data and metadata after the system memory state has been restored from

the image (otherwise the filesystems will be damaged in a nasty way, usually

making them almost impossible to repair).  We therefore freeze tasks that might

cause the on-disk filesystems' data and metadata to be modified after the

hibernation image has been created and before the system is finally powered off.

The majority of these are user space processes, but if any of the kernel threads

may cause something like this to happen, they have to be freezable.

2. The second reason is to prevent user space processes and some kernel threads

from interfering with the suspending and resuming of devices.  A user space

process running on a second CPU while we are suspending devices may, for

example, be troublesome and without the freezing of tasks we would need some

safeguards against race conditions that might occur in such a case.

Although Linus Torvalds doesn't like the freezing of tasks, he said this in one

of the discussions on LKML (http://lkml.org/lkml/2007/4/27/608):

"RJW:> Why we freeze tasks at all or why we freeze kernel threads?

Linus: In many ways, 'at all'.

I _do_ realize the IO request queue issues, and that we cannot actually do

s2ram with some devices in the middle of a DMA.  So we want to be able to

avoid *that*, there's no question about that.  And I suspect that stopping

user threads and then waiting for a sync is practically one of the easier

ways to do so.

So in practice, the 'at all' may become a 'why freeze kernel threads?' and

freezing user threads I don't find really objectionable."

Still, there are kernel threads that may want to be freezable.  For example, if

a kernel that belongs to a device driver accesses the device directly, it in

principle needs to know when the device is suspended, so that it doesn't try to

access it at that time.  However, if the kernel thread is freezable, it will be

frozen before the driver's .suspend() callback is executed and it will be

thawed after the driver's .resume() callback has run, so it won't be accessing

the device while it's suspended.

3. Another reason for freezing tasks is to prevent user space processes from

realizing that hibernation (or suspend) operation takes place.  Ideally, user

space processes should not notice that such a system-wide operation has occurred

and should continue running without any problems after the restore (or resume

from suspend).  Unfortunately, in the most general case this is quite difficult

to achieve without the freezing of tasks.  Consider, for example, a process

that depends on all CPUs being online while it's running.  Since we need to

disable nonboot CPUs during the hibernation, if this process is not frozen, it

may notice that the number of CPUs has changed and may start to work incorrectly

because of that.

V. Are there any problems related to the freezing of tasks?

Yes, there are.

First of all, the freezing of kernel threads may be tricky if they depend one

on another.  For example, if kernel thread A waits for a completion (in the

TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B

and B is frozen in the meantime, then A will be blocked until B is thawed, which

may be undesirable.  That's why kernel threads are not freezable by default.

Second, there are the following two problems related to the freezing of user

space processes:

1. Putting processes into an uninterruptible sleep distorts the load average.

2. Now that we have FUSE, plus the framework for doing device drivers in

userspace, it gets even more complicated because some userspace processes are

now doing the sorts of things that kernel threads do

(https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html).

The problem 1. seems to be fixable, although it hasn't been fixed so far.  The

other one is more serious, but it seems that we can work around it by using

hibernation (and suspend) notifiers (in that case, though, we won't be able to

avoid the realization by the user space processes that the hibernation is taking

place).

There are also problems that the freezing of tasks tends to expose, although

they are not directly related to it.  For example, if request_firmware() is

called from a device driver's .resume() routine, it will timeout and eventually

fail, because the user land process that should respond to the request is frozen

at this point.  So, seemingly, the failure is due to the freezing of tasks.

Suppose, however, that the firmware file is located on a filesystem accessible

only through another device that hasn't been resumed yet.  In that case,

request_firmware() will fail regardless of whether or not the freezing of tasks

is used.  Consequently, the problem is not really related to the freezing of

tasks, since it generally exists anyway.  [The solution to this particular

problem is to keep the firmware in memory after it's loaded for the first time

and upload if from memory to the device whenever necessary.]

KERNEL THREADS

Freezer

Upon entering a suspended state the system will freeze all

tasks. This is done by delivering pseudosignals. This affects

kernel threads, too. To successfully freeze a kernel thread

the thread has to check for the pseudosignal and enter the

refrigerator. Code to do this looks like this:

	do {

		hub_events();

		wait_event_interruptible(khubd_wait, !list_empty(&hub_event_list));

		try_to_freeze();

	} while (!signal_pending(current));

from drivers/usb/core/hub.c::hub_thread()

The Unfreezable

Some kernel threads however, must not be frozen. The kernel must

be able to finish pending IO operations and later on be able to

write the memory image to disk. Kernel threads needed to do IO

must stay awake. Such threads must mark themselves unfreezable

like this:

	/*

	 * This thread doesn't need any user-level access,

	 * so get rid of all our resources.

	 */

	daemonize("usb-storage");

	current->flags |= PF_NOFREEZE;

from drivers/usb/storage/usb.c::usb_stor_control_thread()

Such drivers are themselves responsible for staying quiet during

the actual snapshotting.

website, and not to the Linux Kernel Mailing List. We are working

toward merging suspend2 into the mainline kernel.

Q: A kernel thread must voluntarily freeze itself (call 'refrigerator').

I found some kernel threads that don't do it, and they don't freeze

so the system can't sleep. Is this a known behavior?

A: All such kernel threads need to be fixed, one by one. Select the

place where the thread is safe to be frozen (no kernel semaphores

should be held at that point and it must be safe to sleep there), and

add:

       try_to_freeze();

If the thread is needed for writing the image to storage, you should

instead set the PF_NOFREEZE process flag when creating the thread (and

be very careful).

Q: What is the freezing of tasks and why are we using it?

A: The freezing of tasks is a mechanism by which user space processes and some

kernel threads are controlled during hibernation or system-wide suspend (on some

architectures).  See freezing-of-tasks.txt for details.

Q: What is the difference between "platform" and "shutdown"?

#include <linux/capability.h>

#include <linux/device.h>

#include <linux/kernel.h>

#include <linux/freezer.h>

#include <linux/smp.h>

#include <linux/dmi.h>

#include <linux/suspend.h>

		remove_proc_entry("apm", NULL);

		return err;

	}

	kapmd_task->flags |= PF_NOFREEZE;

	wake_up_process(kapmd_task);

	if (num_online_cpus() > 1 && !smp ) {

		set_pending_irq(i, cpumask_of_cpu(0));

	}

	set_freezable();

	for ( ; ; ) {

		time_remaining = schedule_timeout_interruptible(time_remaining);

		try_to_freeze();