decompressors: add XZ decompressor module

XZ data compression in Linux

============================

Introduction

    XZ is a general purpose data compression format with high compression

    ratio and relatively fast decompression. The primary compression

    algorithm (filter) is LZMA2. Additional filters can be used to improve

    compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters

    improve compression ratio of executable data.

    The XZ decompressor in Linux is called XZ Embedded. It supports

    the LZMA2 filter and optionally also BCJ filters. CRC32 is supported

    for integrity checking. The home page of XZ Embedded is at

    <http://tukaani.org/xz/embedded.html>, where you can find the

    latest version and also information about using the code outside

    the Linux kernel.

    For userspace, XZ Utils provide a zlib-like compression library

    and a gzip-like command line tool. XZ Utils can be downloaded from

    <http://tukaani.org/xz/>.

XZ related components in the kernel

    The xz_dec module provides XZ decompressor with single-call (buffer

    to buffer) and multi-call (stateful) APIs. The usage of the xz_dec

    module is documented in include/linux/xz.h.

    The xz_dec_test module is for testing xz_dec. xz_dec_test is not

    useful unless you are hacking the XZ decompressor. xz_dec_test

    allocates a char device major dynamically to which one can write

    .xz files from userspace. The decompressed output is thrown away.

    Keep an eye on dmesg to see diagnostics printed by xz_dec_test.

    See the xz_dec_test source code for the details.

    For decompressing the kernel image, initramfs, and initrd, there

    is a wrapper function in lib/decompress_unxz.c. Its API is the

    same as in other decompress_*.c files, which is defined in

    include/linux/decompress/generic.h.

    scripts/xz_wrap.sh is a wrapper for the xz command line tool found

    from XZ Utils. The wrapper sets compression options to values suitable

    for compressing the kernel image.

    For kernel makefiles, two commands are provided for use with

    $(call if_needed). The kernel image should be compressed with

    $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2

    dictionary. It will also append a four-byte trailer containing the

    uncompressed size of the file, which is needed by the boot code.

    Other things should be compressed with $(call if_needed,xzmisc)

    which will use no BCJ filter and 1 MiB LZMA2 dictionary.

Notes on compression options

    Since the XZ Embedded supports only streams with no integrity check or

    CRC32, make sure that you don't use some other integrity check type

    when encoding files that are supposed to be decoded by the kernel. With

    liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32

    when encoding. With the xz command line tool, use --check=none or

    --check=crc32.

    Using CRC32 is strongly recommended unless there is some other layer

    which will verify the integrity of the uncompressed data anyway.

    Double checking the integrity would probably be waste of CPU cycles.

    Note that the headers will always have a CRC32 which will be validated

    by the decoder; you can only change the integrity check type (or

    disable it) for the actual uncompressed data.

    In userspace, LZMA2 is typically used with dictionary sizes of several

    megabytes. The decoder needs to have the dictionary in RAM, thus big

    dictionaries cannot be used for files that are intended to be decoded

    by the kernel. 1 MiB is probably the maximum reasonable dictionary

    size for in-kernel use (maybe more is OK for initramfs). The presets

    in XZ Utils may not be optimal when creating files for the kernel,

    so don't hesitate to use custom settings. Example:

        xz --check=crc32 --lzma2=dict=512KiB inputfile

    An exception to above dictionary size limitation is when the decoder

    is used in single-call mode. Decompressing the kernel itself is an

    example of this situation. In single-call mode, the memory usage

    doesn't depend on the dictionary size, and it is perfectly fine to

    use a big dictionary: for maximum compression, the dictionary should

    be at least as big as the uncompressed data itself.

Future plans

    Creating a limited XZ encoder may be considered if people think it is

    useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at

    the fastest settings, so it isn't clear if LZMA2 encoder is wanted

    into the kernel.

    Support for limited random-access reading is planned for the

    decompression code. I don't know if it could have any use in the

    kernel, but I know that it would be useful in some embedded projects

    outside the Linux kernel.

Conformance to the .xz file format specification

    There are a couple of corner cases where things have been simplified

    at expense of detecting errors as early as possible. These should not

    matter in practice all, since they don't cause security issues. But

    it is good to know this if testing the code e.g. with the test files

    from XZ Utils.

Reporting bugs

    Before reporting a bug, please check that it's not fixed already

    at upstream. See <http://tukaani.org/xz/embedded.html> to get the

    latest code.

    Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on

    Freenode and talk to Larhzu. I don't actively read LKML or other

    kernel-related mailing lists, so if there's something I should know,

    you should email to me personally or use IRC.

    Don't bother Igor Pavlov with questions about the XZ implementation

    in the kernel or about XZ Utils. While these two implementations

    include essential code that is directly based on Igor Pavlov's code,

    these implementations aren't maintained nor supported by him.

/*

 * XZ decompressor

 *

 * Authors: Lasse Collin <lasse.collin@tukaani.org>

 *          Igor Pavlov <http://7-zip.org/>

 *

 * This file has been put into the public domain.

 * You can do whatever you want with this file.

 */

#ifndef XZ_H

#define XZ_H

#ifdef __KERNEL__

#	include <linux/stddef.h>

#	include <linux/types.h>

#else

#	include <stddef.h>

#	include <stdint.h>

#endif

/* In Linux, this is used to make extern functions static when needed. */

#ifndef XZ_EXTERN

#	define XZ_EXTERN extern

#endif

/**

 * enum xz_mode - Operation mode

 *

 * @XZ_SINGLE:              Single-call mode. This uses less RAM than

 *                          than multi-call modes, because the LZMA2

 *                          dictionary doesn't need to be allocated as

 *                          part of the decoder state. All required data

 *                          structures are allocated at initialization,

 *                          so xz_dec_run() cannot return XZ_MEM_ERROR.

 * @XZ_PREALLOC:            Multi-call mode with preallocated LZMA2

 *                          dictionary buffer. All data structures are

 *                          allocated at initialization, so xz_dec_run()

 *                          cannot return XZ_MEM_ERROR.

 * @XZ_DYNALLOC:            Multi-call mode. The LZMA2 dictionary is

 *                          allocated once the required size has been

 *                          parsed from the stream headers. If the

 *                          allocation fails, xz_dec_run() will return

 *                          XZ_MEM_ERROR.

 *

 * It is possible to enable support only for a subset of the above

 * modes at compile time by defining XZ_DEC_SINGLE, XZ_DEC_PREALLOC,

 * or XZ_DEC_DYNALLOC. The xz_dec kernel module is always compiled

 * with support for all operation modes, but the preboot code may

 * be built with fewer features to minimize code size.

 */

enum xz_mode {

	XZ_SINGLE,

	XZ_PREALLOC,

	XZ_DYNALLOC

};

/**

 * enum xz_ret - Return codes

 * @XZ_OK:                  Everything is OK so far. More input or more

 *                          output space is required to continue. This

 *                          return code is possible only in multi-call mode

 *                          (XZ_PREALLOC or XZ_DYNALLOC).

 * @XZ_STREAM_END:          Operation finished successfully.

 * @XZ_UNSUPPORTED_CHECK:   Integrity check type is not supported. Decoding

 *                          is still possible in multi-call mode by simply

 *                          calling xz_dec_run() again.

 *                          Note that this return value is used only if

 *                          XZ_DEC_ANY_CHECK was defined at build time,

 *                          which is not used in the kernel. Unsupported

 *                          check types return XZ_OPTIONS_ERROR if

 *                          XZ_DEC_ANY_CHECK was not defined at build time.

 * @XZ_MEM_ERROR:           Allocating memory failed. This return code is

 *                          possible only if the decoder was initialized

 *                          with XZ_DYNALLOC. The amount of memory that was

 *                          tried to be allocated was no more than the

 *                          dict_max argument given to xz_dec_init().

 * @XZ_MEMLIMIT_ERROR:      A bigger LZMA2 dictionary would be needed than

 *                          allowed by the dict_max argument given to

 *                          xz_dec_init(). This return value is possible

 *                          only in multi-call mode (XZ_PREALLOC or

 *                          XZ_DYNALLOC); the single-call mode (XZ_SINGLE)

 *                          ignores the dict_max argument.

 * @XZ_FORMAT_ERROR:        File format was not recognized (wrong magic

 *                          bytes).

 * @XZ_OPTIONS_ERROR:       This implementation doesn't support the requested

 *                          compression options. In the decoder this means

 *                          that the header CRC32 matches, but the header

 *                          itself specifies something that we don't support.

 * @XZ_DATA_ERROR:          Compressed data is corrupt.

 * @XZ_BUF_ERROR:           Cannot make any progress. Details are slightly

 *                          different between multi-call and single-call

 *                          mode; more information below.

 *

 * In multi-call mode, XZ_BUF_ERROR is returned when two consecutive calls

 * to XZ code cannot consume any input and cannot produce any new output.

 * This happens when there is no new input available, or the output buffer

 * is full while at least one output byte is still pending. Assuming your

 * code is not buggy, you can get this error only when decoding a compressed

 * stream that is truncated or otherwise corrupt.

 *

 * In single-call mode, XZ_BUF_ERROR is returned only when the output buffer

 * is too small or the compressed input is corrupt in a way that makes the

 * decoder produce more output than the caller expected. When it is

 * (relatively) clear that the compressed input is truncated, XZ_DATA_ERROR

 * is used instead of XZ_BUF_ERROR.

 */

enum xz_ret {

	XZ_OK,

	XZ_STREAM_END,

	XZ_UNSUPPORTED_CHECK,

	XZ_MEM_ERROR,

	XZ_MEMLIMIT_ERROR,

	XZ_FORMAT_ERROR,

	XZ_OPTIONS_ERROR,

	XZ_DATA_ERROR,

	XZ_BUF_ERROR

};

/**

 * struct xz_buf - Passing input and output buffers to XZ code

 * @in:         Beginning of the input buffer. This may be NULL if and only

 *              if in_pos is equal to in_size.

 * @in_pos:     Current position in the input buffer. This must not exceed

 *              in_size.

 * @in_size:    Size of the input buffer

 * @out:        Beginning of the output buffer. This may be NULL if and only

 *              if out_pos is equal to out_size.

 * @out_pos:    Current position in the output buffer. This must not exceed

 *              out_size.

 * @out_size:   Size of the output buffer

 *

 * Only the contents of the output buffer from out[out_pos] onward, and

 * the variables in_pos and out_pos are modified by the XZ code.

 */

struct xz_buf {

	const uint8_t *in;

	size_t in_pos;

	size_t in_size;

	uint8_t *out;

	size_t out_pos;

	size_t out_size;

};

/**

 * struct xz_dec - Opaque type to hold the XZ decoder state

 */

struct xz_dec;

/**

 * xz_dec_init() - Allocate and initialize a XZ decoder state

 * @mode:       Operation mode

 * @dict_max:   Maximum size of the LZMA2 dictionary (history buffer) for

 *              multi-call decoding. This is ignored in single-call mode

 *              (mode == XZ_SINGLE). LZMA2 dictionary is always 2^n bytes

 *              or 2^n + 2^(n-1) bytes (the latter sizes are less common

 *              in practice), so other values for dict_max don't make sense.

 *              In the kernel, dictionary sizes of 64 KiB, 128 KiB, 256 KiB,

 *              512 KiB, and 1 MiB are probably the only reasonable values,

 *              except for kernel and initramfs images where a bigger

 *              dictionary can be fine and useful.

 *

 * Single-call mode (XZ_SINGLE): xz_dec_run() decodes the whole stream at

 * once. The caller must provide enough output space or the decoding will

 * fail. The output space is used as the dictionary buffer, which is why

 * there is no need to allocate the dictionary as part of the decoder's

 * internal state.

 *

 * Because the output buffer is used as the workspace, streams encoded using

 * a big dictionary are not a problem in single-call mode. It is enough that

 * the output buffer is big enough to hold the actual uncompressed data; it

 * can be smaller than the dictionary size stored in the stream headers.

 *

 * Multi-call mode with preallocated dictionary (XZ_PREALLOC): dict_max bytes

 * of memory is preallocated for the LZMA2 dictionary. This way there is no

 * risk that xz_dec_run() could run out of memory, since xz_dec_run() will

 * never allocate any memory. Instead, if the preallocated dictionary is too

 * small for decoding the given input stream, xz_dec_run() will return

 * XZ_MEMLIMIT_ERROR. Thus, it is important to know what kind of data will be

 * decoded to avoid allocating excessive amount of memory for the dictionary.

 *

 * Multi-call mode with dynamically allocated dictionary (XZ_DYNALLOC):

 * dict_max specifies the maximum allowed dictionary size that xz_dec_run()

 * may allocate once it has parsed the dictionary size from the stream

 * headers. This way excessive allocations can be avoided while still

 * limiting the maximum memory usage to a sane value to prevent running the

 * system out of memory when decompressing streams from untrusted sources.

 *

 * On success, xz_dec_init() returns a pointer to struct xz_dec, which is

 * ready to be used with xz_dec_run(). If memory allocation fails,

 * xz_dec_init() returns NULL.

 */

XZ_EXTERN struct xz_dec *xz_dec_init(enum xz_mode mode, uint32_t dict_max);

/**

 * xz_dec_run() - Run the XZ decoder

 * @s:          Decoder state allocated using xz_dec_init()

 * @b:          Input and output buffers

 *

 * The possible return values depend on build options and operation mode.

 * See enum xz_ret for details.

 *

 * Note that if an error occurs in single-call mode (return value is not

 * XZ_STREAM_END), b->in_pos and b->out_pos are not modified and the

 * contents of the output buffer from b->out[b->out_pos] onward are

 * undefined. This is true even after XZ_BUF_ERROR, because with some filter

 * chains, there may be a second pass over the output buffer, and this pass

 * cannot be properly done if the output buffer is truncated. Thus, you

 * cannot give the single-call decoder a too small buffer and then expect to

 * get that amount valid data from the beginning of the stream. You must use

 * the multi-call decoder if you don't want to uncompress the whole stream.

 */

XZ_EXTERN enum xz_ret xz_dec_run(struct xz_dec *s, struct xz_buf *b);

/**

 * xz_dec_reset() - Reset an already allocated decoder state

 * @s:          Decoder state allocated using xz_dec_init()

 *

 * This function can be used to reset the multi-call decoder state without

 * freeing and reallocating memory with xz_dec_end() and xz_dec_init().

 *

 * In single-call mode, xz_dec_reset() is always called in the beginning of

 * xz_dec_run(). Thus, explicit call to xz_dec_reset() is useful only in

 * multi-call mode.

 */

XZ_EXTERN void xz_dec_reset(struct xz_dec *s);

/**

 * xz_dec_end() - Free the memory allocated for the decoder state

 * @s:          Decoder state allocated using xz_dec_init(). If s is NULL,

 *              this function does nothing.

 */

XZ_EXTERN void xz_dec_end(struct xz_dec *s);

/*

 * Standalone build (userspace build or in-kernel build for boot time use)

 * needs a CRC32 implementation. For normal in-kernel use, kernel's own

 * CRC32 module is used instead, and users of this module don't need to

 * care about the functions below.

 */

#ifndef XZ_INTERNAL_CRC32

#	ifdef __KERNEL__

#		define XZ_INTERNAL_CRC32 0

#	else

#		define XZ_INTERNAL_CRC32 1

#	endif

#endif

#if XZ_INTERNAL_CRC32

/*

 * This must be called before any other xz_* function to initialize

 * the CRC32 lookup table.

 */

XZ_EXTERN void xz_crc32_init(void);

/*

 * Update CRC32 value using the polynomial from IEEE-802.3. To start a new

 * calculation, the third argument must be zero. To continue the calculation,

 * the previously returned value is passed as the third argument.

 */

XZ_EXTERN uint32_t xz_crc32(const uint8_t *buf, size_t size, uint32_t crc);

#endif

#endif

config LZO_DECOMPRESS

	tristate

source "lib/xz/Kconfig"

#

# These all provide a common interface (hence the apparent duplication with

# ZLIB_INFLATE; DECOMPRESS_GZIP is just a wrapper.)

obj-$(CONFIG_REED_SOLOMON) += reed_solomon/

obj-$(CONFIG_LZO_COMPRESS) += lzo/

obj-$(CONFIG_LZO_DECOMPRESS) += lzo/

obj-$(CONFIG_XZ_DEC) += xz/

obj-$(CONFIG_RAID6_PQ) += raid6/

lib-$(CONFIG_DECOMPRESS_GZIP) += decompress_inflate.o

config XZ_DEC

	tristate "XZ decompression support"

	select CRC32

	help

	  LZMA2 compression algorithm and BCJ filters are supported using

	  the .xz file format as the container. For integrity checking,

	  CRC32 is supported. See Documentation/xz.txt for more information.

config XZ_DEC_X86

	bool "x86 BCJ filter decoder" if EMBEDDED

	default y

	depends on XZ_DEC

	select XZ_DEC_BCJ

config XZ_DEC_POWERPC

	bool "PowerPC BCJ filter decoder" if EMBEDDED

	default y

	depends on XZ_DEC

	select XZ_DEC_BCJ

config XZ_DEC_IA64

	bool "IA-64 BCJ filter decoder" if EMBEDDED

	default y

	depends on XZ_DEC

	select XZ_DEC_BCJ

config XZ_DEC_ARM

	bool "ARM BCJ filter decoder" if EMBEDDED

	default y

	depends on XZ_DEC

	select XZ_DEC_BCJ

config XZ_DEC_ARMTHUMB

	bool "ARM-Thumb BCJ filter decoder" if EMBEDDED

	default y

	depends on XZ_DEC

	select XZ_DEC_BCJ

config XZ_DEC_SPARC

	bool "SPARC BCJ filter decoder" if EMBEDDED

	default y

	depends on XZ_DEC

	select XZ_DEC_BCJ

config XZ_DEC_BCJ

	bool

	default n

config XZ_DEC_TEST

	tristate "XZ decompressor tester"

	default n

	depends on XZ_DEC

	help

	  This allows passing .xz files to the in-kernel XZ decoder via

	  a character special file. It calculates CRC32 of the decompressed

	  data and writes diagnostics to the system log.

	  Unless you are developing the XZ decoder, you don't need this

	  and should say N.