doc: compression format

# Compressed binary file internals

This article describes the structure of the

`zephyr.signed.bin` file when image

compression is enabled. You do not need to know these details to use the

image compression subsystem, but they can be beneficial if you want to

use them for verification or custom integration purposes.

For an example, see the following structure of the file:

![LZMA header](./images/decomp.png)

## [LZMA Header](#LZMA-Header)

The Lempel-Ziv-Markov chain Algorithm (LZMA) header is crucial for files

compressed using the LZMA method. It contains metadata essential for

decompression. The `lzma2_header` encodes compression parameters using

two bytes.

### [Calculating compression parameters](#Calculating-compression-parameters)

Compression parameters can be calculated, retrieved, or changed

depending on your needs. For details, see the following sections.

#### [Default values](#Default-values)

Compression parameters have the following default values:

-   `dict_size`: 131072

-   `pb`: 2

-   `lc`: 3

-   `lp`: 1

#### [Adjusting dictionary size](#Adjusting-dictionary-size)

You can calculate the `dict_size` using the following method:

``` {.c}

unsigned int i = 0;

for (i = 0; i < 40; i++) {

    if (raw_dict_size <= (((uint32_t)2 | ((i) & 1)) << ((i) / 2 + 11))) {

        break;

    }

}

dict_size = (uint8_t)i;

```

With this method, `dict_size` can have one of the following values:

 |Hex Value  |  Size      |

 |-----------|------------|

 |0x00       | 4096       |

 |0x01       | 6144       |

 |0x02       | 8192       |

 |0x03       | 12288      |

 |0x04       | 16384      |

 |0x05       | 24576      |

 |0x06       | 32768      |

 |0x07       | 49152      |

 |0x08       | 65536      |

 |0x09       | 98304      |

 |0x0a       | 131072     |

 |0x0b       | 196608     |

 |0x0c       | 262144     |

 |0x0d       | 393216     |

 |0x0e       | 524288     |

 |0x0f       | 786432     |

 |0x10       | 1048576    |

 |0x11       | 1572864    |

 |0x12       | 2097152    |

 |0x13       | 3145728    |

 |0x14       | 4194304    |

 |0x15       | 6291456    |

 |0x16       | 8388608    |

 |0x17       | 12582912   |

 |0x18       | 16777216   |

 |0x19       | 25165824   |

 |0x1a       | 33554432   |

 |0x1b       | 50331648   |

 |0x1c       | 67108864   |

 |0x1d       | 100663296  |

 |0x1e       | 134217728  |

 |0x1f       | 201326592  |

 |0x20       | 268435456  |

 |0x21       | 402653184  |

 |0x22       | 536870912  |

 |0x23       | 805306368  |

 |0x24       | 1073741824 |

 |0x25       | 1610612736 |

 |0x26       | 2147483648 |

 |0x27       | 3221225472 |

#### [Calculating literal context, literal pos, and pos bits](#Calculating-literal-context-literal-pos-and-pos-bits)

The second byte of the `lzma2_header` carries the following parameters:

-   `lc`, which specifies a number of literal context bits

-   `lp`, which specifies a number of literal pos bits

-   `pb`, which specifies a number of pos bits

    These parameters are encoded with the following formula:

    ``` {.c}

    pb_lp_lc = (uint8_t)((pb * 5 + lp) * 9 + lc);

    ```

    To decode these values from the combined `pb_lp_lc` byte, run the

    following code:

    ``` {.c}

    lc = pb_lp_lc % 9;

    pb_lp_lc /= 9;

    pb = pb_lp_lc / 5;

    lp = pb_lp_lc % 5;

    ```

## [Extracting LZMA stream from image](#Extracting-LZMA-stream-from-image)

To extract and decompress the LZMA stream from the image, follow these

steps:

1.  Determine the offset of the compressed stream by adding the

    `lzma2_header` size and the value stored under

    `image_header.ih_hdr_size`. For the size of the compressed stream,

    see `image_header.ih_img_size`.

2.  If the compressed stream is isolated and stored in a file named

    `raw.lzma`, you can perform

    decompression using the following commands:

 -   Without an ARM thumb filter:

     ``` {.bash}

     unlzma --lzma2 --format=raw --suffix=.lzma raw.lzma

     ```

 -   With an ARM thumb filter:

     ``` {.bash}

     unlzma --armthumb --lzma2 --format=raw --suffix=.lzma raw.lzma

     ```

Once the command is executed you will see a newly created file named

`raw`, which is identical to the

image before compression.

## [TLVs](#TLVs)

The following Type-Length-Values (TLVs) are used in the context of

decompressed images:

-   `DECOMP_SIZE (0x70)`: Specifies the size of the decompressed image.

-   `DECOMP_SHA (0x71)`: Contains the hash of the decompressed image.

-   `DECOMP_SIGNATURE (0x72)`: Holds the signature of either the hash or

    the entire image.

These TLVs are placed in the protected TLV section, ensuring they are

included in the hashing and signature calculations during the

verification process. The process for choosing the type of cryptographic

signature and hash algorithm used for securing the image is the same,

regardless of whether the image has undergone compression.

## [Sample](#Sample)

For practical implementation, you can find a simple stand-alone

verification program under the following path

`bootloader/mcuboot/samples/compression_test/independent_cmp.c`

This program demonstrates how to independently verify the integrity and

authenticity of a decompressed image using the specified TLVs.

Image signing takes an image in binary or Intel Hex format intended for the

primary slot and adds a header and trailer that the bootloader is expecting:

    Usage: imgtool.py sign [OPTIONS] INFILE OUTFILE

    Usage: imgtool sign [OPTIONS] INFILE OUTFILE

      Create a signed or unsigned image

      extension, otherwise binary format is used

    Options:

      --vector-to-sign [payload|digest]

                                      send to OUTFILE the payload or payloads

                                      digest instead of complied image. These data

                                      can be used for external image signing

      --sha [auto|256|384|512]        selected sha algorithm to use; defaults to

                                      "auto" which is 256 if no cryptographic

                                      signature is used, or default for signature

                                      type

      --sig-out filename              Path to the file to which signature will be

                                      written. The image signature will be encoded

                                      as base64 formatted string

      --pure                          Expected Pure variant of signature; the Pure

                                      variant is expected to be signature done

                                      over an image rather than hash of that

                                      image.

      --fix-sig-pubkey filename       public key relevant to fixed signature

      --fix-sig filename              fixed signature for the image. It will be

                                      used instead of the signature calculated

                                      using the public key

      -k, --key filename

      --public-key-format [hash|full]

                                      In what format to add the public key to the

                                      image manifest: full key or hash of the key.

      --max-align [8|16|32]           Maximum flash alignment. Set if flash

                                      alignment of the primary and secondary slot

                                      differ and any of them is larger than 8.

      --align [1|2|4|8|16|32]         Alignment used by swap update modes.

      -v, --version TEXT              [required]

      -s, --security-counter TEXT     Specify the value of security counter. Use

                                      the `auto` keyword to automatically generate

                                      it from the image version.

      -d, --dependencies TEXT

      -d, --dependencies TEXT         Add dependence on another image, format:

                                      "(<image_ID>,<image_version>), ... "

      --pad-sig                       Add 0-2 bytes of padding to ECDSA signature

                                    (for MCUboot <1.5)

                                      (for mcuboot <1.5)

      -H, --header-size INTEGER       [required]

      --pad-header                    Add --header-size zeroed bytes at the

                                      beginning of the image

      -S, --slot-size INTEGER       Size of the slot where the image will be

                                    written [required]

      -S, --slot-size INTEGER         Size of the slot. If the slots have

                                      different sizes, use the size of the

                                      secondary slot.  [required]

      --pad                           Pad image to --slot-size bytes, adding

                                      trailer magic

      --confirm                       When padding the image, mark it as confirmed

                                      (implies --pad)

      -M, --max-sectors INTEGER       When padding allow for this amount of

                                      sectors (defaults to 128)

      --boot-record sw_type           Create CBOR encoded boot record TLV. The

                                      firmware). [max. 12 characters]

      --overwrite-only                Use overwrite-only instead of swap upgrades

      -e, --endian [little|big]       Select little or big endian

      -E, --encrypt filename        Encrypt image using the provided public key

      -c, --clear                     Output a non-encrypted image with encryption

                                      capabilities,so it can be installed in the

                                      primary slot, and encrypted when swapped to

                                      the secondary.

      --skip-encryption               Set encryption flags and TLV's without

                                      applying encryption.

      --compression [disabled|lzma2|lzma2armthumb]

                                      Enable image compression using specified

                                      type. Will fall back without image

                                      compression automatically if the compression

                                      increases the image size.

      --encrypt-keylen [128|256]      When encrypting the image using AES, select

                                      a 128 bit or 256 bit key len.

      -E, --encrypt filename          Encrypt image using the provided public key.

                                      (Not supported in direct-xip or ram-load

                                      mode.)

      --save-enctlv                   When upgrading, save encrypted key TLVs

                                      instead of plain keys. Enable when

                                      BOOT_SWAP_SAVE_ENCTLV config option was set.

      -F, --rom-fixed INTEGER         Set flash address the image is built for.

      -L, --load-addr INTEGER         Load address for image when it should run

                                      from RAM.

      -x, --hex-addr INTEGER          Adjust address in hex output file.

      -R, --erased-val [0|0xff]       The value that is read back from erased

                                      flash.

      --custom-tlv [tag] [value]      Custom TLV that will be placed into

                                      protected area. Add "0x" prefix if the value

                                      should be interpreted as an integer,

                                      otherwise it will be interpreted as a

                                      string. Specify the option multiple times to

                                      add multiple TLVs.

      --non-bootable                  Mark the image as non-bootable.

      -h, --help                      Show this message and exit.

The main arguments given are the key file generated above, a version

adjust for the new bytes prepended to the file. The load address of all data

existing in the file should not change.

The `--compression` option enables LZMA compression over payload. Details

about internals of image generated with this option can be found here

[here](./compression_format.md)

This isn't fully supported on the embedded side but can be utilised when

project is built on top of the mcuboot.

The `--slot-size` argument is required and used to check that the firmware

does not overflow into the swap status area (metadata). If swap upgrades are

not being used, `--overwrite-only` can be passed to avoid adding the swap