fs: zms: multiple fixes from previous PR review

	struct zms_ate {

		uint8_t crc8;      /* crc8 check of the entry */

		uint8_t cycle_cnt; /* cycle counter for non erasable devices */

		uint32_t id;       /* data id */

		uint8_t cycle_cnt; /* cycle counter for non-erasable devices */

		uint16_t len;      /* data len within sector */

		uint32_t id;       /* data id */

		union {

			uint8_t data[8]; /* used to store small size data */

			struct {

		};

	} __packed;

.. note:: The data CRC is checked only when the whole data of the element is read.

   The data CRC is not checked for a partial read, as it is computed for the complete set of data.

.. note:: The CRC of the data is checked only when the whole the element is read.

   The CRC of the data is not checked for a partial read, as it is computed for the whole element.

.. note:: Enabling the data CRC feature on a previously existing ZMS content without

   data CRC will make all existing data invalid.

.. note:: Enabling the CRC feature on previously existing ZMS content without CRC enabled

   will make all existing data invalid.

.. _free-space:

Available space for user data (key-value pairs)

***********************************************

For both scenarios ZMS should have always an empty sector to be able to perform the garbage

collection.

So if we suppose that 4 sectors exist in a partition, ZMS will only use 3 sectors to store

Key-value pairs and keep always one (rotating sector) empty to be able to launch GC.

For both scenarios ZMS should always have an empty sector to be able to perform the

garbage collection (GC).

So, if we suppose that 4 sectors exist in a partition, ZMS will only use 3 sectors to store

Key-value pairs and keep one sector empty to be able to launch GC.

The empty sector will rotate between the 4 sectors in the partition.

.. note:: The maximum single data length that could be written at once in a sector is 64K

   (This could change in future versions of ZMS)

Small data values

=================

For small data values (<= 8 bytes), the data is stored within the entry (ATE) itself and no data

is written at the top of the sector.

Values smaller than 8 bytes will be stored within the entry (ATE) itself, without writing data

at the top of the sector.

ZMS has an entry size of 16 bytes which means that the maximum available space in a partition to

store data is computed in this scenario as :

=================

Large data values ( > 8 bytes) are stored separately at the top of the sector.

In this case it is hard to estimate the free available space as this depends on the size of

In this case, it is hard to estimate the free available space, as this depends on the size of

the data. But we can take into account that for N bytes of data (N > 8 bytes) an additional

16 bytes of ATE must be added at the bottom of the sector.

Using storage systems that rely on an erase-value (NVS as an example) will need to emulate the

erase with write operations. This will cause a significant decrease in the life expectancy of

these devices and will cause more delays for write operations and for initialization.

ZMS introduces a cycle count mechanism that avoids emulating erase operation for these devices.

ZMS uses a cycle count mechanism that avoids emulating erase operation for these devices.

It also guarantees that every memory location is written only once for each cycle of sector write.

As an example, to erase a 4096 bytes sector on a non erasable device using NVS, 256 flash writes

As an example, to erase a 4096 bytes sector on a non-erasable device using NVS, 256 flash writes

must be performed (supposing that write-block-size=16 bytes), while using ZMS only 1 write of

16 bytes is needed. This operation is 256 times faster in this case.

Garbage collection operation is also adding some writes to the memory cell life expectancy as it

is moving some blocks from one sector to another.

To make the garbage collector not affect the life expectancy of the device it is recommended

to dimension correctly the partition size. Its size should be the double of the maximum size of

to correctly dimension the partition size. Its size should be the double of the maximum size of

data (including extra headers) that could be written in the storage.

See :ref:`free-space`.

Storage devices whether they are classical Flash or new technologies like RRAM/MRAM has a limited

life expectancy which is determined by the number of times memory cells can be erased/written.

Flash devices are erased one page at a time as part of their functional behavior (otherwise

memory cells cannot be overwritten) and for non erasable storage devices memory cells can be

memory cells cannot be overwritten) and for non-erasable storage devices memory cells can be

overwritten directly.

A typical scenario is shown here to calculate the life expectancy of a device.

A typical scenario is shown here to calculate the life expectancy of a device:

Let's suppose that we store an 8 bytes variable using the same ID but its content changes every

minute. The partition has 4 sectors with 1024 bytes each.

Each write of the variable requires 16 bytes of storage.

=================

Version1

--------

- Supports non erasable devices (only one write operation to erase a sector)

- Supports non-erasable devices (only one write operation to erase a sector)

- Supports large partition size and sector size (64 bits address space)

- Supports large IDs width (32 bits) to store ID/Value pairs

- Supports 32-bit IDs to store ID/Value pairs

- Small sized data ( <= 8 bytes) are stored in the ATE itself

- Built-in Data CRC32 (included in the ATE)

- Versionning of ZMS (to handle future evolution)

- Add multiple format ATE support to be able to use ZMS with different ATE formats that satisfies

  requirements from application

- Add the possibility to skip garbage collector for some application usage where ID/value pairs

  are written periodically and do not exceed half of the partition size (ther is always an old

  are written periodically and do not exceed half of the partition size (there is always an old

  entry with the same ID).

- Divide IDs into namespaces and allocate IDs on demand from application to handle collisions

  between IDs used by different subsystems or samples.

Which one to use in your application will depend on your needs and the hardware you are using,

and this section provides information to help make a choice.

- If you are using a non erasable technology device like RRAM or MRAM, :ref:`ZMS <zms_api>` is definitely the

  best fit for your storage subsystem as it is designed very well to avoid emulating erase for

  these devices and replace it by a single write call.

- If you are using a non-erasable technology device like RRAM or MRAM, :ref:`ZMS <zms_api>` is definitely the

  best fit for your storage subsystem as it is designed to avoid emulating erase operation using

  large block writes for these devices and replaces it with a single write call.

- For devices with large write_block_size and/or needs a sector size that is different than the

  classical flash page size (equal to erase_block_size), :ref:`ZMS <zms_api>` is also the best fit as there is

  the possibility to customize these parameters and add the support of these devices in ZMS.

both solutions could be implemented, the best choice should be based on the calculations of the

life expectancy of the device described in this section: :ref:`wear-leveling`.

Recommendations to increase performance

***************************************

Sector size and count

=====================

- The total size of the storage partition should be well dimensioned to achieve the best

  performance for ZMS.

  All the information regarding the effectively available free space in ZMS can be found

  in the documentation. See :ref:`free-space`.

  We recommend choosing a storage partition that can hold double the size of the key-value pairs

  that will be written in the storage.

- The size of a sector needs to be dimensioned to hold the maximum data length that will be stored.

  Increasing the size of a sector will slow down the garbage collection operation which will

  occur less frequently.

  Decreasing its size, in the opposite, will make the garbage collection operation faster

  which will occur more frequently.

- For some subsystems like :ref:`Settings <settings_api>`, all path-value pairs are split into two ZMS entries (ATEs).

  The header needed by the two entries should be accounted when computing the needed storage space.

- Using small data to store in the ZMS entries can increase the performance, as this data is

  written within the entry header.

  For example, for the :ref:`Settings <settings_api>` subsystem, choosing a path name that is

  less than or equal to 8 bytes can make reads and writes faster.

Dimensioning cache

==================

- When using ZMS API directly, the recommended cache size should be, at least, equal to

  the number of different entries that will be written in the storage.

- Each additional cache entry will add 8 bytes to your RAM usage. Cache size should be carefully

  chosen.

- If you use ZMS through :ref:`Settings <settings_api>`, you have to take into account that each Settings entry is

  divided into two ZMS entries. The recommended cache size should be, at least, twice the number

  of Settings entries.

Sample

******

/*  ZMS: Zephyr Memory Storage

 *

 * Copyright (c) 2024 BayLibre SAS

/* Copyright (c) 2024 BayLibre SAS

 *

 * SPDX-License-Identifier: Apache-2.0

 *

 * ZMS: Zephyr Memory Storage

 */

#ifndef ZEPHYR_INCLUDE_FS_ZMS_H_

#define ZEPHYR_INCLUDE_FS_ZMS_H_

#include <zephyr/drivers/flash.h>

#include <sys/types.h>

#include <zephyr/drivers/flash.h>

#include <zephyr/kernel.h>

#include <zephyr/device.h>

#include <zephyr/toolchain.h>

#endif

/**

 * @brief Zephyr Memory Storage (ZMS)

 * @defgroup zms Zephyr Memory Storage (ZMS)

 * @ingroup file_system_storage

 * @{

 */

/**

 * @brief Zephyr Memory Storage Data Structures

 * @defgroup zms_data_structures Zephyr Memory Storage Data Structures

 * @defgroup zms_data_structures ZMS data structures

 * @ingroup zms

 * @{

 */

/**

 * @brief Zephyr Memory Storage File system structure

 */

/** Zephyr Memory Storage file system structure */

struct zms_fs {

	/** File system offset in flash **/

	/** File system offset in flash */

	off_t offset;

	/** Allocation table entry write address.

	 * Addresses are stored as uint64_t:

	/** Allocation Table Entry (ATE) write address.

	 * Addresses are stored as `uint64_t`:

	 * - high 4 bytes correspond to the sector

	 * - low 4 bytes are the offset in the sector

	 */

	uint64_t ate_wra;

	/** Data write address */

	uint64_t data_wra;

	/** Storage system is split into sectors, each sector size must be multiple of erase-blocks

	 *  if the device has erase capabilities

	/** Storage system is split into sectors. The sector size must be a multiple of

	 *  `erase-block-size` if the device has erase capabilities

	 */

	uint32_t sector_size;

	/** Number of sectors in the file system */

	uint32_t sector_count;

	/** Current cycle counter of the active sector (pointed by ate_wra)*/

	/** Current cycle counter of the active sector (pointed to by `ate_wra`) */

	uint8_t sector_cycle;

	/** Flag indicating if the file system is initialized */

	bool ready;

	/** Mutex */

	/** Mutex used to lock flash writes */

	struct k_mutex zms_lock;

	/** Flash device runtime structure */

	const struct device *flash_device;

	/** Size of an Allocation Table Entry */

	size_t ate_size;

#if CONFIG_ZMS_LOOKUP_CACHE

	/** Lookup table used to cache ATE address of a written ID */

	/** Lookup table used to cache ATE addresses of written IDs */

	uint64_t lookup_cache[CONFIG_ZMS_LOOKUP_CACHE_SIZE];

#endif

};

 */

/**

 * @brief Zephyr Memory Storage APIs

 * @defgroup zms_high_level_api Zephyr Memory Storage APIs

 * @defgroup zms_high_level_api ZMS API

 * @ingroup zms

 * @{

 */

/**

 * @brief Mount a ZMS file system onto the device specified in @p fs.

 * @brief Mount a ZMS file system onto the device specified in `fs`.

 *

 * @param fs Pointer to file system

 * @param fs Pointer to the file system.

 * @retval 0 Success

 * @retval -ERRNO errno code if error

 * @retval -ERRNO Negative errno code on error

 */

int zms_mount(struct zms_fs *fs);

/**

 * @brief Clear the ZMS file system from device.

 *

 * @param fs Pointer to file system

 * @param fs Pointer to the file system.

 * @retval 0 Success

 * @retval -ERRNO errno code if error

 * @retval -ERRNO Negative errno code on error

 */

int zms_clear(struct zms_fs *fs);

/**

 * @brief Write an entry to the file system.

 *

 * @note  When @p len parameter is equal to @p 0 then entry is effectively removed (it is

 * equivalent to calling of zms_delete). It is not possible to distinguish between a deleted

 * @note  When the `len` parameter is equal to `0` the entry is effectively removed (it is

 * equivalent to calling @ref zms_delete()). It is not possible to distinguish between a deleted

 * entry and an entry with data of length 0.

 *

 * @param fs Pointer to file system

 * @param id Id of the entry to be written

 * @param fs Pointer to the file system.

 * @param id ID of the entry to be written

 * @param data Pointer to the data to be written

 * @param len Number of bytes to be written (maximum 64 KB)

 * @param len Number of bytes to be written (maximum 64 KiB)

 *

 * @return Number of bytes written. On success, it will be equal to the number of bytes requested

 * to be written. When a rewrite of the same data already stored is attempted, nothing is written

 * to flash, thus 0 is returned. On error, returns negative value of errno.h defined error codes.

 * to be written or 0.

 * When a rewrite of the same data already stored is attempted, nothing is written to flash,

 * thus 0 is returned. On error, returns negative value of error codes defined in `errno.h`.

 */

ssize_t zms_write(struct zms_fs *fs, uint32_t id, const void *data, size_t len);

/**

 * @brief Delete an entry from the file system

 *

 * @param fs Pointer to file system

 * @param id Id of the entry to be deleted

 * @param fs Pointer to the file system.

 * @param id ID of the entry to be deleted

 * @retval 0 Success

 * @retval -ERRNO errno code if error

 * @retval -ERRNO Negative errno code on error

 */

int zms_delete(struct zms_fs *fs, uint32_t id);

/**

 * @brief Read an entry from the file system.

 *

 * @param fs Pointer to file system

 * @param id Id of the entry to be read

 * @param fs Pointer to the file system.

 * @param id ID of the entry to be read

 * @param data Pointer to data buffer

 * @param len Number of bytes to be read (or size of the allocated read buffer)

 * @param len Number of bytes to read at most

 *

 * @return Number of bytes read. On success, it will be equal to the number of bytes requested

 * to be read. When the return value is less than the number of bytes requested to read this

 * indicates that ATE contain less data than requested. On error, returns negative value of

 * errno.h defined error codes.

 * to be read or less than that if the stored data has a smaller size than the requested one.

 * On error, returns negative value of error codes defined in `errno.h`.

 */

ssize_t zms_read(struct zms_fs *fs, uint32_t id, void *data, size_t len);

/**

 * @brief Read a history entry from the file system.

 *

 * @param fs Pointer to file system

 * @param id Id of the entry to be read

 * @param fs Pointer to the file system.

 * @param id ID of the entry to be read

 * @param data Pointer to data buffer

 * @param len Number of bytes to be read

 * @param cnt History counter: 0: latest entry, 1: one before latest ...

 * @return Number of bytes read. On success, it will be equal to the number of bytes requested

 * to be read. When the return value is larger than the number of bytes requested to read this

 * indicates not all bytes were read, and more data is available. On error, returns negative

 * value of errno.h defined error codes.

 * value of error codes defined in `errno.h`.

 */

ssize_t zms_read_hist(struct zms_fs *fs, uint32_t id, void *data, size_t len, uint32_t cnt);

/**

 * @brief Gets the data size that is stored in an entry with a given id

 * @brief Gets the length of the data that is stored in an entry with a given ID

 *

 * @param fs Pointer to file system

 * @param id Id of the entry that we want to get its data length

 * @param fs Pointer to the file system.

 * @param id ID of the entry whose data length to retrieve.

 *

 * @return Data length contained in the ATE. On success, it will be equal to the number of bytes

 * in the ATE. On error, returns negative value of errno.h defined error codes.

 * in the ATE. On error, returns negative value of error codes defined in `errno.h`.

 */

ssize_t zms_get_data_length(struct zms_fs *fs, uint32_t id);

/**

 * @brief Calculate the available free space in the file system.

 *

 * @param fs Pointer to file system

 * @param fs Pointer to the file system.

 *

 * @return Number of bytes free. On success, it will be equal to the number of bytes that can

 * @return Number of free bytes. On success, it will be equal to the number of bytes that can

 * still be written to the file system.

 * Calculating the free space is a time consuming operation, especially on spi flash.

 * On error, returns negative value of errno.h defined error codes.

 * Calculating the free space is a time-consuming operation, especially on SPI flash.

 * On error, returns negative value of error codes defined in `errno.h`.

 */

ssize_t zms_calc_free_space(struct zms_fs *fs);

/**

 * @brief Tell how many contiguous free space remains in the currently active ZMS sector.

 * @brief Tell how much contiguous free space remains in the currently active ZMS sector.

 *

 * @param fs Pointer to the file system.

 *

 * @return Number of free bytes.

 */

size_t zms_sector_max_data_size(struct zms_fs *fs);

size_t zms_active_sector_free_space(struct zms_fs *fs);

/**

 * @brief Close the currently active sector and switch to the next one.

 * @note The garbage collector is called on the new sector.

 *

 * @warning This routine is made available for specific use cases.

 * It collides with the ZMS goal of avoiding any unnecessary flash erase operations.

 * It collides with ZMS's goal of avoiding any unnecessary flash erase operations.

 * Using this routine extensively can result in premature failure of the flash device.

 *

 * @param fs Pointer to the file system.

 *

 * @return 0 on success. On error, returns negative value of errno.h defined error codes.

 * @return 0 on success. On error, returns negative value of error codes defined in `errno.h`.

 */

int zms_sector_use_next(struct zms_fs *fs);

	int rc = 0;

	char buf[16];

	uint8_t key[8] = {0xDE, 0xAD, 0xBE, 0xEF, 0xDE, 0xAD, 0xBE, 0xEF}, longarray[128];

	uint32_t i_cnt = 0U, i;

	uint32_t i_cnt = 0U;

	uint32_t i;

	uint32_t id = 0;

	ssize_t free_space = 0;

	struct flash_pages_info info;

		rc = zms_read(&fs, KEY_VALUE_ID, &key, sizeof(key));

		if (rc > 0) { /* item was found, show it */

			printk("Id: %x, Key: ", KEY_VALUE_ID);

			for (int n = 0; n < 8; n++) {

			for (uint8_t n = 0; n < 8; n++) {

				printk("%x ", key[n]);

			}

			printk("\n");

		if (rc > 0) {

			/* item was found, show it */

			printk("Id: %d, Longarray: ", LONG_DATA_ID);

			for (int n = 0; n < sizeof(longarray); n++) {

			for (uint16_t n = 0; n < sizeof(longarray); n++) {

				printk("%x ", longarray[n]);

			}

			printk("\n");

	}

	if (i != MAX_ITERATIONS) {

		printk("Error: Something went wrong at iteration %u rc=%d\n", i - 1, rc);

		printk("Error: Something went wrong at iteration %u rc=%d\n", i, rc);

		return 0;

	}

	 * Let's compute free space in storage. But before doing that let's Garbage collect

	 * all sectors where we deleted all entries and then compute the free space

	 */

	for (uint32_t i = 0; i < fs.sector_count; i++) {

	for (i = 0; i < fs.sector_count; i++) {

		rc = zms_sector_use_next(&fs);

		if (rc) {

			printk("Error while changing sector rc=%d\n", rc);

		return 0;

	}

	printk("Free space in storage is %u bytes\n", free_space);

	/* Let's clean the storage now */

	rc = zms_clear(&fs);

	if (rc < 0) {

		printk("Error while cleaning the storage, rc=%d\n", rc);

	}

	printk("Sample code finished Successfully\n");

	return 0;

#Zephyr Memory Storage ZMS

#Copyright (c) 2024 BayLibre SAS

#SPDX-License-Identifier: Apache-2.0

#Zephyr Memory Storage ZMS

config ZMS

	bool "Zephyr Memory Storage"

	select CRC

	help

	  Enables DATA CRC

config ZMS_CUSTOM_BLOCK_SIZE

	bool "Custom buffer size used by ZMS for reads and writes"

config ZMS_CUSTOMIZE_BLOCK_SIZE

	bool "Customize the size of the buffer used internally for reads and writes"

	help

	  ZMS uses internal buffers to read/write and compare stored data.

	  Increasing the size of these buffers should be done carefully in order to not

	  ZMS uses an internal buffer to read/write and compare stored data.

	  Increasing the size of this buffer should be done carefully in order to not

	  overflow the stack.

	  Increasing this buffer means as well that ZMS could work with storage devices

	  that have larger write-block-size which decreases ZMS performance

config ZMS_MAX_BLOCK_SIZE

config ZMS_CUSTOM_BLOCK_SIZE

	int "ZMS internal buffer size"

	default 32

	depends on ZMS_CUSTOM_BLOCK_SIZE

	depends on ZMS_CUSTOMIZE_BLOCK_SIZE

	help

	  Changes the internal buffer size of ZMS