Store
Encoding

Encoding

Store uses a custom encoding scheme to store data more compactly than Solidity. It is comparable to abi.encodePacked, but with some notable differences:

  • Array elements are tightly packed, without any padding. This might cause some elements to wrap around two storage slots, but saves a lot of space. For example, an address[3] array will only use 2 storage slots in MUD instead of 3 in Solidity, leading to a 33% reduction in storage costs.
  • Array lengths are packed into a single slot. Since table are limited to up to 5 dynamic field, and the length of each field is limited to 2**40 (~1 trillion), we can pack the lengths of all dynamic length fields of a table into a single storage slot. Compared to Solidity, this saves ~1 storage slot per dynamic length element.

This encoding scheme greatly reduces gas for storage operations on dynamic length fields and for emitting events. For events it reduces the payload size by ~80% compared to the Solidity default of abi.encode (depending on the table schema). Note that since the encoding happens at runtime and does not have access to compiler internals, there is some overhead for encoding and decoding that is not present in vanilla Solidity.

Schema

Each table has a key schema and a value schema, which define the data types of each column.

Both key schema and value schema use the same data structure: a bytes32 value that encodes the total byte length of all static length fields, the number of static length fields, the number of dynamic length fields, and up to 28 column types.

Note that key schemas can only include static length types, so the value for "number of dynamic length fields" is always set to zero in this case.

For value schemas, up to five of the 28 columns can have dynamic length types. The dynamic length types must come after all static length types.

Types are represented via the SchemaType enum and encoded as one byte per type.

@latticexyz/schema-type/src/solidity/SchemaType.sol
enum SchemaType {
  UINT8,
  UINT16,
  UINT24,
  UINT32,
  UINT40,
  UINT48,
  UINT56,
  UINT64,
  UINT72,
  UINT80,
  UINT88,
  UINT96,
  UINT104,
  UINT112,
  UINT120,
  UINT128,
  UINT136,
  UINT144,
  UINT152,
  UINT160,
  UINT168,
  UINT176,
  UINT184,
  UINT192,
  UINT200,
  UINT208,
  UINT216,
  UINT224,
  UINT232,
  UINT240,
  UINT248,
  UINT256,
  INT8,
  INT16,
  INT24,
  INT32,
  INT40,
  INT48,
  INT56,
  INT64,
  INT72,
  INT80,
  INT88,
  INT96,
  INT104,
  INT112,
  INT120,
  INT128,
  INT136,
  INT144,
  INT152,
  INT160,
  INT168,
  INT176,
  INT184,
  INT192,
  INT200,
  INT208,
  INT216,
  INT224,
  INT232,
  INT240,
  INT248,
  INT256,
  BYTES1,
  BYTES2,
  BYTES3,
  BYTES4,
  BYTES5,
  BYTES6,
  BYTES7,
  BYTES8,
  BYTES9,
  BYTES10,
  BYTES11,
  BYTES12,
  BYTES13,
  BYTES14,
  BYTES15,
  BYTES16,
  BYTES17,
  BYTES18,
  BYTES19,
  BYTES20,
  BYTES21,
  BYTES22,
  BYTES23,
  BYTES24,
  BYTES25,
  BYTES26,
  BYTES27,
  BYTES28,
  BYTES29,
  BYTES30,
  BYTES31,
  BYTES32,
  BOOL,
  ADDRESS,
  UINT8_ARRAY,
  UINT16_ARRAY,
  UINT24_ARRAY,
  UINT32_ARRAY,
  UINT40_ARRAY,
  UINT48_ARRAY,
  UINT56_ARRAY,
  UINT64_ARRAY,
  UINT72_ARRAY,
  UINT80_ARRAY,
  UINT88_ARRAY,
  UINT96_ARRAY,
  UINT104_ARRAY,
  UINT112_ARRAY,
  UINT120_ARRAY,
  UINT128_ARRAY,
  UINT136_ARRAY,
  UINT144_ARRAY,
  UINT152_ARRAY,
  UINT160_ARRAY,
  UINT168_ARRAY,
  UINT176_ARRAY,
  UINT184_ARRAY,
  UINT192_ARRAY,
  UINT200_ARRAY,
  UINT208_ARRAY,
  UINT216_ARRAY,
  UINT224_ARRAY,
  UINT232_ARRAY,
  UINT240_ARRAY,
  UINT248_ARRAY,
  UINT256_ARRAY,
  INT8_ARRAY,
  INT16_ARRAY,
  INT24_ARRAY,
  INT32_ARRAY,
  INT40_ARRAY,
  INT48_ARRAY,
  INT56_ARRAY,
  INT64_ARRAY,
  INT72_ARRAY,
  INT80_ARRAY,
  INT88_ARRAY,
  INT96_ARRAY,
  INT104_ARRAY,
  INT112_ARRAY,
  INT120_ARRAY,
  INT128_ARRAY,
  INT136_ARRAY,
  INT144_ARRAY,
  INT152_ARRAY,
  INT160_ARRAY,
  INT168_ARRAY,
  INT176_ARRAY,
  INT184_ARRAY,
  INT192_ARRAY,
  INT200_ARRAY,
  INT208_ARRAY,
  INT216_ARRAY,
  INT224_ARRAY,
  INT232_ARRAY,
  INT240_ARRAY,
  INT248_ARRAY,
  INT256_ARRAY,
  BYTES1_ARRAY,
  BYTES2_ARRAY,
  BYTES3_ARRAY,
  BYTES4_ARRAY,
  BYTES5_ARRAY,
  BYTES6_ARRAY,
  BYTES7_ARRAY,
  BYTES8_ARRAY,
  BYTES9_ARRAY,
  BYTES10_ARRAY,
  BYTES11_ARRAY,
  BYTES12_ARRAY,
  BYTES13_ARRAY,
  BYTES14_ARRAY,
  BYTES15_ARRAY,
  BYTES16_ARRAY,
  BYTES17_ARRAY,
  BYTES18_ARRAY,
  BYTES19_ARRAY,
  BYTES20_ARRAY,
  BYTES21_ARRAY,
  BYTES22_ARRAY,
  BYTES23_ARRAY,
  BYTES24_ARRAY,
  BYTES25_ARRAY,
  BYTES26_ARRAY,
  BYTES27_ARRAY,
  BYTES28_ARRAY,
  BYTES29_ARRAY,
  BYTES30_ARRAY,
  BYTES31_ARRAY,
  BYTES32_ARRAY,
  BOOL_ARRAY,
  ADDRESS_ARRAY,
  BYTES,
  STRING
}

Example: Schema with 3 fields (uint64, uint40, address[]).

0x 000D 02 01 0704c300000000000000000000000000000000000000000000000000
  |----|--|--|--------------------------------------------------------|
  |    |  |  | Up to 28 column types, one byte each                   |
  |    |  |  | UINT64, UINT40, ADDRESS_ARRAY                          |
  |----|--|--|--------------------------------------------------------|
  |    |  | Number of dynamic length fields                           |
  |    |  | 1                                                         |
  |----|--|--|--------------------------------------------------------|
  |    | Number of static length fields                               |
  |    | 2                                                            |
  |----|--|--|--------------------------------------------------------|
  | Total byte length of all static length fields                     |
  | 13                                                                |

Static length data

Static length data is encoded in the same way as abi.encodePacked, so each element only uses as little space as required by its data type. For example, a bytes1 element takes 1 byte of space, instead of being padded to 32 bytes with abi.encode.

Example: encoded static data of a table with a uint64 and a uint40 field.

0x 0000000000000001 0000000002
  |----------------|----------|
  |0x00........0x07|0x08..0x0B|
  |----------------|----------|
  | First          | Second   |
  | Field          | Field    |
  |----------------|----------|
  | 1              | 2        |

Dynamic length data

Encoded lengths (PackedCounter)

Tables can have up to five dynamic length fields, each of which has a maximum size of 2**40 (~1 trillion) bytes. This makes it possible to pack the lengths of all dynamic length fields of a table in a single bytes32 word, instead of prefixing each dynamic length field with 32 bytes to store its length.

The data structure to store the lengths of all dynamic length fields is called PackedCounter.

Example: PackedCounter with 5 fields.

0x 0000000000000F 0000000001 0000000002 0000000003 0000000004 0000000005
  |--------------|----------|----------|----------|----------|----------|
  |0x00......0x06|0x07..0x0B|0x0C..0x10|0x11..0x15|0x16..0x1A|0x1B..0x1F|
  |--------------|----------|----------|----------|----------|----------|
  | Total        | First    | Second   | Third    | Fourth   | Fifth    |
  | length       | length   | length   | length   | length   | length   |
  |--------------|----------|----------|----------|----------|----------|
  | 15           | 1        | 2        | 3        | 4        | 5        |

Dynamic length data encoding

Elements of dynamic length fields are tightly packed in storage and events. This is comparable to Solidity's storage layout, except there is no guarantee for elements to be aligned with storage slots, which allows arrays to be packed more tightly and without padding. For example, an address[3] array will only use 2 storage slots in MUD instead of 3 in Solidity. (For context: Solidity only packs elements of arrays if each element is smaller than 16 bytes, and applies padding to guarantee alignment with storage slots.)

Example: encoded dynamic data of a table with an address[] field.


0x 1000000000000000000000000000000000000002 3000000000000000000000000000000000000004 5000000000000000000000000000000000000006
  |----------------------------------------|----------------------------------------|----------------------------------------|
  |0x00................................0x13|0x14................................0x27|0x28................................0x3B|
  |----------------------------------------|----------------------------------------|----------------------------------------|
  | address_array[0]                       | address_array[1]                       | address_array[2]                       |
  |----------------------------------------|----------------------------------------|----------------------------------------|
  | 0x10000000000000000000000000000000..02 | 0x30000000000000000000000000000000..04 | 0x50000000000000000000000000000000..06 |

Field layout

Similar to Schema, the FieldLayout data structure is a bytes32 value that encodes the total byte length of all static length fields, the number of static length fields, the number of dynamic length fields. However, instead of encoding the column types in the remaining 28 bytes, it encodes byte size of each static length field.

It is used as an onchain gas optimization for storage operations on static length fields. In onchain storage operations only the byte length of the field is required, not the type, so using FieldLayout instead of Schema saves the gas for translating the static length type to its byte length.

Example: FieldLayout with 3 fields (uint64, uint40, address[]).

0x 000D 02 01 08050000000000000000000000000000000000000000000000000000
  |----|--|--|--------------------------------------------------------|
  |    |  |  | Up to 28 static byte lengths, one byte each            |
  |    |  |  | 8 (uint64), 5 (uint40)                  |
  |----|--|--|--------------------------------------------------------|
  |    |  | Number of dynamic length fields                           |
  |    |  | 1                                                         |
  |----|--|-----------------------------------------------------------|
  |    | Number of static length fields                               |
  |    | 2                                                            |
  |----|--------------------------------------------------------------|
  | Total byte length of all static length fields                     |
  | 13                                                                |

Storage layout

The data of each row of each table is stored in its own location in storage.

The location of the static length data is determined by a hash of the table ID and the key tuple. Since data is tightly packed, fields may wrap around two storage slots.

function _getStaticDataLocation(ResourceId tableId, bytes32[] memory keyTuple) internal pure returns (uint256) {
  return uint256(SLOT ^ keccak256(abi.encodePacked(tableId, keyTuple)));
}

The encoded lengths of all dynamic length fields is stored in a single storage slot, which is determined by a hash of the table ID and the key tuple.

function _getDynamicDataLengthLocation(ResourceId tableId, bytes32[] memory keyTuple) internal pure returns (uint256) {
  return uint256(DYNAMIC_DATA_LENGTH_SLOT ^ keccak256(abi.encodePacked(tableId, keyTuple)));
}

The data for each dynamic length field is stored in its own location in storage, which is determined by a hash of the table ID, the key tuple, and the field index. This allows changing the length of a dynamic length field without having to move the data of other fields.

function _getDynamicDataLocation(
  ResourceId tableId,
  bytes32[] memory keyTuple,
  uint8 dynamicFieldIndex
) internal pure returns (uint256) {
  return uint256(DYNAMIC_DATA_SLOT ^ bytes1(dynamicFieldIndex) ^ keccak256(abi.encodePacked(tableId, keyTuple)));
}

Events

Data emitted as part of events uses the same encoding for static length data, encoded lengths and dynamic length data as described above.

Unlike the layout in storage, where each dynamic length data field is stored in its own storage location, in events all data of dynamic length fields is concatenated to a single bytes blob. The encoded lengths emitted as part of the event contain the information required to slice up this concatenated bytes blob into the individual fields.

This encoding reduces the event payload size by ~80% compared to default Solidity events, depending on the table schema. (For context: Solidity uses abi.encode to encode the event payload, which pads each static length field to 32 bytes, and pads each item of dynamic length fields to 32 bytes.)

More details on the Store event signatures can be found in the API reference.

Table LibrariesStore Hooks