binobj.fields.base module

Base classes and definitions common to all Fields.

DEFAULT = _Default.token

A sentinel value used to indicate that the default value of a setting should be used.

We need this because sometimes None is a valid value for that setting.

class Field(*_args: Any, **kwargs: Any)

Bases: Generic[T]

The base class for all struct fields.

Parameters:

  • name (str) –

    The name of the field.

    Changed in version 0.11.0: Passing a value for this will throw a ConfigurationError for fields declared in a normal class if it doesn’t match the existing name. Only use this argument if you’re building a struct programmatically.

  • const –

    A constant value this field is expected to take. It will always have this value when dumped, and will fail validation if the field isn’t this value when loaded. Useful for reserved fields and file tags.

    This argument must be of the same type as the field, i.e. it must be a string for a String, an integer for an Integer, and so on.

  • default –

    The default value to use if a value for this field isn’t passed to the struct for serialization, (deprecated) or a callable taking no arguments that will return a default value.

    This argument must be of the same type as the field, i.e. it must be a string for a String, an integer for an Integer, and so on.

    Deprecated since version 0.11.0: Do not pass a factory function to this argument. Use factory instead.

  • factory (callable) – A callable taking no arguments that returns a default value for this field.

  • discard (bool) –

    When deserializing, don’t include this field in the returned results. This means that you won’t be able to use the value for anything later. For example, if you need to reference it in a present function like so:

    name_size = fields.UInt16(discard=True)
filename = fields.StringZ(encoding="utf-8")
_filename_padding = fields.Bytes(
    const=b"\0", discard=True, present=lambda f, *_: f["name_size"] % 2
)

    this will crash with a KeyError because name_size was discarded.

  • null_value –

    Either a byte string or a value to use to represent None in serialized data.

    When loading, the returned value will be None if this exact sequence of bytes is encountered. If not given, the field is considered “not nullable” and any attempt to assign None to it will result in a crash upon serialization.

  • size –

    Optional. The size of the field. This can be a number of things:

    • An integer constant. The field will always be the same size, no matter what value is given to it.

    • Another Field object. That field gives the size of this field, in bytes.

    • A string naming another field. It’s equivalent to passing in a Field instance, except used for references in the same class or a field defined in the superclass.

  • validate – A callable or list of callables that validates a given value for this field. The callable(s) will always be passed the deserialized value, so a validator for an Integer field will always be passed an integer, a String validator will always be passed a string, and so on.

  • present (callable) –

    Optional. A callable that, when called with the struct as its argument, returns a boolean indicating if this field is “present” and should be loaded or dumped. For example, if we have a flags field that’s a bitmap indicating what fields come next, we could have something like this:

    flags = fields.UInt16()
foo = fields.StringZ(present=lambda v, *_: v["flags"] & 0x8000)
bar = fields.StringZ(present=lambda v, *_: v["flags"] & 0x4000)

    Thus, if and only if flags has bit 15 set, foo will be read from the stream next. If flags has bit 15 clear, foo will be assigned the field’s not_present_value (defaults to NOT_PRESENT).

    The callable takes three positional arguments:

    • A dict of the fields that have already been loaded or are about to be dumped.

    • The context object passed to from_stream() or to_stream().

    • When loading, the stream being loaded from. The stream pointer MUST be reset to its original position before the function returns.

  • not_present_value – A custom value to return if a field is missing when loading (see the present argument). It can be None or match the datatype of the field, i.e. a string for a String, an integer for an Integer, and so on. If not given, defaults to NOT_PRESENT.

index

The zero-based index of the field in the struct.

Type:

int

offset

The zero-based byte offset of the field in the struct. If the offset can’t be computed (e.g. it’s preceded by a variable-length field), this will be None.

Type:

int

New in version 0.4.5: The present argument.

Changed in version 0.8.0: This now inherits from typing.Generic.

New in version 0.9.0.

  • The not_present_value argument.

  • size has full support for Fields and field name values. This used to be only supported by some fields, with others left out by accident.

Changed in version 0.9.0: null_value can now also be a deserialized value. For example, you could pass r”\N” for a string or 0 for an integer.

Deprecated since version 0.9.0: Passing DEFAULT to null_value for unsized fields such as StringZ is deprecated and will trigger an error in the future. This resolves the asymmetric behavior where using DEFAULT throws an error when dumping but happily loads whatever’s next in the stream when loading.

New in version 0.11.0: The factory argument.

Deprecated since version 0.11.0: Passing a factory function to default is now deprecated. Use factory instead.

property allow_null: bool

Indicate if None an acceptable value for this field.

Type:

bool

bind_to_container(struct_info: StructMetadata, name: str, index: int, offset: int | None = None) → None

Bind this field to a Struct and apply any predefined defaults.

Parameters:

  • struct_info (binobj.structures.StructMetadata) – The metadata object describing the Struct this field will be bound into.

  • name (str) – The name of this field.

  • index (int) – The index of this field in the container.

  • offset (int) – The byte offset of this field in the container, or None if unknown. This is usually equal to the sum of the sizes of the fields preceding this one in the container.

Changed in version 0.10.0: Added the struct_info parameter.

compute_value_for_dump(all_values: Mapping[str, Any], context: Any | None = None) → T | None | _NotPresent

Calculate the value for this field upon dumping.

Parameters:

  • all_values (dict) – The dictionary of all the field data that’s about to be dumped.

  • context –

    The context object passed to the containing Struct’s to_bytes() or to_stream() method.

    New in version 0.11.0.

Returns:

The value the dumper will use for this field, or NOT_PRESENT if the field shouldn’t be serialized. It will not return not_present_value in this case, as the field should not be dumped at all.

Raises:

MissingRequiredValueError – No value could be derived for this field. It’s missing in the input data, there’s no default defined, and it doesn’t have a compute function defined either.

New in version 0.3.1.

Changed in version 0.8.0: If default is given by a callable and that callable returns UNDEFINED, it will throw MissingRequiredValueError instead of returning UNDEFINED.

Changed in version 0.11.0: * The context argument was added, and is now passed to the present callable. * present() is now always called, even if the value of the field is set. Before, if a field had a value explicitly set, it would be included in the output even if present() would’ve returned False.

computes(method: Callable[[Field[T], Mapping[str, Any]], T | None]) → None

Decorator that marks a function as computing the value for a field.

You can use this for automatically assigning values based on other fields. For example, suppose we have this struct:

class MyStruct(Struct):
    n_numbers = UInt8()
    numbers = Array(UInt8(), count=n_numbers)

This works great for loading, but when we’re dumping we have to pass in a value for n_numbers explicitly. We can use the computes decorator to relieve us of that burden:

class MyStruct(Struct):
    n_numbers = UInt8()
    numbers = Array(UInt8(), count=n_numbers)

    @n_numbers.computes
    def _assign_n_numbers(self, all_fields):
        return len(all_fields['numbers'])

Some usage notes:

  • The computing function will not be called if

    • A value is explicitly set for the field by the calling code.

    • The field has a default or const value.

  • Computed fields are executed in the order that the fields are dumped, so a computed field must not rely on the value of another computed field occurring after it.

New in version 0.3.0.

const: T | _Undefined

The fixed value of a field, if applicable.

This is mostly useful for fields that act as magic numbers or reserved fields in a struct that should be set to nulls.

property default: T | None | _Undefined

The default value of this field, or UNDEFINED.

Changed in version 0.6.1: If no default is defined but const is, this property returns the value for const.

discard: bool

If True, indicates that a field should be discarded when read.

This is best used for filler fields that are of no use to the application but are nonetheless important to ensure the proper layout of the struct.

from_bytes(data: bytes, context: Any | None = None, exact: bool = True, loaded_fields: Mapping[str, Any] | None = None) → T | None | _NotPresent

Load from the given byte string.

Parameters:

  • data (bytes) – A bytes-like object to get the data from.

  • context – Additional data to pass to this method. Subclasses must ignore anything they don’t recognize.

  • exact (bool) – data must contain exactly the number of bytes required. If not all the bytes in data were used when reading the struct, throw an exception.

  • loaded_fields (dict) – A dictionary of the fields that have already been loaded. This is set automatically when a field is loaded by a Struct.

Returns:

The deserialized data, or NOT_PRESENT if the field is missing.

from_stream(stream: BinaryIO, context: Any | None = None, loaded_fields: Mapping[str, Any] | None = None) → T | None | _NotPresent

Load data from the given stream.

Parameters:

  • stream (BinaryIO) – The stream to load data from.

  • context – Additional data to pass to this method. Subclasses must ignore anything they don’t recognize.

  • loaded_fields (dict) – A dictionary of the fields that have already been loaded. This is set automatically when a field is loaded by a Struct.

Returns:

The deserialized data, or NOT_PRESENT

get_expected_size(field_values: Mapping[str, Any]) → int

Determine the size of this field in bytes, given values for other fields.

Parameters:

field_values (dict) – A dict mapping field names to their resolved values.

Returns:

The number of bytes this field is expected to occupy.

Return type:

int

Raises:

  • MissingRequiredValueError – The field’s size references another field but the other field is missing from field_values.

  • UndefinedSizeError – The field doesn’t have a defined size nor refers to another field to determine its size.

Changed in version 0.9.0: This used to be a private method. _get_expected_size() is still present for compatibility, but it will eventually be removed.

property has_fixed_size: bool

Does this field have a fixed size?

New in version 0.9.0.

property is_computed_field: bool

Indicate if this field is computed from the value of other fields.

Computed fields cannot have their values set directly. Attempting to do so will throw an ImmutableFieldError.

name: str

The name of the field.

Technical Note: This attribute can only be None if the field was created without passing a value for name to the constructor and the field has never been bound to a struct. Since this is highly unlikely in normal usage, this attribute is declared as str rather than Optional[str].

property required: bool

Indicates if this field is required for serialization.

Type:

bool

property size: int | str | Field[int] | None

The size of this field, in bytes.

If the field is of variable size, such as a null-terminated string, this will be None. Builtin fields set this automatically if const is given but you’ll need to implement _size_for_value() in custom fields.

to_bytes(data: T | None | _Default = _Default.token, context: Any | None = None, all_fields: Mapping[str, Any] | None = None) → bytes

Convert the given data into bytes.

Parameters:

  • data – The data to dump. Can be omitted only if this is a constant field or a default value is defined.

  • context – Additional data to pass to this method. Subclasses must ignore anything they don’t recognize.

  • all_fields (dict) – A dictionary of the fields about to be dumped. This is automatically set by the field’s containing Struct.

Returns:

The serialized data.

Return type:

bytes

to_stream(stream: BinaryIO, data: T | None | _Default = _Default.token, context: Any | None = None, all_fields: Mapping[str, Any] | None = None) → None

Convert the given data into bytes and write it to stream.

Parameters:

  • stream (BinaryIO) – The stream to write the serialized data into.

  • data – The data to dump. Can be omitted only if this is a constant field or if a default value is defined.

  • context – Additional data to pass to this method. Subclasses must ignore anything they don’t recognize.

  • all_fields (dict) – A dictionary of the fields about to be dumped. This is automatically set by the field’s containing Struct.

NOT_PRESENT = _NotPresent.token

A sentinel value used to indicate that a field is not present.

New in version 0.4.5.

UNDEFINED = _Undefined.token

A sentinel value used to indicate that a setting or field is undefined.