Table of Contents
Shortcuts

torchfunc.hooks.recorders

This module allows one to record neural network state (for example when data passes through it).

recorders are organized similarly to torch.nn.Module’s hooks (e.g. backward, forward and forward pre). Additionally, each can record input or output from specified modules, which gives us, for example, ForwardInput (record input to specified module(s) during forward pass).

Example should make it more clear:

# MNIST classifier
model = torch.nn.Sequential(
    torch.nn.Linear(784, 100),
    torch.nn.ReLU(),
    torch.nn.Linear(100, 50),
    torch.nn.ReLU(),
    torch.nn.Linear(50, 10),
)

# Recorder which sums layer inputs from consecutive forward calls
recorder = torchfunc.hooks.recorders.ForwardPre(reduction=lambda x, y: x+y)
# Record inputs going into Linear(100, 50) and Linear(50, 10)
recorder.children(model, indices=(2, 3))
# Train your network normally (pass data through it somehow)
...
# Save tensors (of shape 100 and 50) in folder, each named 1.pt and 2.pt respectively
recorder.save(pathlib.Path("./analysis"))

You could specify types instead of indices (for example all forward inputs to torch.nn.Linear will be registered), iterate over modules recursively instead of shallow iteration with children method etc.

Each recorder has one or more subrecorders; those usually correspond to specific layer for which recording will be done. In the above case, there are two subrecorders, both of torch.nn.Linear type.

Additionally one can post-process data contained within recorder using apply functionality.

Concrete methods recording different data passing through network are specified below:

class torchfunc.hooks.recorders.BackwardInput(condition: Callable = None, reduction: Callable = None)[source]

Record input gradients after those are calculated w.r.t. specified module.

You can record only some of the data based on external conditions if condition callable is specified.

Data can be cumulated together via reduction parameter, which is advised from the memory perspective.

Parameters

  • condition (Callable, optional) – No argument callable. If True returned, record data. Can be used to save data based on external environment (e.g. dataset’s label). If None, will record every data point. Default: None

  • reduction (Callable, optional) – Operation to use on incoming data. Should take two arguments, and return one. Acts similarly to reduction argument of Python’s functools.reduce. If None, data will be added to list, which may be very memory intensive. Default: None

data

Keeps data passing through subrecorders, optionally reduced by reduction. Each item represents data for specified subrecorder.

Type

List

subrecorders

List containing registered subrecorders.

Type

List[Hooks]

handles

Handles for registered subrecorders, each corresponds to specific subrecorder. Can be used to unregister certain subrecorders (though discouraged, please use remove method).

Type

List[torch.utils.subrecorders.RemovableHandle]

apply(function: Callable)

Apply function to data contained in each subrecorder.

Data will be modified an saved inside data of each subrecorder. This function may make recorder unusable, it’s up to user to ensure correct functioning after this functionality was used.

Parameters

function (Callable) – Single argument (torch.Tensor data from subrecorder) callable returning anything.

apply_sample(function: Callable) → None

Apply function to data contained in each subrecorder.

Works like apply, except Callable is passed number of samples passed through subrecorder as second argument

Parameters

function (Callable) – Two argument (torch.Tensor data from subrecorder and number of samples which passed through it) Callable returning anything.

children(network, types: Tuple[Any] = None, indices: List[int] = None)

Register subrecorders using types and/or indices via children method.

This function will use children method of torch.nn.Module to iterate over available submodules. If you wish to iterate recursively, use modules.

Important:

If types and indices are left with their default values, all modules will have subrecorders registered.

Parameters

  • module (torch.nn.Module) – Module (usually neural network) for which inputs will be collected.

  • types (Tuple[typing.Any], optional) – Module types for which data will be recorded. E.g. (torch.nn.Conv2d, torch.nn.Linear) will register subrecorders on every module being instance of either Conv2d or Linear. Default: None

  • indices (Iterable[int], optional) – Indices of modules whose inputs will be registered. Default: None

Returns

Return type

self

iter_samples()

Iterate over count of samples for each subrecorder.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

How many samples passed through this subrecorder.

Return type

int

modules(module: torch.nn.modules.module.Module, types: Tuple[Any] = None, indices: List[int] = None)

Register subrecorders using types and/or indices via modules method.

This function will use modules method of torch.nn.Module to iterate over available submodules. If you wish to iterate non-recursively, use children.

Important:

If types and indices are left with their default values, all modules will have subrecorders registered.

Parameters

  • module (torch.nn.Module) – Module (usually neural network) for which inputs will be collected.

  • types (Tuple[typing.Any], optional) – Module types for which data will be recorded. E.g. (torch.nn.Conv2d, torch.nn.Linear) will register subrecorders on every module being instance of either Conv2d or Linear. Default: None

  • indices (Iterable[int], optional) – Indices of modules whose inputs will be registered. Default: None

Returns

Return type

self

remove(index)

Remove subrecorder specified by index.

Subrecorder will not record data passing through it and will be removed from subrecorders attribute.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

Data contained in subrecorder.

Return type

torch.Tensor

samples(index) → int

Count of samples passed through subrecorder under index.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

How many samples passed through specified subrecorder.

Return type

int

save(path: pathlib.Path, mkdir: bool = False, *args, **kwargs)

Save data tensors within specified path.

Each data tensor will be indexed by integer [0, N), where indices represent consecutive subrecorders.

Parameters

  • path (pathlib.Path) – Path where tensors will be saved.

  • mkdir (bool, optional) – If True, create directory if doesn’t exists. Default: False

  • *args – Varargs passed to pathlib.Path’s mkdir method if mkdir argument set to True.

  • *kwargs – Kwarargs passed to pathlib.Path’s mkdir method if mkdir argument set to True.

class torchfunc.hooks.recorders.BackwardOutput(condition: Callable = None, reduction: Callable = None)[source]

Record output gradients after those are calculated w.r.t. specified module.

You can record only some of the data based on external conditions if condition callable is specified.

Data can be cumulated together via reduction parameter, which is advised from the memory perspective.

Parameters

  • condition (Callable, optional) – No argument callable. If True returned, record data. Can be used to save data based on external environment (e.g. dataset’s label). If None, will record every data point. Default: None

  • reduction (Callable, optional) – Operation to use on incoming data. Should take two arguments, and return one. Acts similarly to reduction argument of Python’s functools.reduce. If None, data will be added to list, which may be very memory intensive. Default: None

data

Keeps data passing through subrecorders, optionally reduced by reduction. Each item represents data for specified subrecorder.

Type

List

subrecorders

List containing registered subrecorders.

Type

List[Hooks]

handles

Handles for registered subrecorders, each corresponds to specific subrecorder. Can be used to unregister certain subrecorders (though discouraged, please use remove method).

Type

List[torch.utils.subrecorders.RemovableHandle]

apply(function: Callable)

Apply function to data contained in each subrecorder.

Data will be modified an saved inside data of each subrecorder. This function may make recorder unusable, it’s up to user to ensure correct functioning after this functionality was used.

Parameters

function (Callable) – Single argument (torch.Tensor data from subrecorder) callable returning anything.

apply_sample(function: Callable) → None

Apply function to data contained in each subrecorder.

Works like apply, except Callable is passed number of samples passed through subrecorder as second argument

Parameters

function (Callable) – Two argument (torch.Tensor data from subrecorder and number of samples which passed through it) Callable returning anything.

children(network, types: Tuple[Any] = None, indices: List[int] = None)

Register subrecorders using types and/or indices via children method.

This function will use children method of torch.nn.Module to iterate over available submodules. If you wish to iterate recursively, use modules.

Important:

If types and indices are left with their default values, all modules will have subrecorders registered.

Parameters

  • module (torch.nn.Module) – Module (usually neural network) for which inputs will be collected.

  • types (Tuple[typing.Any], optional) – Module types for which data will be recorded. E.g. (torch.nn.Conv2d, torch.nn.Linear) will register subrecorders on every module being instance of either Conv2d or Linear. Default: None

  • indices (Iterable[int], optional) – Indices of modules whose inputs will be registered. Default: None

Returns

Return type

self

iter_samples()

Iterate over count of samples for each subrecorder.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

How many samples passed through this subrecorder.

Return type

int

modules(module: torch.nn.modules.module.Module, types: Tuple[Any] = None, indices: List[int] = None)

Register subrecorders using types and/or indices via modules method.

This function will use modules method of torch.nn.Module to iterate over available submodules. If you wish to iterate non-recursively, use children.

Important:

If types and indices are left with their default values, all modules will have subrecorders registered.

Parameters

  • module (torch.nn.Module) – Module (usually neural network) for which inputs will be collected.

  • types (Tuple[typing.Any], optional) – Module types for which data will be recorded. E.g. (torch.nn.Conv2d, torch.nn.Linear) will register subrecorders on every module being instance of either Conv2d or Linear. Default: None

  • indices (Iterable[int], optional) – Indices of modules whose inputs will be registered. Default: None

Returns

Return type

self

remove(index)

Remove subrecorder specified by index.

Subrecorder will not record data passing through it and will be removed from subrecorders attribute.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

Data contained in subrecorder.

Return type

torch.Tensor

samples(index) → int

Count of samples passed through subrecorder under index.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

How many samples passed through specified subrecorder.

Return type

int

save(path: pathlib.Path, mkdir: bool = False, *args, **kwargs)

Save data tensors within specified path.

Each data tensor will be indexed by integer [0, N), where indices represent consecutive subrecorders.

Parameters

  • path (pathlib.Path) – Path where tensors will be saved.

  • mkdir (bool, optional) – If True, create directory if doesn’t exists. Default: False

  • *args – Varargs passed to pathlib.Path’s mkdir method if mkdir argument set to True.

  • *kwargs – Kwarargs passed to pathlib.Path’s mkdir method if mkdir argument set to True.

class torchfunc.hooks.recorders.ForwardInput(condition: Callable = None, reduction: Callable = None)[source]

Record input values after forward of specified layer(s).

You can record only some of the data based on external conditions if condition callable is specified.

Data can be cumulated together via reduction parameter, which is advised from the memory perspective.

Parameters

  • condition (Callable, optional) – No argument callable. If True returned, record data. Can be used to save data based on external environment (e.g. dataset’s label). If None, will record every data point. Default: None

  • reduction (Callable, optional) – Operation to use on incoming data. Should take two arguments, and return one. Acts similarly to reduction argument of Python’s functools.reduce. If None, data will be added to list, which may be very memory intensive. Default: None

data

Keeps data passing through subrecorders, optionally reduced by reduction. Each item represents data for specified subrecorder.

Type

List

subrecorders

List containing registered subrecorders.

Type

List[Hooks]

handles

Handles for registered subrecorders, each corresponds to specific subrecorder. Can be used to unregister certain subrecorders (though discouraged, please use remove method).

Type

List[torch.utils.subrecorders.RemovableHandle]

apply(function: Callable)

Apply function to data contained in each subrecorder.

Data will be modified an saved inside data of each subrecorder. This function may make recorder unusable, it’s up to user to ensure correct functioning after this functionality was used.

Parameters

function (Callable) – Single argument (torch.Tensor data from subrecorder) callable returning anything.

apply_sample(function: Callable) → None

Apply function to data contained in each subrecorder.

Works like apply, except Callable is passed number of samples passed through subrecorder as second argument

Parameters

function (Callable) – Two argument (torch.Tensor data from subrecorder and number of samples which passed through it) Callable returning anything.

children(network, types: Tuple[Any] = None, indices: List[int] = None)

Register subrecorders using types and/or indices via children method.

This function will use children method of torch.nn.Module to iterate over available submodules. If you wish to iterate recursively, use modules.

Important:

If types and indices are left with their default values, all modules will have subrecorders registered.

Parameters

  • module (torch.nn.Module) – Module (usually neural network) for which inputs will be collected.

  • types (Tuple[typing.Any], optional) – Module types for which data will be recorded. E.g. (torch.nn.Conv2d, torch.nn.Linear) will register subrecorders on every module being instance of either Conv2d or Linear. Default: None

  • indices (Iterable[int], optional) – Indices of modules whose inputs will be registered. Default: None

Returns

Return type

self

iter_samples()

Iterate over count of samples for each subrecorder.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

How many samples passed through this subrecorder.

Return type

int

modules(module: torch.nn.modules.module.Module, types: Tuple[Any] = None, indices: List[int] = None)

Register subrecorders using types and/or indices via modules method.

This function will use modules method of torch.nn.Module to iterate over available submodules. If you wish to iterate non-recursively, use children.

Important:

If types and indices are left with their default values, all modules will have subrecorders registered.

Parameters

  • module (torch.nn.Module) – Module (usually neural network) for which inputs will be collected.

  • types (Tuple[typing.Any], optional) – Module types for which data will be recorded. E.g. (torch.nn.Conv2d, torch.nn.Linear) will register subrecorders on every module being instance of either Conv2d or Linear. Default: None

  • indices (Iterable[int], optional) – Indices of modules whose inputs will be registered. Default: None

Returns

Return type

self

remove(index)

Remove subrecorder specified by index.

Subrecorder will not record data passing through it and will be removed from subrecorders attribute.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

Data contained in subrecorder.

Return type

torch.Tensor

samples(index) → int

Count of samples passed through subrecorder under index.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

How many samples passed through specified subrecorder.

Return type

int

save(path: pathlib.Path, mkdir: bool = False, *args, **kwargs)

Save data tensors within specified path.

Each data tensor will be indexed by integer [0, N), where indices represent consecutive subrecorders.

Parameters

  • path (pathlib.Path) – Path where tensors will be saved.

  • mkdir (bool, optional) – If True, create directory if doesn’t exists. Default: False

  • *args – Varargs passed to pathlib.Path’s mkdir method if mkdir argument set to True.

  • *kwargs – Kwarargs passed to pathlib.Path’s mkdir method if mkdir argument set to True.

class torchfunc.hooks.recorders.ForwardOutput(condition: Callable = None, reduction: Callable = None)[source]

Record output values after forward of specified layer(s).

You can record only some of the data based on external conditions if condition callable is specified.

Data can be cumulated together via reduction parameter, which is advised from the memory perspective.

Parameters

  • condition (Callable, optional) – No argument callable. If True returned, record data. Can be used to save data based on external environment (e.g. dataset’s label). If None, will record every data point. Default: None

  • reduction (Callable, optional) – Operation to use on incoming data. Should take two arguments, and return one. Acts similarly to reduction argument of Python’s functools.reduce. If None, data will be added to list, which may be very memory intensive. Default: None

data

Keeps data passing through subrecorders, optionally reduced by reduction. Each item represents data for specified subrecorder.

Type

List

subrecorders

List containing registered subrecorders.

Type

List[Hooks]

handles

Handles for registered subrecorders, each corresponds to specific subrecorder. Can be used to unregister certain subrecorders (though discouraged, please use remove method).

Type

List[torch.utils.subrecorders.RemovableHandle]

apply(function: Callable)

Apply function to data contained in each subrecorder.

Data will be modified an saved inside data of each subrecorder. This function may make recorder unusable, it’s up to user to ensure correct functioning after this functionality was used.

Parameters

function (Callable) – Single argument (torch.Tensor data from subrecorder) callable returning anything.

apply_sample(function: Callable) → None

Apply function to data contained in each subrecorder.

Works like apply, except Callable is passed number of samples passed through subrecorder as second argument

Parameters

function (Callable) – Two argument (torch.Tensor data from subrecorder and number of samples which passed through it) Callable returning anything.

children(network, types: Tuple[Any] = None, indices: List[int] = None)

Register subrecorders using types and/or indices via children method.

This function will use children method of torch.nn.Module to iterate over available submodules. If you wish to iterate recursively, use modules.

Important:

If types and indices are left with their default values, all modules will have subrecorders registered.

Parameters

  • module (torch.nn.Module) – Module (usually neural network) for which inputs will be collected.

  • types (Tuple[typing.Any], optional) – Module types for which data will be recorded. E.g. (torch.nn.Conv2d, torch.nn.Linear) will register subrecorders on every module being instance of either Conv2d or Linear. Default: None

  • indices (Iterable[int], optional) – Indices of modules whose inputs will be registered. Default: None

Returns

Return type

self

iter_samples()

Iterate over count of samples for each subrecorder.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

How many samples passed through this subrecorder.

Return type

int

modules(module: torch.nn.modules.module.Module, types: Tuple[Any] = None, indices: List[int] = None)

Register subrecorders using types and/or indices via modules method.

This function will use modules method of torch.nn.Module to iterate over available submodules. If you wish to iterate non-recursively, use children.

Important:

If types and indices are left with their default values, all modules will have subrecorders registered.

Parameters

  • module (torch.nn.Module) – Module (usually neural network) for which inputs will be collected.

  • types (Tuple[typing.Any], optional) – Module types for which data will be recorded. E.g. (torch.nn.Conv2d, torch.nn.Linear) will register subrecorders on every module being instance of either Conv2d or Linear. Default: None

  • indices (Iterable[int], optional) – Indices of modules whose inputs will be registered. Default: None

Returns

Return type

self

remove(index)

Remove subrecorder specified by index.

Subrecorder will not record data passing through it and will be removed from subrecorders attribute.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

Data contained in subrecorder.

Return type

torch.Tensor

samples(index) → int

Count of samples passed through subrecorder under index.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

How many samples passed through specified subrecorder.

Return type

int

save(path: pathlib.Path, mkdir: bool = False, *args, **kwargs)

Save data tensors within specified path.

Each data tensor will be indexed by integer [0, N), where indices represent consecutive subrecorders.

Parameters

  • path (pathlib.Path) – Path where tensors will be saved.

  • mkdir (bool, optional) – If True, create directory if doesn’t exists. Default: False

  • *args – Varargs passed to pathlib.Path’s mkdir method if mkdir argument set to True.

  • *kwargs – Kwarargs passed to pathlib.Path’s mkdir method if mkdir argument set to True.

class torchfunc.hooks.recorders.ForwardPre(condition: Callable = None, reduction: Callable = None)[source]

Record input values before forward of specified layer(s).

You can record only some of the data based on external conditions if condition callable is specified.

Data can be cumulated together via reduction parameter, which is advised from the memory perspective.

Parameters

  • condition (Callable, optional) – No argument callable. If True returned, record data. Can be used to save data based on external environment (e.g. dataset’s label). If None, will record every data point. Default: None

  • reduction (Callable, optional) – Operation to use on incoming data. Should take two arguments, and return one. Acts similarly to reduction argument of Python’s functools.reduce. If None, data will be added to list, which may be very memory intensive. Default: None

data

Keeps data passing through subrecorders, optionally reduced by reduction. Each item represents data for specified subrecorder.

Type

List

subrecorders

List containing registered subrecorders.

Type

List[Hooks]

handles

Handles for registered subrecorders, each corresponds to specific subrecorder. Can be used to unregister certain subrecorders (though discouraged, please use remove method).

Type

List[torch.utils.subrecorders.RemovableHandle]

apply(function: Callable)

Apply function to data contained in each subrecorder.

Data will be modified an saved inside data of each subrecorder. This function may make recorder unusable, it’s up to user to ensure correct functioning after this functionality was used.

Parameters

function (Callable) – Single argument (torch.Tensor data from subrecorder) callable returning anything.

apply_sample(function: Callable) → None

Apply function to data contained in each subrecorder.

Works like apply, except Callable is passed number of samples passed through subrecorder as second argument

Parameters

function (Callable) – Two argument (torch.Tensor data from subrecorder and number of samples which passed through it) Callable returning anything.

children(network, types: Tuple[Any] = None, indices: List[int] = None)

Register subrecorders using types and/or indices via children method.

This function will use children method of torch.nn.Module to iterate over available submodules. If you wish to iterate recursively, use modules.

Important:

If types and indices are left with their default values, all modules will have subrecorders registered.

Parameters

  • module (torch.nn.Module) – Module (usually neural network) for which inputs will be collected.

  • types (Tuple[typing.Any], optional) – Module types for which data will be recorded. E.g. (torch.nn.Conv2d, torch.nn.Linear) will register subrecorders on every module being instance of either Conv2d or Linear. Default: None

  • indices (Iterable[int], optional) – Indices of modules whose inputs will be registered. Default: None

Returns

Return type

self

iter_samples()

Iterate over count of samples for each subrecorder.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

How many samples passed through this subrecorder.

Return type

int

modules(module: torch.nn.modules.module.Module, types: Tuple[Any] = None, indices: List[int] = None)

Register subrecorders using types and/or indices via modules method.

This function will use modules method of torch.nn.Module to iterate over available submodules. If you wish to iterate non-recursively, use children.

Important:

If types and indices are left with their default values, all modules will have subrecorders registered.

Parameters

  • module (torch.nn.Module) – Module (usually neural network) for which inputs will be collected.

  • types (Tuple[typing.Any], optional) – Module types for which data will be recorded. E.g. (torch.nn.Conv2d, torch.nn.Linear) will register subrecorders on every module being instance of either Conv2d or Linear. Default: None

  • indices (Iterable[int], optional) – Indices of modules whose inputs will be registered. Default: None

Returns

Return type

self

remove(index)

Remove subrecorder specified by index.

Subrecorder will not record data passing through it and will be removed from subrecorders attribute.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

Data contained in subrecorder.

Return type

torch.Tensor

samples(index) → int

Count of samples passed through subrecorder under index.

Parameters

index (int) – Index of subrecorder (usually layer)

Returns

How many samples passed through specified subrecorder.

Return type

int

save(path: pathlib.Path, mkdir: bool = False, *args, **kwargs)

Save data tensors within specified path.

Each data tensor will be indexed by integer [0, N), where indices represent consecutive subrecorders.

Parameters

  • path (pathlib.Path) – Path where tensors will be saved.

  • mkdir (bool, optional) – If True, create directory if doesn’t exists. Default: False

  • *args – Varargs passed to pathlib.Path’s mkdir method if mkdir argument set to True.

  • *kwargs – Kwarargs passed to pathlib.Path’s mkdir method if mkdir argument set to True.