Table of Contents
Shortcuts

torchlayers.convolution module

class torchlayers.convolution.ChannelShuffle(groups: int)[source]

Shuffle output channels.

When using group convolution knowledge transfer between next layers is reduced (as the same input channels are convolved with the same output channels).

This layer reshuffles output channels via simple reshape in order to mix the representation from separate groups and improve knowledge transfer.

Originally proposed by Xiangyu Zhang et. al in: ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

Example:

import torchlayers as tl

model = tl.Sequential(
    tl.Conv(64),
    tl.Swish(),
    tl.Conv(128, groups=16),
    tl.ChannelShuffle(groups=16),
    tl.Conv(256),
    tl.GlobalMaxPool(),
    tl.Linear(10),
)
Parameters

groups (int) – Number of groups used in the previous convolutional layer.

forward(inputs)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class torchlayers.convolution.ChannelSplit(p: float, dim: int = 1)[source]

Convenience layer splitting tensor using p.

Returns two outputs, splitted accordingly to parameters.

Example:

import torchlayers as tl


class Net(tl.Module):
    def __init__(self):
        super().__init__()
        self.layer = tl.Conv(256, groups=16)
        self.splitter = tl.ChannelSplit(0.5)

    def forward(x):
        outputs = self.layer(x)
        half, rest = self.splitter(outputs)
        return half # for some reason
Parameters

  • p (float) – Percentage of channels to go into first group

  • dim (int, optional) – Dimension along which input will be splitted. Default: 1 (channel dimension)

forward(inputs)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class torchlayers.convolution.Conv(in_channels: int, out_channels: int, kernel_size=3, stride=1, padding='same', dilation=1, groups: int = 1, bias: bool = True, padding_mode: str = 'zeros')[source]

Standard convolution layer.

Based on input shape it either creates 1D, 2D or 3D convolution for inputs of shape 3D, 4D, 5D respectively (including batch as first dimension).

Additional same padding mode was added and set as default. This mode preserves all dimensions excepts channels.

kernel_size got a default value of 3.

Otherwise acts exactly like PyTorch’s Convolution, see documentation.

Example:

import torchlayers as tl


class Classifier(tl.Module):
    def __init__(self, out_shape):
        super().__init__()
        self.conv1 = tl.Conv(64, 6)
        self.conv2 = tl.Conv(128)
        self.conv3 = tl.Conv(256)
        self.pooling = tl.GlobalMaxPool()
        self.dense = tl.Linear(out_shape)

    def forward(self, x):
        x = torch.relu(self.conv1(x))
        x = torch.relu(self.conv2(x))
        x = torch.relu(self.conv3(x))
        return self.dense(self.pooling(x))

Note

IMPORTANT: same currently works only for odd values of kernel_size, dilation and stride. If any of those is even you should explicitly pad your input asymmetrically with torch.functional.pad or a-like.

Parameters

  • in_channels (int) – Number of channels in the input image

  • out_channels (int) – Number of channels produced by the convolution

  • kernel_size (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Size of the convolving kernel. User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: 3

  • stride (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Stride of the convolution. User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: 3

  • padding (Union[str, int, Tuple[int, int], Tuple[int, int, int]], optional) – Padding added to both sides of the input. String “same” can be used with odd kernel_size, stride and dilation User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: same

  • dilation (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Spacing between kernel elements. String “same” can be used with odd kernel_size, stride and dilation User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: 1

  • groups (int, optional) – Number of blocked connections from input channels to output channels. Default: 1

  • bias (bool, optional) – If True, adds a learnable bias to the output. Default: True

  • padding_mode (string, optional) – Accepted values zeros and circular Default: zeros

class torchlayers.convolution.ConvTranspose(in_channels: int, out_channels: int, kernel_size=3, stride=1, padding='same', output_padding=0, dilation=1, groups: int = 1, bias: bool = True, padding_mode='zeros')[source]

Standard transposed convolution layer.

Based on input shape it either creates 1D, 2D or 3D convolution (for inputs of shape 3D, 4D, 5D including batch as first dimension).

Otherwise acts exactly like PyTorch’s Convolution, see documentation.

Default argument for kernel_size was added equal to 3.

Note

IMPORTANT: same currently works only for odd values of kernel_size, dilation and stride. If any of those is even you should explicitly pad your input asymmetrically with torch.functional.pad or a-like.

Parameters

  • in_channels (int) – Number of channels in the input image

  • out_channels (int) – Number of channels produced by the convolution

  • kernel_size (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Size of the convolving kernel. User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: 3

  • stride (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Stride of the convolution. User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: 3

  • padding (Union[str, int, Tuple[int, int], Tuple[int, int, int]], optional) – Padding added to both sides of the input. String “same” can be used with odd kernel_size, stride and dilation User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: same

  • output_padding (int or tuple, optional) – Additional size added to one side of the output shape. Default: 0

  • dilation (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Spacing between kernel elements. String “same” can be used with odd kernel_size, stride and dilation User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: 1

  • groups (int, optional) – Number of blocked connections from input channels to output channels. Default: 1

  • bias (bool, optional) – If True, adds a learnable bias to the output. Default: True dilation (int or tuple, optional): Spacing between kernel elements. Default: 1

  • padding_mode (string, optional) – Accepted values zeros and circular Default: zeros

class torchlayers.convolution.Dense(module: torch.nn.modules.module.Module, dim: int = 1)[source]

Dense residual connection concatenating input channels and output channels of provided module.

Originally proposed by Gao Huang et. al in Densely Connected Convolutional Networks

Can be used just like torchlayers.convolution.Residual but concatenates channels (dimension can be specified) instead of adding.

Parameters

  • module (torch.nn.Module) – Convolutional PyTorch module (or other compatible module). Shape of module’s inputs has to be equal to it’s outputs, both should be addable torch.Tensor instances.

  • dim (int, optional) – Dimension along which input and module’s output will be concatenated. Default: 1 (channel-wise)

forward(inputs)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class torchlayers.convolution.DepthwiseConv(in_channels: int, out_channels: int, kernel_size=3, stride=1, padding='same', dilation=1, bias: bool = True, padding_mode: str = 'zeros')[source]

Depthwise convolution layer.

Based on input shape it either creates 1D, 2D or 3D depthwise convolution for inputs of shape 3D, 4D, 5D respectively (including batch as first dimension).

Additional same padding mode was added and set as default. This mode preserves all dimensions excepts channels.

kernel_size got a default value of 3.

Note

IMPORTANT: same currently works only for odd values of kernel_size, dilation and stride. If any of those is even you should explicitly pad your input asymmetrically with torch.functional.pad or a-like.

Note

IMPORTANT: out_channels has to be divisible by in_channels without remainder (e.g. out_channels=64 and in_channels=32), otherwise error is thrown.

Parameters

  • in_channels (int) – Number of channels in the input image

  • out_channels (int) – Number of channels produced by the convolution

  • kernel_size (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Size of the convolving kernel. User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: 3

  • stride (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Stride of the convolution. User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: 3

  • padding (Union[str, int, Tuple[int, int], Tuple[int, int, int]], optional) – Padding added to both sides of the input. String “same” can be used with odd kernel_size, stride and dilation User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: same

  • dilation (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Spacing between kernel elements. String “same” can be used with odd kernel_size, stride and dilation User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: 1

  • bias (bool, optional) – If True, adds a learnable bias to the output. Default: True

  • padding_mode (string, optional) – Accepted values zeros and circular Default: zeros

class torchlayers.convolution.Fire(in_channels: int, out_channels: int, hidden_channels=None, p: float = 0.5)[source]

Squeeze and Expand number of channels efficiently operation-wise.

First input channels will be squeezed to hidden channels and 1x11 x 1 convolution. After that those will be expanded to out_channels partially done by 3x33 x 3 convolution and partially by 1x11 x 1 convolution (as specified by p parameter).

Originally proposed by Forrest N. Iandola et. al in SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size

Parameters

  • in_channels (int) – Number of channels in the input

  • out_channels (int) – Number of channels produced by Fire module

  • hidden_channels (int, optional) – Number of hidden channels (squeeze convolution layer). Default: None (half of in_channels)

  • p (float, optional) – Ratio of 1x11 x 1 convolution taken from total out_channels. The more, the more 1x11 x 1 convolution will be used during expanding. Default: 0.5 (half of out_channels)

forward(inputs)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class torchlayers.convolution.InvertedResidualBottleneck(in_channels: int, hidden_channels: int = None, activation=None, batchnorm: bool = True, squeeze_excitation: bool = True, squeeze_excitation_hidden: int = None, squeeze_excitation_activation=None, squeeze_excitation_sigmoid=None)[source]

Inverted residual block used in MobileNetV2, MNasNet, Efficient Net and other architectures.

Originally proposed by Mark Sandler et. al in MobileNetV2: Inverted Residuals and Linear Bottlenecks <0.5MB model size

Expanded with SqueezeExcitation after depthwise convolution by Mingxing Tan et. al in MnasNet: Platform-Aware Neural Architecture Search for Mobile

Due to it’s customizable nature blocks from other research papers could be easily produced, e.g. Searching for MobileNetV3 by providing torchlayers.HardSwish() as activation, torchlayers.HardSigmoid() as squeeze_excitation_activation and squeeze_excitation_hidden equal to hidden_channels // 4.

Parameters

  • in_channels (int) – Number of channels in the input

  • hidden_channels (int, optional) – Number of hidden channels (expanded). Should be greater than in_channels, usually by factor of 4. Default: in_channels * 4

  • activation (typing.Callable, optional) – One argument callable performing activation after hidden layer. Default: torch.nn.ReLU6()

  • batchnorm (bool, optional) – Whether to apply Batch Normalization layer after initial convolution, depthwise expanding part (before squeeze excitation) and final squeeze. Default: True

  • squeeze_excitation (bool, optional) – Whether to use standard SqueezeExcitation (see SqueezeExcitation module) after depthwise convolution. Default: True

  • squeeze_excitation_hidden (int, optional) – Size of the hidden torch.nn.Linear layer. Usually smaller than in_channels (at least in original research paper). Default: 1/16 of in_channels as suggested by original paper.

  • squeeze_excitation_activation (typing.Callable, optional) – One argument callable performing activation after hidden layer. Default: torch.nn.ReLU()

  • squeeze_excitation_sigmoid (typing.Callable, optional) – One argument callable squashing values after excitation. Default: torch.nn.Sigmoid

forward(inputs)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class torchlayers.convolution.MPoly(*poly_modules: torch.nn.modules.module.Module)[source]

Apply multiple modules to input multiple times and sum.

It’s equation for poly_modules length equal to NN could be expressed by

I+F1+F1(F0)+...+FN(FN−1...F0)I + F_1 + F_1(F_0) + ... + F_N(F_{N-1}...F_0)

where II is identity and consecutive FNF_N are consecutive modules applied to output of previous ones.

Originally proposed by Xingcheng Zhang et. al in PolyNet: A Pursuit of Structural Diversity in Very Deep Networks

Parameters

*poly_modules (torch.nn.Module) – Variable arg of modules to use. If empty, acts as an identity. For single module acts like ResNet. 2 was used in original paper. All modules need inputs and outputs of equal shape.

forward(inputs)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class torchlayers.convolution.Poly(module: torch.nn.modules.module.Module, order: int = 2)[source]

Apply one module to input multiple times and sum.

It’s equation for order equal to NN could be expressed as

I+F+F2+...+FNI + F + F^2 + ... + F^N

where II is identity mapping and FF is output of module applied N^N times.

Originally proposed by Xingcheng Zhang et. al in PolyNet: A Pursuit of Structural Diversity in Very Deep Networks

Example:

import torchlayers as tl

# Any input will be passed 3 times
# Through the same convolutional layer (weights and biases shared)
layer = tl.Sequential(tl.Conv(64), tl.Poly(tl.Conv(64), order=3))
layer(torch.randn(1, 3, 32, 32))

Above can be rewritten by the following:

x = torch.randn(1, 3, 32, 32)

first_convolution = tl.Conv(64)
output = first_convolution(x)

shared_convolution = tl.Conv(64)
first_level = shared_convolution(output)
second_level = shared_convolution(first_level)
third_level = shared_convolution(second_level)

# That's what tl.Poly would return
final = output + first_level + second_level + third_level
Parameters

  • module (torch.nn.Module) – Convolutional PyTorch module (or other compatible module). inputs shape has to be equal to it’s output shape (for 2D convolution it would be (C,H,W)(C, H, W) )

  • order (int, optional) – Order of PolyInception module. For order equal to 1 acts just like ResNet, order of 2 was used in original paper. Default: 2

extra_repr()[source]

Set the extra representation of the module

To print customized extra information, you should reimplement this method in your own modules. Both single-line and multi-line strings are acceptable.

forward(inputs)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class torchlayers.convolution.Residual(module: torch.nn.modules.module.Module, projection: torch.nn.modules.module.Module = None)[source]

Residual connection adding input to output of provided module.

Originally proposed by He et. al in ResNet

For correct usage it is advised to keep input line (skip connection) without any layer or activation and implement transformations only in module arguments (as per Identity Mappings in Deep Residual Networks).

Example:

import torch
import torchlayers as tl

# ResNet-like block
class _BlockImpl(tl.Module):
    def __init__(self, in_channels: int):
        self.block = tl.Residual(
            tl.Sequential(
                tl.Conv(in_channels),
                tl.ReLU(),
                tl.Conv(4 * in_channels),
                tl.ReLU(),
                tl.Conv(in_channels),
            )
        )

    def forward(self, x):
        return self.block(x)


Block = tl.infer(_BlockImpl)
Parameters

  • module (torch.nn.Module) – Convolutional PyTorch module (or other compatible module). Shape of module’s inputs has to be equal to it’s outputs, both should be addable torch.Tensor instances.

  • projection (torch.nn.Module, optional) – If shapes of inputs and module results are different, it’s user responsibility to add custom projection module (usually 1x1 convolution). Default: None

forward(inputs)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class torchlayers.convolution.SeparableConv(in_channels: int, out_channels: int, kernel_size=3, stride=1, padding='same', dilation=1, bias: bool = True, padding_mode: str = 'zeros')[source]

Separable convolution layer (a.k.a. depthwise separable convolution).

Based on input shape it either creates 1D, 2D or 3D separable convolution for inputs of shape 3D, 4D, 5D respectively (including batch as first dimension).

Additional same padding mode was added and set as default. This mode preserves all dimensions excepts channels.

kernel_size got a default value of 3.

Note

IMPORTANT: same currently works only for odd values of kernel_size, dilation and stride. If any of those is even you should explicitly pad your input asymmetrically with torch.functional.pad or a-like.

Parameters

  • in_channels (int) – Number of channels in the input image

  • out_channels (int) – Number of channels produced by the convolution

  • kernel_size (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Size of the convolving kernel. User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: 3

  • stride (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Stride of the convolution. User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: 3

  • padding (Union[str, int, Tuple[int, int], Tuple[int, int, int]], optional) – Padding added to both sides of the input. String “same” can be used with odd kernel_size, stride and dilation User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: same

  • dilation (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Spacing between kernel elements. String “same” can be used with odd kernel_size, stride and dilation User can specify int or 2-tuple (for Conv2d) or 3-tuple (for Conv3d). Default: 1

  • bias (bool, optional) – If True, adds a learnable bias to the output. Default: True

  • padding_mode (string, optional) – Accepted values zeros and circular Default: zeros

forward(inputs)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class torchlayers.convolution.SqueezeExcitation(in_channels: int, hidden: int = None, activation=None, sigmoid=None)[source]

Learn channel-wise excitation maps for inputs.

Provided inputs will be squeezed into in_channels via average pooling, passed through two non-linear layers, rescaled to [0,1][0, 1] via sigmoid-like function and multiplied with original input channel-wise.

Originally proposed by Xingcheng Zhang et. al in Squeeze-and-Excitation Networks

Example:

import torchlayers as tl

# Assume only 128 channels can be an input in this case
block = tl.Residual(tl.Conv(128), tl.SqueezeExcitation(), tl.Conv(128))
Parameters

  • in_channels (int) – Number of channels in the input

  • hidden (int, optional) – Size of the hidden torch.nn.Linear layer. Usually smaller than in_channels (at least in original research paper). Default: 1/16 of in_channels as suggested by original paper.

  • activation (Callable[[Tensor], Tensor], optional) – One argument callable performing activation after hidden layer. Default: torch.nn.ReLU()

  • sigmoid (Callable[[Tensor], Tensor], optional) – One argument callable squashing values after excitation. Default: torch.nn.Sigmoid

forward(inputs)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class torchlayers.convolution.WayPoly(*poly_modules: torch.nn.modules.module.Module)[source]

Apply multiple modules to input and sum.

It’s equation for poly_modules length equal to NN could be expressed by

I+F1(I)+F2(I)+...+FNI + F_1(I) + F_2(I) + ... + F_N

where II is identity and consecutive FNF_N are consecutive poly_modules applied to input.

Could be considered as an extension of standard ResNet to many parallel modules.

Originally proposed by Xingcheng Zhang et. al in PolyNet: A Pursuit of Structural Diversity in Very Deep Networks

Parameters

*poly_modules (torch.nn.Module) – Variable arg of modules to use. If empty, acts as an identity. For single module acts like ResNet. 2 was used in original paper. All modules need inputs and outputs of equal shape.

forward(inputs)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.