torchlayers.convolution module¶
-
class
torchlayers.convolution.
ChannelShuffle
(groups: int)[source]¶ Shuffle output channels.
When using group convolution knowledge transfer between next layers is reduced (as the same input channels are convolved with the same output channels).
This layer reshuffles output channels via simple
reshape
in order to mix the representation from separate groups and improve knowledge transfer.Originally proposed by Xiangyu Zhang et. al in: ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Example:
import torchlayers as tl model = tl.Sequential( tl.Conv(64), tl.Swish(), tl.Conv(128, groups=16), tl.ChannelShuffle(groups=16), tl.Conv(256), tl.GlobalMaxPool(), tl.Linear(10), )
- Parameters
groups (int) – Number of groups used in the previous convolutional layer.
-
forward
(inputs)[source]¶ Defines the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
-
class
torchlayers.convolution.
ChannelSplit
(p: float, dim: int = 1)[source]¶ Convenience layer splitting tensor using
p
.Returns two outputs, splitted accordingly to parameters.
Example:
import torchlayers as tl class Net(tl.Module): def __init__(self): super().__init__() self.layer = tl.Conv(256, groups=16) self.splitter = tl.ChannelSplit(0.5) def forward(x): outputs = self.layer(x) half, rest = self.splitter(outputs) return half # for some reason
- Parameters
-
forward
(inputs)[source]¶ Defines the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
-
class
torchlayers.convolution.
Conv
(in_channels: int, out_channels: int, kernel_size=3, stride=1, padding='same', dilation=1, groups: int = 1, bias: bool = True, padding_mode: str = 'zeros')[source]¶ Standard convolution layer.
Based on input shape it either creates 1D, 2D or 3D convolution for inputs of shape 3D, 4D, 5D respectively (including batch as first dimension).
Additional
same
padding
mode was added and set as default. This mode preserves all dimensions excepts channels.kernel_size
got a default value of3
.Otherwise acts exactly like PyTorch’s Convolution, see documentation.
Example:
import torchlayers as tl class Classifier(tl.Module): def __init__(self, out_shape): super().__init__() self.conv1 = tl.Conv(64, 6) self.conv2 = tl.Conv(128) self.conv3 = tl.Conv(256) self.pooling = tl.GlobalMaxPool() self.dense = tl.Linear(out_shape) def forward(self, x): x = torch.relu(self.conv1(x)) x = torch.relu(self.conv2(x)) x = torch.relu(self.conv3(x)) return self.dense(self.pooling(x))
Note
IMPORTANT:
same
currently works only for odd values ofkernel_size
,dilation
andstride
. If any of those is even you should explicitly pad your input asymmetrically withtorch.functional.pad
or a-like.- Parameters
in_channels (int) – Number of channels in the input image
out_channels (int) – Number of channels produced by the convolution
kernel_size (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Size of the convolving kernel. User can specify
int
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:3
stride (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Stride of the convolution. User can specify
int
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:3
padding (Union[str, int, Tuple[int, int], Tuple[int, int, int]], optional) – Padding added to both sides of the input. String “same” can be used with odd
kernel_size
,stride
anddilation
User can specifyint
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:same
dilation (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Spacing between kernel elements. String “same” can be used with odd
kernel_size
,stride
anddilation
User can specifyint
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:1
groups (int, optional) – Number of blocked connections from input channels to output channels. Default: 1
bias (bool, optional) – If
True
, adds a learnable bias to the output. Default:True
padding_mode (string, optional) – Accepted values
zeros
andcircular
Default:zeros
-
class
torchlayers.convolution.
ConvTranspose
(in_channels: int, out_channels: int, kernel_size=3, stride=1, padding='same', output_padding=0, dilation=1, groups: int = 1, bias: bool = True, padding_mode='zeros')[source]¶ Standard transposed convolution layer.
Based on input shape it either creates 1D, 2D or 3D convolution (for inputs of shape 3D, 4D, 5D including batch as first dimension).
Otherwise acts exactly like PyTorch’s Convolution, see documentation.
Default argument for
kernel_size
was added equal to3
.Note
IMPORTANT:
same
currently works only for odd values ofkernel_size
,dilation
andstride
. If any of those is even you should explicitly pad your input asymmetrically withtorch.functional.pad
or a-like.- Parameters
in_channels (int) – Number of channels in the input image
out_channels (int) – Number of channels produced by the convolution
kernel_size (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Size of the convolving kernel. User can specify
int
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:3
stride (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Stride of the convolution. User can specify
int
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:3
padding (Union[str, int, Tuple[int, int], Tuple[int, int, int]], optional) – Padding added to both sides of the input. String “same” can be used with odd
kernel_size
,stride
anddilation
User can specifyint
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:same
output_padding (int or tuple, optional) – Additional size added to one side of the output shape. Default: 0
dilation (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Spacing between kernel elements. String “same” can be used with odd
kernel_size
,stride
anddilation
User can specifyint
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:1
groups (int, optional) – Number of blocked connections from input channels to output channels. Default: 1
bias (bool, optional) – If
True
, adds a learnable bias to the output. Default:True
dilation (int or tuple, optional): Spacing between kernel elements. Default: 1padding_mode (string, optional) – Accepted values
zeros
andcircular
Default:zeros
-
class
torchlayers.convolution.
Dense
(module: torch.nn.modules.module.Module, dim: int = 1)[source]¶ Dense residual connection concatenating input channels and output channels of provided module.
Originally proposed by Gao Huang et. al in Densely Connected Convolutional Networks
Can be used just like
torchlayers.convolution.Residual
but concatenates channels (dimension can be specified) instead of adding.- Parameters
module (torch.nn.Module) – Convolutional PyTorch module (or other compatible module). Shape of module’s
inputs
has to be equal to it’soutputs
, both should be addabletorch.Tensor
instances.dim (int, optional) – Dimension along which
input
and module’soutput
will be concatenated. Default:1
(channel-wise)
-
forward
(inputs)[source]¶ Defines the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
-
class
torchlayers.convolution.
DepthwiseConv
(in_channels: int, out_channels: int, kernel_size=3, stride=1, padding='same', dilation=1, bias: bool = True, padding_mode: str = 'zeros')[source]¶ Depthwise convolution layer.
Based on input shape it either creates 1D, 2D or 3D depthwise convolution for inputs of shape 3D, 4D, 5D respectively (including batch as first dimension).
Additional
same
padding
mode was added and set as default. This mode preserves all dimensions excepts channels.kernel_size
got a default value of3
.Note
IMPORTANT:
same
currently works only for odd values ofkernel_size
,dilation
andstride
. If any of those is even you should explicitly pad your input asymmetrically withtorch.functional.pad
or a-like.Note
IMPORTANT:
out_channels
has to be divisible byin_channels
without remainder (e.g.out_channels=64
andin_channels=32
), otherwise error is thrown.- Parameters
in_channels (int) – Number of channels in the input image
out_channels (int) – Number of channels produced by the convolution
kernel_size (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Size of the convolving kernel. User can specify
int
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:3
stride (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Stride of the convolution. User can specify
int
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:3
padding (Union[str, int, Tuple[int, int], Tuple[int, int, int]], optional) – Padding added to both sides of the input. String “same” can be used with odd
kernel_size
,stride
anddilation
User can specifyint
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:same
dilation (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Spacing between kernel elements. String “same” can be used with odd
kernel_size
,stride
anddilation
User can specifyint
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:1
bias (bool, optional) – If
True
, adds a learnable bias to the output. Default:True
padding_mode (string, optional) – Accepted values
zeros
andcircular
Default:zeros
-
class
torchlayers.convolution.
Fire
(in_channels: int, out_channels: int, hidden_channels=None, p: float = 0.5)[source]¶ Squeeze and Expand number of channels efficiently operation-wise.
First input channels will be squeezed to
hidden
channels and convolution. After that those will be expanded toout_channels
partially done by convolution and partially by convolution (as specified byp
parameter).Originally proposed by Forrest N. Iandola et. al in SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Parameters
in_channels (int) – Number of channels in the input
out_channels (int) – Number of channels produced by Fire module
hidden_channels (int, optional) – Number of hidden channels (squeeze convolution layer). Default:
None
(half ofin_channels
)p (float, optional) – Ratio of convolution taken from total
out_channels
. The more, the more convolution will be used during expanding. Default:0.5
(half ofout_channels
)
-
forward
(inputs)[source]¶ Defines the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
-
class
torchlayers.convolution.
InvertedResidualBottleneck
(in_channels: int, hidden_channels: int = None, activation=None, batchnorm: bool = True, squeeze_excitation: bool = True, squeeze_excitation_hidden: int = None, squeeze_excitation_activation=None, squeeze_excitation_sigmoid=None)[source]¶ Inverted residual block used in MobileNetV2, MNasNet, Efficient Net and other architectures.
Originally proposed by Mark Sandler et. al in MobileNetV2: Inverted Residuals and Linear Bottlenecks <0.5MB model size
Expanded with
SqueezeExcitation
after depthwise convolution by Mingxing Tan et. al in MnasNet: Platform-Aware Neural Architecture Search for MobileDue to it’s customizable nature blocks from other research papers could be easily produced, e.g. Searching for MobileNetV3 by providing
torchlayers.HardSwish()
asactivation
,torchlayers.HardSigmoid()
assqueeze_excitation_activation
andsqueeze_excitation_hidden
equal tohidden_channels // 4
.- Parameters
in_channels (int) – Number of channels in the input
hidden_channels (int, optional) – Number of hidden channels (expanded). Should be greater than
in_channels
, usually by factor of4
. Default:in_channels * 4
activation (typing.Callable, optional) – One argument callable performing activation after hidden layer. Default:
torch.nn.ReLU6()
batchnorm (bool, optional) – Whether to apply Batch Normalization layer after initial convolution, depthwise expanding part (before squeeze excitation) and final squeeze. Default:
True
squeeze_excitation (bool, optional) – Whether to use standard
SqueezeExcitation
(seeSqueezeExcitation
module) after depthwise convolution. Default:True
squeeze_excitation_hidden (int, optional) – Size of the hidden
torch.nn.Linear
layer. Usually smaller thanin_channels
(at least in original research paper). Default:1/16
ofin_channels
as suggested by original paper.squeeze_excitation_activation (typing.Callable, optional) – One argument callable performing activation after hidden layer. Default:
torch.nn.ReLU()
squeeze_excitation_sigmoid (typing.Callable, optional) – One argument callable squashing values after excitation. Default:
torch.nn.Sigmoid
-
forward
(inputs)[source]¶ Defines the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
-
class
torchlayers.convolution.
MPoly
(*poly_modules: torch.nn.modules.module.Module)[source]¶ Apply multiple modules to input multiple times and sum.
It’s equation for
poly_modules
length equal to could be expressed bywhere is identity and consecutive are consecutive modules applied to output of previous ones.
Originally proposed by Xingcheng Zhang et. al in PolyNet: A Pursuit of Structural Diversity in Very Deep Networks
- Parameters
*poly_modules (torch.nn.Module) – Variable arg of modules to use. If empty, acts as an identity. For single module acts like
ResNet
.2
was used in original paper. All modules needinputs
andoutputs
of equalshape
.
-
forward
(inputs)[source]¶ Defines the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
-
class
torchlayers.convolution.
Poly
(module: torch.nn.modules.module.Module, order: int = 2)[source]¶ Apply one module to input multiple times and sum.
It’s equation for
order
equal to could be expressed aswhere is identity mapping and is output of
module
applied times.Originally proposed by Xingcheng Zhang et. al in PolyNet: A Pursuit of Structural Diversity in Very Deep Networks
Example:
import torchlayers as tl # Any input will be passed 3 times # Through the same convolutional layer (weights and biases shared) layer = tl.Sequential(tl.Conv(64), tl.Poly(tl.Conv(64), order=3)) layer(torch.randn(1, 3, 32, 32))
Above can be rewritten by the following:
x = torch.randn(1, 3, 32, 32) first_convolution = tl.Conv(64) output = first_convolution(x) shared_convolution = tl.Conv(64) first_level = shared_convolution(output) second_level = shared_convolution(first_level) third_level = shared_convolution(second_level) # That's what tl.Poly would return final = output + first_level + second_level + third_level
- Parameters
module (torch.nn.Module) – Convolutional PyTorch module (or other compatible module).
inputs
shape has to be equal to it’soutput
shape (for 2D convolution it would be )order (int, optional) – Order of PolyInception module. For order equal to
1
acts just like ResNet, order of2
was used in original paper. Default:2
-
extra_repr
()[source]¶ Set the extra representation of the module
To print customized extra information, you should reimplement this method in your own modules. Both single-line and multi-line strings are acceptable.
-
forward
(inputs)[source]¶ Defines the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
-
class
torchlayers.convolution.
Residual
(module: torch.nn.modules.module.Module, projection: torch.nn.modules.module.Module = None)[source]¶ Residual connection adding input to output of provided module.
Originally proposed by He et. al in ResNet
For correct usage it is advised to keep input line (skip connection) without any layer or activation and implement transformations only in module arguments (as per Identity Mappings in Deep Residual Networks).
Example:
import torch import torchlayers as tl # ResNet-like block class _BlockImpl(tl.Module): def __init__(self, in_channels: int): self.block = tl.Residual( tl.Sequential( tl.Conv(in_channels), tl.ReLU(), tl.Conv(4 * in_channels), tl.ReLU(), tl.Conv(in_channels), ) ) def forward(self, x): return self.block(x) Block = tl.infer(_BlockImpl)
- Parameters
module (torch.nn.Module) – Convolutional PyTorch module (or other compatible module). Shape of module’s
inputs
has to be equal to it’soutputs
, both should be addabletorch.Tensor
instances.projection (torch.nn.Module, optional) – If shapes of
inputs
andmodule
results are different, it’s user responsibility to add customprojection
module (usually1x1
convolution). Default:None
-
forward
(inputs)[source]¶ Defines the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
-
class
torchlayers.convolution.
SeparableConv
(in_channels: int, out_channels: int, kernel_size=3, stride=1, padding='same', dilation=1, bias: bool = True, padding_mode: str = 'zeros')[source]¶ Separable convolution layer (a.k.a. depthwise separable convolution).
Based on input shape it either creates 1D, 2D or 3D separable convolution for inputs of shape 3D, 4D, 5D respectively (including batch as first dimension).
Additional
same
padding
mode was added and set as default. This mode preserves all dimensions excepts channels.kernel_size
got a default value of3
.Note
IMPORTANT:
same
currently works only for odd values ofkernel_size
,dilation
andstride
. If any of those is even you should explicitly pad your input asymmetrically withtorch.functional.pad
or a-like.- Parameters
in_channels (int) – Number of channels in the input image
out_channels (int) – Number of channels produced by the convolution
kernel_size (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Size of the convolving kernel. User can specify
int
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:3
stride (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Stride of the convolution. User can specify
int
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:3
padding (Union[str, int, Tuple[int, int], Tuple[int, int, int]], optional) – Padding added to both sides of the input. String “same” can be used with odd
kernel_size
,stride
anddilation
User can specifyint
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:same
dilation (Union[int, Tuple[int, int], Tuple[int, int, int]], optional) – Spacing between kernel elements. String “same” can be used with odd
kernel_size
,stride
anddilation
User can specifyint
or 2-tuple (forConv2d
) or 3-tuple (forConv3d
). Default:1
bias (bool, optional) – If
True
, adds a learnable bias to the output. Default:True
padding_mode (string, optional) – Accepted values
zeros
andcircular
Default:zeros
-
forward
(inputs)[source]¶ Defines the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
-
class
torchlayers.convolution.
SqueezeExcitation
(in_channels: int, hidden: int = None, activation=None, sigmoid=None)[source]¶ Learn channel-wise excitation maps for
inputs
.Provided
inputs
will be squeezed intoin_channels
via average pooling, passed through two non-linear layers, rescaled to viasigmoid
-like function and multiplied with original input channel-wise.Originally proposed by Xingcheng Zhang et. al in Squeeze-and-Excitation Networks
Example:
import torchlayers as tl # Assume only 128 channels can be an input in this case block = tl.Residual(tl.Conv(128), tl.SqueezeExcitation(), tl.Conv(128))
- Parameters
in_channels (int) – Number of channels in the input
hidden (int, optional) – Size of the hidden
torch.nn.Linear
layer. Usually smaller thanin_channels
(at least in original research paper). Default:1/16
ofin_channels
as suggested by original paper.activation (Callable[[Tensor], Tensor], optional) – One argument callable performing activation after hidden layer. Default:
torch.nn.ReLU()
sigmoid (Callable[[Tensor], Tensor], optional) – One argument callable squashing values after excitation. Default:
torch.nn.Sigmoid
-
forward
(inputs)[source]¶ Defines the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
-
class
torchlayers.convolution.
WayPoly
(*poly_modules: torch.nn.modules.module.Module)[source]¶ Apply multiple modules to input and sum.
It’s equation for
poly_modules
length equal to could be expressed bywhere is identity and consecutive are consecutive
poly_modules
applied to input.Could be considered as an extension of standard
ResNet
to many parallel modules.Originally proposed by Xingcheng Zhang et. al in PolyNet: A Pursuit of Structural Diversity in Very Deep Networks
- Parameters
*poly_modules (torch.nn.Module) – Variable arg of modules to use. If empty, acts as an identity. For single module acts like
ResNet
.2
was used in original paper. All modules needinputs
andoutputs
of equalshape
.
-
forward
(inputs)[source]¶ Defines the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.