o ,h@sddlZddlmZddlmZmZmZddlZddlm m Z ddlm Z m Z ddl mZmZddlmZddlmZmZdd lmZd d lmZmZmZd d lmZd d lmZmZgdZ dej dej fddZ!ej"#ddej dej de$e%dej fddZ&ej"#dGddde j'Z(Gddde j'Z)      d[de de d e d!e de$e%d"e%d#e$e%d$e*d%e*d&ee d'ee d(eej d)e+de fd*d+Z,ej"#d+Gd,d-d-e j'Z-Gd.d/d/e-Z.Gd0d1d1e j'Z/Gd2d3d3e/Z0Gd4d5d5e j'Z1d6e$e%d7e%d8e$e%d"e$e%de$e%d9e*d:eed;e+dZ2d?eiZ3Gd@dAdAeZ4GdBdCdCeZ5GdDdEdEeZ6GdFdGdGeZ7GdHdIdIeZ8GdJdKdKeZ9eedLe4j:fdMdddNd:ee4d;e+deedLe8j:fdMdddNd:ee8d;e+d&rAcr@) PatchMergingV2zPatch Merging Layer for Swin Transformer V2. Args: dim (int): Number of input channels. norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. rBrC.csDtt|||_tjd|d|dd|_|d||_dSrDrHrNrPr3r4rJ`rRzPatchMergingV2.__init__r#cCrSrT)r5rLrMrUr3r3r4rVgrWzPatchMergingV2.forwardrXr3r3rPr4rbYrarbTinput qkv_weight proj_weightr> num_heads shift_sizeattention_dropoutdropoutqkv_bias proj_bias logit_scaletrainingc $ Cs|j\} }}}|d||d|d}|d||d|d}t|ddd|d|f}|j\}}}}|}|d|krDd|d<|d|krNd|d<t|dkrdtj||d |d fdd}||d||d}|| ||d|d||d|d|}|dddddd | ||d|d|}| d ur| d ur| } | d}| |d| t ||| }| |d|dd|||ddddd}|d|d|d}}}| d urtj|d d tj|d d d d }tj| td d} || }n|||d}||d d }||}t|dkr|||f}d|d f|d |d f|d d ff}d|d f|d |d f|d d ff} d}!|D]}"| D]}#|!||"d|"d|#d|#df<|!d7}!qgqc|||d|d||d|d}|dddd ||d|d}|d|d}||dktd|dktd}||d||||d|d}||dd}|d ||d|d}tj|d d }tj||| d}||dd |d|d|}t ||| }tj||| d}|| ||d||d|d|d|}|dddddd | |||}t|dkr^tj||d|dfdd}|d d d |d |d d f}|S)a Window based multi-head self attention (W-MSA) module with relative position bias. It supports both of shifted and non-shifted window. Args: input (Tensor[N, H, W, C]): The input tensor or 4-dimensions. qkv_weight (Tensor[in_dim, out_dim]): The weight tensor of query, key, value. proj_weight (Tensor[out_dim, out_dim]): The weight tensor of projection. relative_position_bias (Tensor): The learned relative position bias added to attention. window_size (List[int]): Window size. num_heads (int): Number of attention heads. shift_size (List[int]): Shift size for shifted window attention. attention_dropout (float): Dropout ratio of attention weight. Default: 0.0. dropout (float): Dropout ratio of output. Default: 0.0. qkv_bias (Tensor[out_dim], optional): The bias tensor of query, key, value. Default: None. proj_bias (Tensor[out_dim], optional): The bias tensor of projection. Default: None. logit_scale (Tensor[out_dim], optional): Logit scale of cosine attention for Swin Transformer V2. Default: None. training (bool, optional): Training flag used by the dropout parameters. Default: True. Returns: Tensor[N, H, W, C]: The output tensor after shifted window attention. rr)rr)shiftsdimsrrENr&)rBgY@)maxggYrc)prn)r'r(r)copysumr*rollr9r:reshapeclonenumelzero_linearsize normalize transposeclampmathlogexpmatmul new_zerosr< masked_fillfloatsoftmaxrjr;)$rdrerfr>r8rgrhrirjrkrlrmrnBr,r-Cpad_rpad_br#r.pad_Hpad_W num_windowslengthqkvqkvattn attn_maskh_slicesw_slicescounthwr3r3r4shifted_window_attentiontsr#    ,. 0 $ ..$ ($$((, $rcseZdZdZ    ddedeedeededed ed ed effd d ZddZ ddZ de j fddZ de de fddZZS)ShiftedWindowAttentionz/ See :func:`shifted_window_attention`. TrcrBr8rhrgrkrlrirjc stt|dkst|dkrtd||_||_||_||_||_t j ||d|d|_ t j |||d|_ | |dS)Nrz.window_size and shift_size must be of length 2rqrF)rIrJlen ValueErrorr8rhrgrirjrrKrproj#define_relative_position_bias_tabledefine_relative_position_index) rOrBr8rhrgrkrlrirjrPr3r4rJs  zShiftedWindowAttention.__init__cCsLttd|jddd|jdd|j|_tjj|jdddS)Nrrr{Gz?std) r Parameterr*zerosr8rgr6init trunc_normal_rOr3r3r4r s,z:ShiftedWindowAttention.define_relative_position_bias_tablecCst|jd}t|jd}ttj||dd}t|d}|dddddf|dddddf}|ddd}|dddddf|jdd7<|dddddf|jdd7<|dddddfd|jdd9<|d}| d|dS)Nrrijindexingrr&r7) r*aranger8stackmeshgridflattenr:r;rwregister_buffer)rOcoords_hcoords_wcoordscoords_flattenrelative_coordsr7r3r3r4rs ,((,z5ShiftedWindowAttention.define_relative_position_indexr$cCst|j|j|jSN)r?r6r7r8rr3r3r4get_relative_position_biass z1ShiftedWindowAttention.get_relative_position_biasr#cCsD|}t||jj|jj||j|j|j|j|j |jj |jj |j d S) Args: x (Tensor): Tensor with layout of [B, H, W, C] Returns: Tensor with same layout as input, i.e. [B, H, W, C] )rhrirjrkrlrn) rrrweightrr8rgrhrirjrGrnrOr#r>r3r3r4rV#szShiftedWindowAttention.forwardTTrcrc)rYrZr[r\r^listboolrrJrrr*rrrVr`r3r3rPr4rs6  rcszeZdZdZ    ddedeedeededed ed ed effd d ZddZ de j fddZ de fddZ ZS)ShiftedWindowAttentionV2z2 See :func:`shifted_window_attention_v2`. TrcrBr8rhrgrkrlrirjc stj||||||||dttdt|ddf|_ttj ddddtj ddtj d|d d|_ |rP|j j d } |j j | d| jdSdS) N)rkrlrirj rriTrF)inplaceFrq)rIrJrrr*ronesrm SequentialrKReLUcpb_mlprrGr{datar|) rOrBr8rhrgrkrlrirjrrPr3r4rJ@s$ " &z!ShiftedWindowAttentionV2.__init__cCstj|jdd |jdtjd}tj|jdd |jdtjd}ttj||gdd}|dddd}|dddddddf|jdd<|dddddddf|jdd<|d9}t |t t |dd }| d |dS) Nrr)dtyperrrg?g@relative_coords_table) r*rr8float32rrr:r;r<signlog2absr)rOrelative_coords_hrelative_coords_wrr3r3r4r_s$$.. zr3r3r4rosz3ShiftedWindowAttentionV2.get_relative_position_biasr#cCsH|}t||jj|jj||j|j|j|j|j |jj |jj |j |j d S)r)rhrirjrkrlrmrn) rrrrrr8rgrhrirjrGrmrnrr3r3r4rVxs z ShiftedWindowAttentionV2.forwardr)rYrZr[r\r^rrrrJrr*rrrVr`r3r3rPr4r;s4   rceZdZdZddddejefdededeedeede d e d e d e d e d ej fde d ej fffdd Z de fddZZS)SwinTransformerBlocka Swin Transformer Block. Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. window_size (List[int]): Window size. shift_size (List[int]): Shift size for shifted window attention. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.0. dropout (float): Dropout rate. Default: 0.0. attention_dropout (float): Attention dropout rate. Default: 0.0. stochastic_depth_prob: (float): Stochastic depth rate. Default: 0.0. norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. attn_layer (nn.Module): Attention layer. Default: ShiftedWindowAttention @rcrBrgr8rh mlp_ratiorjristochastic_depth_probrC. attn_layerc stt|| ||_| ||||||d|_t|d|_| ||_t|t |||gt j d|d|_ |j D]} t| t jrWt j| j| jdurWt jj| jddq:dS)N)rirjrow)activation_layerrrjgư>r)rIrJrnorm1rr stochastic_depthnorm2r r^rGELUmlpmodules isinstancerKrxavier_uniform_rrGnormal_) rOrBrgr8rhrrjrirrCrmrPr3r4rJs*    "  zSwinTransformerBlock.__init__r#cC8|||||}|||||}|Sr)rrrrrrUr3r3r4rVszSwinTransformerBlock.forward)rYrZr[r\rr]rr^rrrr_rJrrVr`r3r3rPr4rs<   #rcr)SwinTransformerBlockV2a Swin Transformer V2 Block. Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. window_size (List[int]): Window size. shift_size (List[int]): Shift size for shifted window attention. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.0. dropout (float): Dropout rate. Default: 0.0. attention_dropout (float): Attention dropout rate. Default: 0.0. stochastic_depth_prob: (float): Stochastic depth rate. Default: 0.0. norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. attn_layer (nn.Module): Attention layer. Default: ShiftedWindowAttentionV2. rrcrBrgr8rhrrjrirrC.rc s$tj||||||||| | d dS)N)rrjrirrCr)rIrJ) rOrBrgr8rhrrjrirrCrrPr3r4rJs  zSwinTransformerBlockV2.__init__r#cCrr)rrrrrrUr3r3r4rVszSwinTransformerBlockV2.forward)rYrZr[r\rr]rr^rrrr_rJrrVr`r3r3rPr4rs<   rcseZdZdZdddddddefdeeded eed eed eed ed ededededee de j fdee de j fde de j fffdd Z ddZ ZS)ra; Implements Swin Transformer from the `"Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" `_ paper. Args: patch_size (List[int]): Patch size. embed_dim (int): Patch embedding dimension. depths (List(int)): Depth of each Swin Transformer layer. num_heads (List(int)): Number of attention heads in different layers. window_size (List[int]): Window size. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.0. dropout (float): Dropout rate. Default: 0.0. attention_dropout (float): Attention dropout rate. Default: 0.0. stochastic_depth_prob (float): Stochastic depth rate. Default: 0.1. num_classes (int): Number of classes for classification head. Default: 1000. block (nn.Module, optional): SwinTransformer Block. Default: None. norm_layer (nn.Module, optional): Normalization layer. Default: None. downsample_layer (nn.Module): Downsample layer (patch merging). Default: PatchMerging. rrcg?iN patch_size embed_dimdepthsrgr8rrjrir num_classesrC.blockdownsample_layercstt|| |_| durt} | durttjdd} g}|t tj d||d|df|d|dfdt gd| |t |}d}t t|D]R}g}|d|}t ||D])| t||d}|| ||||fd d |D||||| d |d7}q^|tj ||t|dkr|| || qNtj ||_|dt|d}| ||_t gd |_td|_td|_t|| |_|D]}t|tjrtjj|jd d|jdurtj|jqdS)Ngh㈵>)epsrqrr) kernel_sizestride)rrrqrrcs$g|]}ddkr dn|dqS)rrr3).0ri_layerr3r4 Cs$z,SwinTransformer.__init__..)r8rhrrjrirrC)rrqrrrr) rIrJrrrrrr]appendrConv2dr rwrangerrfeaturesrMr:AdaptiveAvgPool2davgpoolFlattenrrKheadrrrrrrGzeros_)rOrrrrgr8rrjrirrrCrrlayerstotal_stage_blocksstage_block_idi_stagestagerBsd_prob num_featuresrrPrr4rJsp             zSwinTransformer.__init__cCs@||}||}||}||}||}||}|Sr)rrMr:rrrrUr3r3r4rV_s      zSwinTransformer.forward)rYrZr[r\rArr^rrrrr_rJrVr`r3r3rPr4rsL      Orrrrrweightsprogresskwargsc KsZ|durt|dt|jdtd||||||d|} |dur+| |j|dd| S)Nr categories)rrrrgr8rT)r check_hashr3)rrmetarload_state_dictget_state_dict) rrrrgr8rrrrmodelr3r3r4_swin_transformeris  rrc @NeZdZedeeddejdieddddd d d id d dddZ e Z dS)rz7https://download.pytorch.org/models/swin_t-704ceda3.pth crop_size resize_size interpolationibrrUhttps://github.com/pytorch/vision/tree/main/references/classification#swintransformer ImageNet-1KguV^T@glW@zacc@1zacc@5gX9@g\( [@YThese weights reproduce closely the results of the paper using a similar training recipe. num_paramsmin_sizerecipe_metrics_ops _file_size_docsurl transformsr N rYrZr[rrr r BICUBIC _COMMON_META IMAGENET1K_V1DEFAULTr3r3r3r4r. rc @r)rz7https://download.pytorch.org/models/swin_s-5e29d889.pthrrirrrrgCT@gףp= X@rgZd{!@gx&g@rrr#Nr&r3r3r3r4rr+rc @r)rz7https://download.pytorch.org/models/swin_b-68c6b09e.pthrri<;rrrgh|?T@g)\(X@rg&1.@gt@rrr#Nr&r3r3r3r4rr+rc @r)rz:https://download.pytorch.org/models/swin_v2_t-b137f0e2.pthriRr.r.Xhttps://github.com/pytorch/vision/tree/main/references/classification#swintransformer-v2rgS㥛T@g rX@rg(\@gMb([@rrr#Nr&r3r3r3r4rr+rc @r)rz:https://download.pytorch.org/models/swin_v2_s-637d8ceb.pthr.r/rir0r1rg!rhT@gNbX94X@rgd;O'@gg@rrr#Nr&r3r3r3r4rr+rc @r)rz:https://download.pytorch.org/models/swin_v2_b-781e5279.pthr.iri=r0r1rgI +U@gK7X@rg33333S4@gˡEu@rrr#Nr&r3r3r3r4r r+r pretrained)r)rrc K:t|}td ddgdgdgdddgd||d|S) a Constructs a swin_tiny architecture from `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows `_. Args: weights (:class:`~torchvision.models.Swin_T_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_T_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_T_Weights :members: rE`rrrrqr6 皙?rrrrgr8rrrNr3)rverifyrrrrr3r3r4r"  rc Kr3) a Constructs a swin_small architecture from `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows `_. Args: weights (:class:`~torchvision.models.Swin_S_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_S_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_S_Weights :members: rEr4rrrr7r:333333?r<Nr3)rr=rr>r3r3r4rHr?rc Kr3) a Constructs a swin_base architecture from `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows `_. Args: weights (:class:`~torchvision.models.Swin_B_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_B_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_B_Weights :members: rEr@rErr r:?r<Nr3)rr=rr>r3r3r4rnr?rc K>t|}td ddgdgdgdddgd||ttd |S) a Constructs a swin_v2_tiny architecture from `Swin Transformer V2: Scaling Up Capacity and Resolution `_. Args: weights (:class:`~torchvision.models.Swin_V2_T_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_V2_T_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_V2_T_Weights :members: rEr4r5r7rr; rrrrgr8rrrrrNr3)rr=rrrbr>r3r3r4r   r c KrG) a Constructs a swin_v2_small architecture from `Swin Transformer V2: Scaling Up Capacity and Resolution `_. Args: weights (:class:`~torchvision.models.Swin_V2_S_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_V2_S_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_V2_S_Weights :members: rEr4r@r7rrBrHNr3)rr=rrrbr>r3r3r4r!rIr!c KrG) a Constructs a swin_v2_base architecture from `Swin Transformer V2: Scaling Up Capacity and Resolution `_. Args: weights (:class:`~torchvision.models.Swin_V2_B_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_V2_B_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_V2_B_Weights :members: rErCr@rDrrFrHNr3)rr=rrrbr>r3r3r4r"rIr")rcrcNNNT)Ar functoolsrtypingrrrr*torch.nn.functionalr functionalr(rops.miscr r ops.stochastic_depthr transforms._presetsr r utilsr_apirrr_metar_utilsrr__all__r5fxwraprr^r?r_rArbrrrrrrrrrr(rrrrrrr)rrrr r!r"r3r3r3r4s      #      sQV92m  *$*$*$*&*&.