o ,h@sddlZddlmZddlmZddlmZddlmZm Z m Z ddl Z ddl Z ddl mZddlmZmZddlmZdd lmZd d lmZmZmZd d lmZd d lmZmZgdZeGdddZ dee!de!fddZ"de j#de!de!de$e j#e!ffddZ%de j#de!de!de!de j#f ddZ&e j'(de j'(dGdddej)Z*de j#d e!de j#fd!d"Z+d#e j#d$e j#d%e$e!e!e!fd&e$e!e!e!fd'e j#d(e j#d)e j#de j#fd*d+Z,de j#d,e j#d-e-fd.d/Z.e j'(d+e j'(d/Gd0d1d1ej)Z/Gd2d3d3ej)Z0Gd4d5d5ej)Z1Gd6d7d7ej)Z2d8e3e d9e4d:e ed;e-dZ5Gd?d@d@eZ6GdAdBdBeZ7eedCe6j8fdDddEdFd:e e6d;e-dr?r@layers __class__r'r(rBBs    z Pool.__init__r0thwc Cst|dd\}}tj|ddd\}}|dd}|jdd\}}}||||f|}|jr<|jdur<||}| |}|jdd\}} } ||||ddd}tj ||fdd}|jsm|jdurm||}t |dd|}||| | ffS) Nr,r,r )indicesr3rr3) r9torch tensor_split transposer6reshape contiguousr@rFr=catr;) rGr0rKr8 class_tokenBNCTHWr'r'r(forwardSs    z Pool.forward)NF)r!r"r#rDModulerboolrBrQTensortupler$r^ __classcell__r'r'rIr(r<As>r< embeddingdcCs@|jd|kr |Stjj|ddd|dddddS)Nrr,linear)sizemode)r6rD functional interpolatepermuter4r:)rdrer'r'r( _interpolatemsrlattnqq_thwk_thw rel_pos_h rel_pos_w rel_pos_tc% Cs6|\}}} |\} } } tdt|| d} tdt| | d}tdt|| d}t| |d}t|| d}t|dddf|t| dddfd| |}t| | d}t| | d}t| dddf|t| dddfd| |}t| |d}t|| d}t|dddf|t| dddfd| |}t|| }t||}t||}||}||}||}|j\}}}}|ddddddf||||| |} td| |}!td| |}"| dddddd ||||| |} t | | dd dd}#|# |||| ||  dddddd }#|!ddddddddddddddf|"ddddddddddddddf|#ddddddddddddddf||||| | | | }$|ddddddddf|$7<|S) Nr r,?zbythwc,hkc->bythwkzbythwc,wkc->bythwkrrrL) r$maxrQarangerllongr6rTeinsumrkmatmulrSview)%rmrnrorprqrrrsq_tq_hq_wk_tk_hk_wdhdwdt q_h_ratio k_h_ratiodist_h q_w_ratio k_w_ratiodist_w q_t_ratio k_t_ratiodist_tRhRwRtrXn_head_r3r_qrel_h_qrel_w_qrel_q_trel_posr'r'r( _add_rel_pos|sH <<<      **$...(rshortcutresidual_with_cls_embedc CsZ|r |||S|ddddddddf|ddddddddf7<|Sr+)add_)r0rrr'r'r( _add_shortcuts  DrcseZdZdejfdeededededeedeedeed eed ed ed ed ede dej fddffdd Z de j deeeefdee j eeeefffddZZS)MultiscaleAttention input_size embed_dim output_dimrrrrr residual_poolr rel_pos_embeddropout norm_layer.r*Nc spt||_||_||_|||_dt|j|_| |_ | |_ t |d||_ t ||g}| dkr@|t j| ddt j||_d|_t|dksUt|dkrrdd|D}tt j|j|j||||jd d | |j|_d|_d|_t|dkst|dkrd d|D}tt j|j|j||||jd d | |j|_tt j|j|j||||jd d | |j|_d|_d|_d|_| r6t|dd}t|d kr||dn|}t|d kr||dn|}d t||d}d |d d}t t||j|_t t||j|_t t||j|_t j j!|jddt j j!|jddt j j!|jdddSdS)NrtrrTinplacer,cSg|]}t|dqSr r$).0rnr'r'r( z0MultiscaleAttention.__init__..F)stridepaddinggroupsbiascSrrr)rkvr'r'r(rrrr {Gz?std)"rArBrrrhead_dimmathsqrtscalerrrrDLinearqkvrCDropoutrEprojectpool_qr/r<Conv3dpool_kpool_vrqrrrsrvlen ParameterrQzerosinit trunc_normal_)rGrrrrrrrr rrrrrrH padding_q padding_kvrgq_sizekv_size spatial_dim temporal_dimrIr'r(rBs         zMultiscaleAttention.__init__r0rKc Cs:|j\}}}||||d|j|jddjdd\}}}|jdur-|||\}} n|} |jdur<|||d}|j durI| ||\}}t |j ||dd} |j durr|jdurr|jdurrt| ||| |j |j|j} | jdd} t | |}|jrt|||j|dd|d|j}||}||fS)Nrr,r rPrrO)r6rrTrrrSunbindrrrrQrzrrqrrrsrsoftmaxrrrrr) rGr0rKrXrYrZrnkr.rprmr'r'r(r^!s6 2     zMultiscaleAttention.forward)r!r"r#rD LayerNormr&r$r`floatrr_rBrQrarbr^rcr'r'rIr(rsB      >\rcseZdZddejfdeedededededede d e d e d ej fd d ffdd Z de jdeeeefd ee jeeeefffddZZS)MultiscaleBlockrrcnfrrrproj_after_attnrstochastic_depth_probr.r*Nc st||_d|_t|jdkr.dd|jD} dd| D} ttj| |j| dd|_|r3|j n|j } | |j |_ | | |_ t |j tj|_t||j | |j|j|j|j|j||||| d |_t| d| |j gtj|dd|_t|d |_d|_|j |j krt|j |j |_dSdS) Nr,cSs g|] }|dkr |dn|qSrMr')rr)r'r'r(rVs z,MultiscaleBlock.__init__..cSrrr)rrr'r'r(rWr)rr) rrrr rrrrrrL)activation_layerrrrow)rArBr pool_skipr/rr<rD MaxPool3drrnorm1norm2 isinstance BatchNorm1dneeds_transposalrrrrr rmr GELUmlpr stochastic_depthrr) rGrrrrrrrrr kernel_skip padding_skipattn_dimrIr'r(rBEsP      zMultiscaleBlock.__init__r0rKc Cs|jr||ddddn||}|||\}}|jdus%|js'|n||}|jdur3|n|||d}|||}|jrR||ddddn||}|jdus_|jra|n||}||| ||fS)Nr,r r) rrrSrmrrrrrr) rGr0rKx_norm1x_attnthw_newx_skipx_norm2x_projr'r'r(r^s**zMultiscaleBlock.forward)r!r"r#rDrr&r$rr`rrr_rBrQrarbr^rcr'r'rIr(rDs4    >:rc sPeZdZdedeeefdededdf fdd Zd ejdejfd d Z Z S) PositionalEncoding embed_size spatial_size temporal_sizerr*Ncst||_||_tt||_d|_ d|_ d|_ |sGtt|jd|jd||_ tt|j||_ tt||_ dSdS)Nrr,) rArBrrrDrrQrrW spatial_pos temporal_pos class_pos)rGrrrrrIr'r(rBs $zPositionalEncoding.__init__r0cCs|j|ddd}tj||fdd}|jdur\|jdur\|jdur\|jj \}}tj |j|dd}| |jd|j dd d|tj|jd|fddd}| ||S)NrrOr,rP)rWexpandrgr4rQrVrrrr6repeat_interleaverrrT)rGr0rWhw_sizer pos_embeddingr'r'r(r^s &  zPositionalEncoding.forward) r!r"r#r$rbr`rBrQrar^rcr'r'rIr(rs*rc$seZdZ         ddeeefd ed eed ed ed edededededede e de j fde e de j fdeeeefdeeeefdeeeefddf"fdd Z dejdejfddZZS) r?rNrrr rLrLr,rrrr block_settingrrrrrattention_dropoutr num_classesblock.rpatch_embed_kernelpatch_embed_stridepatch_embed_paddingr*cstt|t|}|dkrtd| durt} | dur&ttjdd} tj d|dj |||d|_ dd t |f||j j D}t|dj |d |d f|d|d |_t|_t|D]/\}}| ||d }|j| ||||||| || d t|jdkrdd t ||jD}q`| |dj|_ttj|ddt|dj| |_|D][}t|tjrtjj|jddt|tjr|j durtj!|j dqt|tjr|jdurtj!|jd |j durtj!|j dqt|tr |"D] }tjj|ddqqdS)a MViT main class. Args: spatial_size (tuple of ints): The spacial size of the input as ``(H, W)``. temporal_size (int): The temporal size ``T`` of the input. block_setting (sequence of MSBlockConfig): The Network structure. residual_pool (bool): If True, use MViTv2 pooling residual connection. residual_with_cls_embed (bool): If True, the addition on the residual connection will include the class embedding. rel_pos_embed (bool): If True, use MViTv2's relative positional embeddings. proj_after_attn (bool): If True, apply the projection after the attention. dropout (float): Dropout rate. Default: 0.0. attention_dropout (float): Attention dropout rate. Default: 0.0. stochastic_depth_prob: (float): Stochastic depth rate. Default: 0.0. num_classes (int): The number of classes. block (callable, optional): Module specifying the layer which consists of the attention and mlp. norm_layer (callable, optional): Module specifying the normalization layer to use. patch_embed_kernel (tuple of ints): The kernel of the convolution that patchifies the input. patch_embed_stride (tuple of ints): The stride of the convolution that patchifies the input. patch_embed_padding (tuple of ints): The padding of the convolution that patchifies the input. rz+The configuration parameter can't be empty.Ngư>)epsr) in_channels out_channels kernel_sizerrcSg|]\}}||qSr'r'rrgrr'r'r(rrz!MViT.__init__..r,r )rrrrrt) rrrrrrrrrcSrr'r'rr'r'r(r rrOTrrrr)#rArBr rr5rrrDrrr conv_projziprr pos_encoding ModuleListblocks enumeraterCrrr>rErrheadmodulesrrrweightr constant_ parameters)rGrrrrrrrrrrrrrrrrtotal_stage_blocksrstage_block_idrsd_probmweightsrIr'r(rBs )         z MViT.__init__r0cCst|ddd}||}|ddd}||}|jjf|jj}|jD] }|||\}}q'||}|dddf}| |}|S)Nrur rr,) r9r flattenrSr rrr r>r)rGr0rKrr'r'r(r^"s     z MViT.forward) rrrrNNrrr)r!r"r#rbr$rrr`rrrrDr_rBrQrar^rcr'r'rIr(rs\          xrrrrprogresskwargsc Ks|dur1t|dt|jd|jdd|jddksJt|d|jdt|d|jd|dd }|dd }td||||d d |d d|dd |dd |d|}|durk||j|dd|S)Nr categoriesmin_sizerr,rrmin_temporal_sizer rFrTrr)rrrrrrrr)r check_hashr')rrmetapoprload_state_dictget_state_dict)rrrrrrrmodelr'r'r(_mvit9s,         r(c @JeZdZedeedddddddedd d d d d diddd dZeZdS)rz:https://download.pytorch.org/models/mvit_v1_b-dbeb1030.pthr?r-r-?r/r/ crop_size resize_sizemeanrr!zShttps://github.com/facebookresearch/pytorchvideo/blob/main/docs/source/model_zoo.mdThe weights were ported from the paper. The accuracies are estimated on video-level with parameters `frame_rate=7.5`, `clips_per_video=5`, and `clip_len=16`ip. Kinetics-400gJ +S@gh|?eW@zacc@1zacc@5guVQ@g rxa@ rrrrecipe_docs num_params_metrics_ops _file_sizeurl transformsr#N r!r"r#rrr rKINETICS400_V1DEFAULTr'r'r'r(rZ4rc @r))rz:https://download.pytorch.org/models/mvit_v2_s-ae3be167.pthrr*r,r.r0r!zChttps://github.com/facebookresearch/SlowFast/blob/main/MODEL_ZOO.mdr4ir5g r0T@g(\W@r6guVP@g?5^I|`@r7r>NrAr'r'r'r(r{rDr pretrained)rT)rrcKst|}gdgdgdggdggdgggggggggggdgggdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgggdggdgggggggggggdgggdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgd }g}tt|d D],}|t|d ||d ||d ||d ||d||d||d|d qtddd|dd|dd||d|S)a Constructs a base MViTV1 architecture from `Multiscale Vision Transformers `__. .. betastatus:: video module Args: weights (:class:`~torchvision.models.video.MViT_V1_B_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.video.MViT_V1_B_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.video.MViT`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.video.MViT_V1_B_Weights :members: r,r r rLrLrLrLrLrLrLrLrLrLrLrG`rJrKrKrKrKrKrKrKrKrKrKrL)rJrJrKrKrKrKrKrKrKrKrKrKrKrLrLrLrrrr,r r r,rGrGr,rLrLr,r,r,rrrrrrr rrrrrrr rr!Fr皙?)rrrrrrrrNr')rverifyrangerrCrr(r$rrrconfigrir'r'r(rs ..,          rcKsDt|}gdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgdggdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgdggdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgdggdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgd }g}tt|d D],}|t|d ||d ||d ||d ||d||d||d|d qtddd|dddd|dd||d |S)aCConstructs a small MViTV2 architecture from `Multiscale Vision Transformers `__ and `MViTv2: Improved Multiscale Vision Transformers for Classification and Detection `__. .. betastatus:: video module Args: weights (:class:`~torchvision.models.video.MViT_V2_S_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.video.MViT_V2_S_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.video.MViT`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.video.MViT_V2_S_Weights :members: rF)rIrIrJrJrKrKrKrKrKrKrKrKrKrKrKrLrHrMrQrNrOrPrRrrrrrrr rr!TFrrS) rrrrrrrrrrNr')rrTrUrrCrr(r$rVr'r'r(rs N          r);rcollections.abcr dataclassesr functoolsrtypingrrrrQtorch.fxtorch.nnrDopsr r transforms._presetsr utilsr _apirrr_metar_utilsrr__all__rr$r/rarbr9r;fxwrapr_r<rlrr`rrrrrr&rr(rrrBrrr'r'r'r(s         &"  ,   <  H !!!*`.