o ,h|}@sUddlZddlmZddlmZddlmZmZmZm Z ddl Z ddl m Z ddl mZmZddlmZmZddlmZd d lmZmZmZd d lmZd d lmZmZgd ZGdddeZGdddeZ Gddde j!Z"Gddde j!Z#Gddde j!Z$de%de%de%de%de%de ede&ded e$fd!d"Z'd#eiZ(e)e*efe+d$<ie(d%d&d'Z,Gd(d)d)eZ-Gd*d+d+eZ.Gd,d-d-eZ/Gd.d/d/eZ0Gd0d1d1eZ1eed2e-j2fd3dd4d5de e-de&ded e$fd6d7Z3eed2e.j2fd3dd4d5de e.de&ded e$fd8d9Z4eed2e/j2fd3dd4d5de e/de&ded e$fd:d;Z5eed2e0j2fd3dd4d5de e0de&ded e$fdd3dd4d5de e1de&ded e$fd?d@Z7 A BdJdCe%de%dDdEdFe*dGe&d dEf dHdIZ8dS)KN) OrderedDict)partial)AnyCallable NamedTupleOptional)Conv2dNormActivationMLP)ImageClassificationInterpolationMode)_log_api_usage_once)register_modelWeights WeightsEnum)_IMAGENET_CATEGORIES)_ovewrite_named_paramhandle_legacy_interface) VisionTransformerViT_B_16_WeightsViT_B_32_WeightsViT_L_16_WeightsViT_L_32_WeightsViT_H_14_Weightsvit_b_16vit_b_32vit_l_16vit_l_32vit_h_14c@sVeZdZUeed<eed<eed<ejZedej fed<ej Z edej fed<dS)ConvStemConfig out_channels kernel_sizestride. norm_layeractivation_layerN) __name__ __module__ __qualname__int__annotations__nn BatchNorm2dr$rModuleReLUr%r/r/`/var/www/html/scripts/venv/lib/python3.10/site-packages/torchvision/models/vision_transformer.pyr s r cs>eZdZdZdZdededeffdd Zfdd ZZ S) MLPBlockzTransformer MLP block.rin_dimmlp_dimdropoutcsdtj|||gtjd|d|D]}t|tjr/tj|j |j dur/tjj |j ddqdS)N)r%inplacer4ư>std) super__init__r+GELUmodules isinstanceLinearinitxavier_uniform_weightbiasnormal_)selfr2r3r4m __class__r/r0r:-s   zMLPBlock.__init__c s|dd}|dus|dkrTanhheadsr=rr? trunc_normal_rAmathsqrtrBzeros_rrChasattrr in_featuresr)rDrrr|rbrcr3r4rdrrr$rseq_proj prev_channelsrWconv_stem_layer_configr{ heads_layersfan_inrFr/r0r:s            zVisionTransformer.__init__rsreturnc Cs|j\}}}}|j}t||jkd|jd|dt||jkd|jd|d||}||}||}|||j||}|ddd}|S)NzWrong image height! Expected z but got !zWrong image width! Expected rrr) rrrrorprrreshapercpermute) rDrsnchwpn_hn_wr/r/r0_process_input s"" z VisionTransformer._process_inputcCs^||}|jd}|j|dd}tj||gdd}||}|dddf}||}|S)Nrrrq)rrrrexpandrocatrr)rDrsrbatch_class_tokenr/r/r0rv!s    zVisionTransformer.forward)r&r'r(r[rr+rxr)r]rrror-listr r:ryrrvr^r/r/rFr0rsF       irrr|rbrcr3weightsprogresskwargsrc Ks|dur*t|dt|jd|jdd|jddksJt|d|jdd|dd}td ||||||d|} |rJ| |j|d d | S) Nr categoriesmin_sizerrr)rrr|rbrcr3T)r check_hashr/)rlenmetarNrload_state_dictget_state_dict) rr|rbrcr3rrrrmodelr/r/r0_vision_transformer4s$    rr _COMMON_METAz(https://github.com/facebookresearch/SWAGz:https://github.com/facebookresearch/SWAG/blob/main/LICENSE)recipelicensec @seZdZedeeddiedddddd d id d d ddZedeeddej die dddddd idddddZ edeeddej die ddddddd id d dd dZ eZ d!S)"rz9https://download.pytorch.org/models/vit_b_16-c867db91.pthr crop_sizei(rrzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_b_16 ImageNet-1KgS㥛DT@g1ZW@zacc@1zacc@5gMb1@g(\t@ These weights were trained from scratch by using a modified version of `DeIT `_'s training recipe.  num_paramsrr_metrics_ops _file_size_docsurl transformsrz>https://download.pytorch.org/models/vit_b_16_swag-9ac1b537.pthr resize_size interpolationi^-)rrg~jtSU@giX@gˡEK@g|?5^t@ These weights are learnt via transfer learning by end-to-end fine-tuning the original `SWAG `_ weights on ImageNet-1K data. rrrrrrzAhttps://download.pytorch.org/models/vit_b_16_lc_swag-4e70ced5.pth+https://github.com/pytorch/vision/pull/5793gbX9xT@gQ X@ These weights are composed of the original frozen `SWAG `_ trunk weights and a linear classifier learnt on top of them trained on ImageNet-1K data. rrrrrrrNr&r'r(rrr r IMAGENET1K_V1r BICUBIC_COMMON_SWAG_METAIMAGENET1K_SWAG_E2E_V1IMAGENET1K_SWAG_LINEAR_V1DEFAULTr/r/r/r0r_s rc @HeZdZedeeddiedddddd d id d d ddZeZdS)rz9https://download.pytorch.org/models/vit_b_32-d86f8d99.pthrri1BrzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_b_32rg|?5^R@gW@rgA`Т@gl u@rrrN r&r'r(rrr rrrr/r/r/r0r* rc @seZdZedeedddieddddd d d id d dddZedeeddej die dddddd idddddZ edeeddej die ddddddd id d d d!dZ eZ d"S)#rz9https://download.pytorch.org/models/vit_l_16-852ce7e3.pthr)rri#rzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_l_16rg|?5^S@gFԨW@rgףp= N@g;O$@a These weights were trained from scratch by using a modified version of TorchVision's `new training recipe `_. rrz>https://download.pytorch.org/models/vit_l_16_swag-4f3808c9.pthri0)rrgjtV@gT㥛ĠX@gƟv@gy&11@rrzAhttps://download.pytorch.org/models/vit_l_16_lc_swag-4d563306.pthrgMbXIU@g^I [X@rrNrr/r/r/r0rs rc @r)rz9https://download.pytorch.org/models/vit_l_32-c7638314.pthrri[ErzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_l_32rg|?5>S@gGzDW@rgK7.@gE@rrrNrr/r/r/r0rrrc @seZdZedeeddejdiedddddd id d d d dZ edeeddejdieddddddd idddddZ e Z dS)rz>https://download.pytorch.org/models/vit_h_14_swag-80465313.pthri%)rrrgS#V@g#~jX@rg~jŏ@gK7I@rrrzAhttps://download.pytorch.org/models/vit_h_14_lc_swag-c1eb923e.pthrri@%rgZd;OmU@gQnX@g= ףpd@gI k֢@rrN) r&r'r(rrr r rrrrrr/r/r/r0r2sbr pretrained)rT)rrc K(t|}tdddddd||d|S)a Constructs a vit_b_16 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_B_16_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_B_16_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_B_16_Weights :members:   rr|rbrcr3rrNr/)rverifyrrrrr/r/r0rk rc Kr)a Constructs a vit_b_32 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_B_32_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_B_32_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_B_32_Weights :members: rrrrNr/)rrrrr/r/r0rrrc Ks(t|}tdddddd||d|S)a Constructs a vit_l_16 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_L_16_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_L_16_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_L_16_Weights :members: rrNr/)rrrrr/r/r0rrrc K(t|}tdddddd||d|S) a Constructs a vit_l_32 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_L_32_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_L_32_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_L_32_Weights :members: rrrrrrNr/)rrrrr/r/r0rrr)rNc Kr) a Constructs a vit_h_14 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_H_14_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_H_14_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_H_14_Weights :members: rriirNr/)rrrrr/r/r0rrrbicubicFr model_statezOrderedDict[str, torch.Tensor]interpolation_mode reset_headscCs`|d}|j\}}}|dkrtd|j||dd} | |kr|d8}| d8} |ddddddf} |ddddddf} | ddd} tt|} | | |krdtd| | d|| d|| | } ||} tjj | | |d d }|d|| }|ddd}t j | |gdd }||d<|rt }| D] \}}|d s|||<q|}|S) aThis function helps interpolate positional embeddings during checkpoint loading, especially when you want to apply a pre-trained model on images with different resolution. Args: image_size (int): Image size of the new model. patch_size (int): Patch size of the new model. model_state (OrderedDict[str, torch.Tensor]): State dict of the pre-trained model. interpolation_mode (str): The algorithm used for upsampling. Default: bicubic. reset_heads (bool): If true, not copying the state of heads. Default: False. Returns: OrderedDict[str, torch.Tensor]: A state dict which can be loaded into the new model. zencoder.pos_embeddingrz%Unexpected position embedding shape: rNrzPseq_length is not a perfect square! Instead got seq_length_1d * seq_length_1d = z and seq_length = T)sizemode align_cornersrr)rr ValueErrorrr)rrrr+ functional interpolaterorritems startswith)rrrrrrrr{rcnew_seq_lengthpos_embedding_tokenpos_embedding_img seq_length_1dnew_seq_length_1dnew_pos_embedding_imgnew_pos_embeddingmodel_state_copykvr/r/r0interpolate_embeddingssH   r)rF)9r collectionsr functoolsrtypingrrrrrotorch.nnr+ops.miscr r transforms._presetsr r utilsr _apirrr_metar_utilsrr__all__r r1r-r_rzrr)boolrrdictstrr*rrrrrrrrrrrrrr/r/r/r0s     .$& !OP9* * * * *$