o ,hB6#@s UddlZddlZddlmZmZddlZddlmZejdedZiZ e ej j efe d<ehdZded ed eegeffd d Zed d d;dddddejdejdejdeejdededed ejfddZdeeded efddZdejdeded ejfd d!Zd"ejd#ejd$ed%edeed&ed ejfd'd(Zd$ed%ed dfd)d*Zd"ejd#ejd$ed%edeed ejf d+d,Zed-d   d op_type opset_versionreturncsdtdtffdd }|S)zDDecorator to register an ONNX operator with a custom implementation.funcrcsPd}tjjdd|dd|}|ttttjj|<|||S)Nopsetzonnx::.) mutates_args)torchlibrary custom_oprgetattropsonnx register_fake)roverloadtorch_opr r rO/var/www/html/scripts/venv/lib/python3.10/site-packages/torch/onnx/ops/_impl.py decorators  z_onnx_op..decorator)r)r r rrrr_onnx_ops r RotaryEmbeddingF) interleaved num_headsrotary_embedding_dimx cos_cache sin_cache position_idsr#r$r%csNjd}jd}tjdkr2jd} t|dkfdd| |} |||| g} t| ttjdkddjd} |dkrJ| }d d d d d d d |f} d d d d d d |d f} |d}|d ur{||}||}n|}|}|d d d d d |f}|d d d d d |f}t|d}t|d}|r| d d d d d d dd df}| d d d d d d dd df}n tj| dd d \}}||||}||||}|rt|d }t|d }tj||fd d }t|| j} n tj||fd d } tj| | fd d }tjdkr%t|j}|S) z_RotaryEmbedding-23 https://onnx.ai/onnx/operators/onnx__RotaryEmbedding.html#rotaryembedding-23rrcs djS)NzKnum_heads must be provided for 3D inputs. Received input tensor with shape )shaperr&rr>s z%rotary_embedding_23..cSdS)Nzx should be a 4D tensor by nowrrrrrr.CNdim)r,lenr_checkreshape unsqueezechunkcat)r&r'r(r)r#r$r% batch_sizesequence_length hidden_size head_size new_shapex_rotate x_not_rotaterotary_embedding_dim_halfcossinx1x2realimagx_rotate_concatoutputrr-rrotary_embedding_23+sn        "$  rKscaler>cCs|dur|Sdt|S)z/Get the scale factor for attention computation.Ng?)mathsqrt)rLr>rrr_get_scale_factorsrOtensorr;cCs:|jd|jd}}||}|||||ddS)z1Reshape 3D tensor to 4D for multi-head attention.rr+)r,view transpose contiguous)rPr;r$r<r=r>rrr_reshape_3d_to_4ds rTQKcurrent_q_num_headscurrent_kv_num_headsqk_matmul_output_modec Cs2|dkr t|||||Stt||ddS)z1Get QK output tensor based on the specified mode.rr2)_compute_qk_output_for_mode_0r zeros_likematmulrR)rUrVrWrXrLrYrrr_get_qk_output_for_aten_spdas   r^cs"tdkfdddS)z-Validate Group Query Attention configuration.rcsdddS)Nz q_num_heads (z%) must be divisible by kv_num_heads (z ) for GQArrrXrWrrr.sz-_validate_gqa_configuration..N)rr6)rWrXrr_r_validate_gqa_configurations  r`c Cs`|}||kr||}|j|dd}t||jd}t|}||} ||} t| | ddS)zDHelper function to compute QK output for qk_matmul_output_mode == 0.rr3r*rZr2)repeat_interleaverOr,rMrNrr]rR) rUrVrWrXrLK_for_qk repeat_factor scale_factor sqrt_scaleQ_scaledK_scaledrrrr[s  r[ Attention) is_causal kv_num_heads q_num_headsrYrLsoftcapsoftmax_precisionV attn_maskpast_key past_valuerjrkrlrmrnc( Csd\} }}t|j}|jd}t|jdkr;t|dko|dkdd|jd}t|||}t|||}t|||}tt|jdkoQt|jdkoQt|jdkdd|j|}t| |} |d urmtj||g|d n|}|d ur~tj||g|d n|}||}}|j| }|j| }|j|}|j|}| d ko| dko| d uo|d up|jtj k}t |||rd }|d ur|jtj kr|n|}tj j j ||||d || t ||kd }t||||| | }n||kr||}|j|| d }|j|| d }tj|||j|jd }|r+t|d uddttj||tj |jd }||td}|d urE|jtj krA||td}n||}t| |jd} t| }!||!}"||!}#t|"|#dd}$|$}|$|}%| dkrq|%}| d kr| t|%| }%| dkr|%}| d ur| tvr|%j}&|%tj| }%tj|%dd }'|'|&}'ntj|%dd }'ntj|%dd }'| dkr|'}t|'|}|dkr|dd !||d}||||fS)zMAttention-23 https://onnx.ai/onnx/operators/onnx__Attention.html#attention-23)rr+r*rr*cSr0)Nz;q_num_heads and kv_num_heads must be provided for 3D inputsrrrrrr.r1zattention_23..rr/cSr0)Nz'Q, K, and V should be 4D tensors by nowrrrrrr.r1Nr3ri)rp dropout_prjrL enable_gqa)dtypedevicecSr0)Nz'Cannot use both is_causal and attn_maskrrrrrr.@r1z-infrZr2r+)"r5r,rr6rTrOr:cloneruboolr`nn functionalscaled_dot_product_attentionr^razerosrvtrilones masked_fillfloatrMrNr]rRtanh-_ATTENTION_23_ALLOWED_INTERMEDIATE_PRECISIONStorONNX_DTYPE_TO_TORCH_DTYPEsoftmaxrSrQ)(rUrVrorprqrrrjrkrlrYrLrmrn num_head_dim sequence_dimhead_diminput_shape_lenr;q_sequence_length q_head_size present_key present_valuerWrXkv_sequence_length can_use_sdpasdpa_attn_maskrJ qk_outputrc attn_bias causal_maskrdrerfrgqk_matmul_output qk_with_biasoriginal_dtype qk_softmaxrrr attention_23s       (                        r)N)NNN)rMtypingrrrtorch.onnx.opsrTypeVarrrdict_ops OpOverload__annotations__ frozensetrstrintr TensorrxrKrrOrTr^r`r[tuplerrrrrs    U