ɀ^8 @ s d d l Z d d l Z d d l Z d d l Z d d l m Z d d l m Z d d l m Z m
Z
m Z m Z d d l
m Z m Z y d d l m Z m Z Wn Yn Xy d d l m Z Wn d d l m Z Yn Xd d l m Z d d
l Td d
l Td d Z e d
d d d Z d d Z d d Z d d Z d d d d Z e d
d d d Z e d
d d d Z! e d
d d d! Z" d" d# Z# d d$ d% Z$ d& d' Z% e d
d d( d) Z& e d
d d* d+ Z' e d
d d, d- Z( d. d d/ d0 Z) d1 Z* e j e* Z+ e j, d2 e+ d3 e+ d4 Z- e d
d d5 d6 Z. e d
d d7 d8 d9 Z/ e d
d: d; d< Z0 d7 d d= d> Z1 d? d@ Z2 dA dB Z3 dC dD Z4 d S)E N)fnmatch)escape)urljoinurlsplit
urlunsplit
quote_plus)
make_responseResponse)UnionTuple) lru_cache )CONSTS)*c C s t j j t j | S)N)ospathjoinr ZMIRROR_ROOT)filename r !/var/www/zmirror/zmirror/utils.py zmirror_root s r maxsizei c C s | j d d S)zM
equivalent to s.replace("/",r"\/")
:type s: str
:rtype: str
/z\/)replace)sr r r s_esc s r c C s | j d j d j d } | j d } t | d d k oS | d d k } t | d k s t | d k r | r t d f S| r d j | d d d j | d d f Sd j | d d d j | d d f Sd S)u
提取出一个域名的根域名
支持二级顶级域名, 允许包含端口(端口会被舍去)
:param domain: eg: dwn.cdn.google.co.jp[:233]
:type domain: str
:return: root_domain, sub_domain
:rtype: Tuple[str, str]
0123456789:.r
comnetorgcoedumilgovac N)zcomznetzorgzcozeduzmilzgovzacr- r, r, )rstripstripsplitlen
target_domainr )domaintempZ
is_level2_tldr r r extract_root_domain) s
!&*
0r5 c 8 C s t d d | d d | d d | d d | d d
| d d | j d d
d d | j d d d t d | d t d | d t d | d d | d d | d d | j d d d d | j d d d d | j d d d d | j d d d d | j d d d d | j d d d t d | d t d | d! t d | d" t d | d# t d
| d$ t t d | d% t t d | d& t t d | S)'uN 生成各种形式的scheme变体
:type _domain: str
:rtype: bool
slashz//httpzhttp://httpszhttps://Z
double_quotedz"%s"
single_quotedz'%s'Z hex_lowerr z\x2fZ hex_upperz\x2FZ slash_escZhttp_escZ https_escZdouble_quoted_escz\"%s\"Zsingle_quoted_escz\'%s\'Zslash_double_escz\\/Zhttp_double_escZhttps_double_escZslash_triple_escz\\\/Zhttp_triple_escZhttps_triple_escZslash_ueZhttp_ueZhttps_ueZdouble_quoted_ueZsingle_quoted_ueZ slash_esc_ueZ http_esc_ueZ https_esc_ue)dictr r r )_domainr r r calc_domain_replace_prefixC s6
r< c C s d d l } | j j j S)zeReturns the current line number in our program.
:return: current line number
:rtype: int
r N)inspect currentframef_backf_lineno)r= r r r current_line_numberm s rA s We Got An Unknown Errori c C s
t | | S)zO
:type errormsg: bytes
:type error_code: int
:rtype: Response
)r )errormsg
error_coder r r generate_simple_resp_pagev s rD c C s1 | j } x t D] } | | k r d Sq Wd S)z
Determine whether an mime is text (eg: text/html: True, image/png: False)
:param input_mime: str
:return: bool
TF)lowertext_like_mime_keywords)Z
input_mimeZ input_mime_lZ text_wordr r r is_mime_represents_text s
rH c C s= | j d } | d k r% | j S| d | j Sd S)ue 从content-type中提取出mime, 如 'text/html; encoding=utf-8' --> 'text/html'
:rtype: str
;r
Nr+ )findrF )
_content_typecr r r extract_mime_from_content_type s
rM c C s$ t | } | t k r | Sd Sd S)u0 根据content-type确定该资源是否使用CDNFN)rM mime_to_use_cdn)rK _mimer r r is_content_type_using_cdn s rP c C s: y | j | } Wn d SYn X| d k r2 | Sd Sd S)z0return a blank string if the match group is Noner* N)group)name match_objobjr r r get_group s rU c C s d S)uC 旧版本遗留函数, 已经不再需要, 永远返回空字符串r* r )Z
ext_domainZ force_httpsr r r "get_ext_domain_inurl_scheme_prefix s rV c G s< d } x" | D] } | t | d 7} q
W| j d | S)z
:return: str
r* )strr. )argsoutputargr r r strx s
r\ c C s; | d k s t r d S| j } t | k r3 d Sd Sd S)u( 该user-agent是否满足全局白名单NFT)global_ua_white_namerF )ua_strr r r check_global_ua_pass s r_ c C s( x! t D] } t | | r d Sq Wd S)u
域名是否匹配 `domains_whitelist_auto_add_glob_list` 中设置的通配符
:type domain: str
:rtype: bool
TF)$domains_whitelist_auto_add_glob_listr )r3 Z domain_globr r r is_domain_match_glob_whitelist s
ra c C s% x t D] } | | k r d Sq Wd S)u#
根据content-type判断是否应该用stream模式传输(服务器下载的同时发送给用户)
视频/音频/图片等二进制内容默认用stream模式传输
:param mime: mime or content-type, eg: "plain/text; encoding=utf-8"
:type mime: str
:rtype: bool
TF)steamed_mime_keywords)mimeZstreamed_keywordr r r is_mime_streamed s
rd r* c C s] d | t | t | | d t | t | t | f } | j d } t d | S)u 生成一个HTML重定向页面
某些浏览器在301/302页面不接受cookies, 所以需要用html重定向页面来传cookie
:type target_url: str
:type msg: str
:type delay_sec: int
:rtype: Response
u
重定向 (Page Redirect)
%s
You are now redirecting to %s, if it didn't redirect automatically, please click that link.
r
zutf-8response) html_escapeencoder )Z
target_urlmsgZ delay_secZ resp_contentr r r generate_html_redirect_page s
"ri Zzm26_z+(?Pz?)_\.(?P[a-zA-Z0-9-_]+=*)\._z_\.[a-zA-Z\d]+\bc C s d t d | d d k r" d St j | } t d | } | d | j d } t j | } t d | } | r t j | } | j d d } t
| d
| } | S) u4
将 embed_real_url_to_embedded_url() 编码后的url转换为原来的带有参数的url
`cdn_redirect_encode_query_str_into_url`设置依赖于本函数, 详细说明请看配置文件中这个参数的部分
eg: https://cdn.domain.com/a.php_zm24_.cT1zb21ldGhpbmc=._zm24_.css
---> https://foo.com/a.php?q=something (assume it returns an css) (base64 only)
eg2: https://cdn.domain.com/a/b/_zm24_.bG92ZT1saXZl._zm24_.jpg
---> https://foo.com/a/b/?love=live (assume it returns an jpg) (base64 only)
eg3: https://cdn.domain.com/a/b/_zm24z_.[some long long base64 encoded string]._zm24_.jpg
---> https://foo.com/a/b/?love=live[and a long long query string] (assume it returns an jpg) (gzip + base64)
eg4:https://cdn.domain.com/a (no change)
---> (no query string): https://foo.com/a (assume it returns an png) (no change)
:param embedded_url: 可能被编码的URL
:return: 如果传入的是编码后的URL, 则返回解码后的URL, 否则返回None
:type embedded_url: str
:rtype: Union[str, None]
z._z_. Nb64r gzipencodingzutf-8?i) cdn_url_query_encode_salt®ex_extract_base64_from_embedded_urlsearchrU spanbase64urlsafe_b64decodezlib
decompressdecoder )Z embedded_urlmrl Zreal_request_url_no_queryZquery_string_byteZ
is_gzipped query_stringresultr r r "extract_real_url_from_embedded_url s r| Fc
C s | r | j d d } n | } t | } | j s: | S| j j } t | d k rs d } t j | } n d } t j | j } | j
d t | d | d t d t | } t
| j | j | d d f } | r t | } | S) u
将url的参数(?q=some&foo=bar)编码到url路径中, 并在url末添加一个文件扩展名
在某些对url参数支持不好的CDN中, 可以减少错误
`cdn_redirect_encode_query_str_into_url`设置依赖于本函数, 详细说明可以看配置文件中的对应部分
解码由 extract_real_url_from_embedded_url() 函数进行, 对应的例子也请看这个函数
:rtype: str
z\/r rE zr* rj z_.z._)r r queryrg r1 rv compressrt urlsafe_b64encoderx r _url_saltrN r schemenetlocr )
Z real_url_rawurl_mime escape_slashreal_urlZurl_spZ
byte_queryZ
gzip_labelZ b64_queryZ
mixed_pathr{ r r r embed_real_url_to_embedded_url7 s"
1! r @ c C sK d | k r d Sd | k r3 d | k r, d Sd Sn d | k rC d Sd Sd S)
uL 根据 slash(/) 的格式, 猜测最有可能与之搭配的 colon(:) 格式%r z%25Fz%253Az%253az%3Az%3aNr )r6 r r r guess_colon_from_slash] s r c C s | r i n d } x t | D] } | d d k r | d d d k r t t | | } | r t | } | | k r | d | d j | } | r | | | 之前
如果head中不存在