|
Server : Apache System : Linux iad1-shared-b8-43 6.6.49-grsec-jammy+ #10 SMP Thu Sep 12 23:23:08 UTC 2024 x86_64 User : dh_edsupp ( 6597262) PHP Version : 8.2.26 Disable Function : NONE Directory : /lib/python3/dist-packages/future/backports/urllib/__pycache__/ |
Upload File : |
o
,�]� � @ s� d dl mZmZmZ d dlmZ d dlmZ d dlm Z
m Z e
e_ e e_ dgZ
G dd� de�ZG dd� de�ZG d d
� d
e�Zd S ) � )�absolute_import�division�unicode_literals��str)�urllib)�parse�request�RobotFileParserc @ sZ e Zd ZdZddd�Zdd� Zdd� Zd d
� Zd d � Zd
d� Z dd� Z
dd� Z dd� Z dS )r
zs This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
� c C s, g | _ d | _d| _d| _| �|� d| _d S )NFr )�entries�
default_entry� disallow_all� allow_all�set_url� last_checked��self�url� r �E/usr/lib/python3/dist-packages/future/backports/urllib/robotparser.py�__init__ s
zRobotFileParser.__init__c C s | j S )z�Returns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
)r �r r r r �mtime& s zRobotFileParser.mtimec C s ddl }|� � | _dS )zYSets the time the robots.txt file was last fetched to the
current time.
r N)�timer )r r r r r �modified/ s zRobotFileParser.modifiedc C s& || _ tj�|�dd� \| _| _dS )z,Sets the URL referring to a robots.txt file.� � N)r r r �urlparse�host�pathr r r r r 7 s zRobotFileParser.set_urlc
C s� z t j�| j�}W n4 t jjy= } z&|jdv r d| _n|jdkr+d| _W Y d}~dS W Y d}~dS W Y d}~dS d}~ww |� � }| �
|� d�� � � dS )z4Reads the robots.txt URL and feeds it to the parser.)i� i� Ti� Nzutf-8)
r r �urlopenr �error� HTTPError�coder r �readr �decode�
splitlines)r �f�err�rawr r r r% < s
����zRobotFileParser.readc C s2 d|j v r| jd u r|| _d S d S | j�|� d S �N�*)�
useragentsr
r �append)r �entryr r r �
_add_entryI s
�zRobotFileParser._add_entryc C sj d}t � }|D ]�}|s#|dkrt � }d}n|dkr#| �|� t � }d}|�d�}|dkr2|d|� }|�� }|s9q|�dd�}t|�dkr�|d �� �� |d< tj� |d �� �|d< |d dkrx|dkrm| �|� t � }|j
� |d � d}q|d dkr�|dkr�|j � t
|d d �� d}q|d d
kr�|dkr�|j � t
|d d �� d}q|dkr�| �|� dS dS ) z�Parse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines.
r r � �#N�:z
user-agent�disallowF�allowT)�Entryr0 �find�strip�split�len�lowerr r �unquoter- r. � rulelines�RuleLine)r �lines�stater/ �line�ir r r r R sP
� ��zRobotFileParser.parsec C s� | j rdS | jr
dS tj�tj�|��}tj�dd|j|j|j |j
f�}tj� |�}|s.d}| j D ]}|�
|�r?|�|� S q1| jrI| j�|�S dS )z=using the parsed robots.txt decide if useragent can fetch urlFTr �/)r r r r r r< �
urlunparser �params�query�fragment�quoter �
applies_to� allowancer
)r � useragentr �
parsed_urlr/ r r r � can_fetch� s$ �
� zRobotFileParser.can_fetchc C s d� dd� | jD ��S )Nr c S s g | ]}t |�d �qS )�
r )�.0r/ r r r �
<listcomp>� s z+RobotFileParser.__str__.<locals>.<listcomp>)�joinr r r r r �__str__� � zRobotFileParser.__str__N)r )
�__name__�
__module__� __qualname__�__doc__r r r r r% r0 r rM rR r r r r r
s
3 c @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) r> zoA rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.c C s( |dkr|sd}t j�|�| _|| _d S )Nr T)r r rH r rJ )r r rJ r r r r � s
zRuleLine.__init__c C s | j dkp
|�| j �S r+ )r �
startswith)r �filenamer r r rI � rS zRuleLine.applies_toc C s | j rdpdd | j S )N�Allow�Disallowz: )rJ r r r r r rR � s zRuleLine.__str__N)rT rU rV rW r rI rR r r r r r> � s
r> c @ s0 e Zd ZdZdd� Zdd� Zdd� Zdd � Zd
S ) r6 z?An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _d S )N)r- r= r r r r r � s
zEntry.__init__c C sH g }| j D ]
}|�d|dg� q| jD ] }|�t|�dg� qd�|�S )Nz User-agent: rN r )r- �extendr= r rQ )r �ret�agentrA r r r rR � s
z
Entry.__str__c C sF |� d�d �� }| jD ]}|dkr dS |�� }||v r dS q dS )z2check if this entry applies to the specified agentrC r r, TF)r9 r; r- )r rK r^ r r r rI � s
�zEntry.applies_toc C s$ | j D ] }|�|�r|j S qdS )zZPreconditions:
- our agent applies to this entry
- filename is URL decodedT)r= rI rJ )r rY rA r r r rJ � s
�zEntry.allowanceN)rT rU rV rW r rR rI rJ r r r r r6 � s
r6 N)�
__future__r r r �future.builtinsr �future.backportsr �future.backports.urllibr �_parser �_request�__all__�objectr
r> r6 r r r r �<module> s