ž
¨ÿ fõ c @ sd d Z d d l Z d d l Z d g Z Gd d „ d ƒ Z Gd d „ d ƒ Z Gd d „ d ƒ Z d S( u< robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package:
1) GNU GPLv2
2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in
http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
i Nu RobotFileParserc B s‰ | Ee Z d Z d Z d d d „ Z d d „ Z d d „ Z d d
„ Z d d „ Z d
d „ Z d d „ Z
d d „ Z d d „ Z d S( u RobotFileParserus This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
u c C s> g | _ d | _ d | _ d | _ | j | ƒ d | _ d S( Ni F( u entriesu Noneu
default_entryu Falseu disallow_allu allow_allu set_urlu last_checked( u selfu url( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu __init__ s
u RobotFileParser.__init__c C s | j S( u· Returns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
( u last_checked( u self( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu mtime s u RobotFileParser.mtimec C s d d l } | j ƒ | _ d S( uY Sets the time the robots.txt file was last fetched to the
current time.
i N( u timeu last_checked( u selfu time( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu modified( s u RobotFileParser.modifiedc C s5 | | _ t j j | ƒ d d … \ | _ | _ d S( u, Sets the URL referring to a robots.txt file.i i N( u urlu urllibu parseu urlparseu hostu path( u selfu url( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu set_url0 s u RobotFileParser.set_urlc C s© y t j j | j ƒ } Wna t j j k
r| } z; | j d k rO d | _ n | j d k rj d | _ n WYd d } ~ Xn) X| j
ƒ } | j | j d ƒ j
ƒ ƒ d S( u4 Reads the robots.txt URL and feeds it to the parser.i‘ i“ i Nu utf-8( i‘ i“ T( u urllibu requestu urlopenu urlu erroru HTTPErroru codeu Trueu disallow_allu allow_allu readu parseu decodeu
splitlines( u selfu fu erru raw( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu read5 s u RobotFileParser.readc C sA d | j k r- | j d k r= | | _ q= n | j j | ƒ d S( Nu *( u
useragentsu
default_entryu Noneu entriesu append( u selfu entry( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu
_add_entryB s u RobotFileParser._add_entryc C s d } t ƒ } xã| D]Û} | sn | d k r@ t ƒ } d } qn | d k rn | j | ƒ t ƒ } d } qn n | j d ƒ } | d k rœ | d | … } n | j ƒ } | s´ q n | j d d ƒ } t | ƒ d k r | d j ƒ j ƒ | d – s u+ RobotFileParser.__str__..( u joinu entries( u self( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu __str__• s u RobotFileParser.__str__N(
u __name__u
__module__u __qualname__u __doc__u __init__u mtimeu modifiedu set_urlu readu
_add_entryu parseu can_fetchu __str__( u
__locals__( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu RobotFileParser s
3 c B s> | Ee Z d Z d Z d d „ Z d d „ Z d d „ Z d S( u RuleLineuo A rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.c C s\ | d k r | r d } n t j j t j j | ƒ ƒ } t j j | ƒ | _ | | _ d S( Nu T( u Trueu urllibu parseu
urlunparseu urlparseu quoteu pathu allowance( u selfu pathu allowance( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu __init__œ s
u RuleLine.__init__c C s | j d k p | j | j ƒ S( Nu *( u pathu
startswith( u selfu filename( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu
applies_to¤ s u RuleLine.applies_toc C s | j r d p d d | j S( Nu Allowu Disallowu : ( u allowanceu path( u self( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu __str__§ s u RuleLine.__str__N( u __name__u
__module__u __qualname__u __doc__u __init__u
applies_tou __str__( u
__locals__( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu RuleLine™ s u RuleLinec B sJ | Ee Z d Z d Z d d „ Z d d „ Z d d „ Z d d „ Z d
S( u Entryu? An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _ d S( N( u
useragentsu rulelines( u self( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu __init__ s u Entry.__init__c C sj g } x' | j D] } | j d | d g ƒ q Wx* | j D] } | j t | ƒ d g ƒ q: Wd j | ƒ S( Nu User-agent: u
u ( u
useragentsu extendu rulelinesu stru join( u selfu retu agentu line( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu __str__± s u
Entry.__str__c C s] | j d ƒ d j ƒ } x= | j D]2 } | d k r9 d S| j ƒ } | | k r# d Sq# Wd S( u2 check if this entry applies to the specified agentu /i u *TF( u splitu loweru
useragentsu Trueu False( u selfu useragentu agent( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu
applies_to¹ s u Entry.applies_toc C s. x' | j D] } | j | ƒ r
| j Sq
Wd S( uZ Preconditions:
- our agent applies to this entry
- filename is URL decodedT( u rulelinesu
applies_tou allowanceu True( u selfu filenameu line( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu allowanceÆ s u Entry.allowanceN( u __name__u
__module__u __qualname__u __doc__u __init__u __str__u
applies_tou allowance( u
__locals__( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu Entry« s
u Entry( u __doc__u urllib.parseu urllibu urllib.requestu __all__u RobotFileParseru RuleLineu Entry( ( ( u7 /opt/alt/python33/lib64/python3.3/urllib/robotparser.pyu s
ˆ