Spot the Bug SOLUTION: ​WWW::Robot​Rules

I recently discovered a rather serious bug in the Perl module WWW::RobotRules (Source)
HINT: It's in this block of code:

}
elsif (/^\s*User-Agent\s*:\s*(.*)/i) {
    $ua = $1;
    $ua =~ s/\s+$//;
    if ($is_me) {
	# This record already had a User-agent that
	# we matched, so just continue.
    }
    elsif ($ua eq '*') {
	$is_anon = 1;
    }
    elsif($self->is_me($ua)) {
	$is_me = 1;
    }
}

Here is a robots.txt file that exposes the bug

Here's how robots.txt files are supposed to work

Original post

I don't read hideously ugly Perl, do you want to do a break down for those of us without brain-damage?

Manip wrote:

I don't read hideously ugly Perl, do you want to do a break down for those of us without brain-damage?

Sure

This code gets run for each line of the robots.txt file in order

# elsif is the same as c's "else if" or VB's ElseIf
elsif (
    # The current line of the robots.txt file looks like
    # User-Agent: something-or-other
    /^\s*User-Agent\s*:\s*(.*)/i
) {

    # Store "something-or-other" in the $ua variable
    $ua = $1;

    # take any trailing whitespace off the end of $ua
    $ua =~ s/\s+$//;

    # $is_me is a boolean variable
    if ($is_me) {
        # This record already had a User-agent that
        # we matched, so just continue.
    }

    # check if $ua is one character long
    # and that character is a literal asterisk
    elsif ($ua eq '*') {

        # set the $is_anon boolean variable
        $is_anon = 1;
    }

    # $self is like VB's "Me" or C#'s "this"
    # is_me(string) checks whether the
    # user-agent in the robots.txt file matches
    # this spider
    elsif($self->is_me($ua)) {

        # set the $is_me boolean variable
        $is_me = 1;
    }
}

I think I found the problem .... This code is written in Perl.... To fix it I suggest writing it in something less crappy like C or ASM or pure binary....

I've long felt that Perl is a write-only language.
Edit: (But admittedly this snippet isn't too bad.)

I appear to be an idiot.  This may not be a bug at all, but a counter-intuitive reading of a malformed robots.txt file.

From the spec:
Lines containing only a comment are discarded completely, and therefore do not indicate a record boundary.

I don't really know that much about perl, but since the example file uses "User-agent" without the capital A, and the regular expression checks for capital A, you  wouldn't be getting any matches...

what's even worse you could get dissalowed path's for user-agent string you  wouldn't want... how ? consider:

User-Agent: *
Disallow: a

#next line doesn't match a new user-agent
User-agent: Mozilla
Disallow: /extra/private/secret/path

That would dissalow the secret path to all agent strings...

Just one question, are robots.txt files meant to stop robots, or is it meant to stop people with browsers as well?



Oooh Perl... *shudders*,
Loadsgood.

RicardoClerigo wrote:

the example file uses "User-agent" without the capital A, and the regular expression checks for capital A, you  wouldn't be getting any matches...

The "i" in /.../i means case "i"nsensitive.  The error is a little more subtle than that.

RicardoClerigo wrote:

what's even worse you could get dissalowed path's for user-agent string you  wouldn't want... how ? consider:

User-Agent: *
Disallow: a

#next line doesn't match a new user-agent
User-agent: Mozilla
Disallow: /extra/private/secret/path

That would dissalow the secret path to all agent strings...

This is actually really close.

Loadsgood wrote:

Just one question, are robots.txt files meant to stop robots, or is it meant to stop people with browsers as well?

It's meant to stop well-behaved robots.  Browsers don't bother to check the robots.txt file.  Well-behaved robots check the robots.txt file and don't go where it says not to go.  Badly-behaved robots don't check the robots.txt file.  Really badly-behaved robots check the robots.txt file and concentrate their efforts on the "don't go here" parts of the site. :O

Maurits wrote:

It's meant to stop well-behaved robots.  Browsers don't bother to check the robots.txt file.  Well-behaved robots check the robots.txt file and don't go where it says not to go.  Badly-behaved robots don't check the robots.txt file.  Really badly-behaved robots check the robots.txt file and concentrate their efforts on the "don't go here" parts of the site.

Or the robot that says:

"Checking Robots.txt file...
Finished checking robots.txt file.
Too restrictive ignoring..."



Funny robot,
Loadsgood.

Solution in the form of a patch

Gisle Aas, the owner of libwww-perl, will include the fix in the next released version.

Maurits wrote:

Solution in the form of a patch

Gisle Aas, the owner of libwww-perl, will include the fix in the next released version.

And... it's live!