Subject: [Tools-discuss] agenda.ics files

From: Michael Richardson <[email-masked]>
Date: Tue, 29 Oct 2013 09:08:16 -0400

I select a bunch of WG, and I copy-link from the "download as an .ics file"

https://datatracker.ietf.org/meeting/88/agenda.ics?6man,dice,....

I paste that into Google Calendar, and a minute or two later, I get:

"Could not fetch the URL because robots.txt prevents us from crawling the
URL"

http://www.sandelman.ca/tmp/mcrcapture/1112.2013-10-29/capture1.png

Robots.txt says:
User-agent: *
Disallow: /

Perhaps google calendar has changed.  May I ask that we have an allow on
the /meeting/*/agenda.ics, or /meeting/88/agenda.ics if we can't use wildcards.
I was able to download my .ics file and upload it again, but that misses
changes to the agenda.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Julian Reschke <[email-masked]>
Date: Tue, 29 Oct 2013 14:24:10 +0100

> "Could not fetch the URL because robots.txt prevents us from crawling the

But then why does Google Calendar want to "crawl" in the first place? It 
just needs to fetch the content from that URL...

 > ...

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 29 Oct 2013 15:11:57 +0100

Hi Michael,

For some reason, it seems like the http server points at the wrong robots.txt
file.  The robots.txt at the top of the static-files directory says the right
thing:

  User-agent: *

I'll try to sort this out.

But *please*, don't use this discussion list as a ticket system.  If you have
a bug report, please report as following:

Please report datatracker.ietf.org bugs at http://tools.ietf.org/tools/ietfdb
Please reports tools.ietf.org bugs by email to [email-masked] or
at http://tools.ietf.org/tools/issues

This information is also in the list footer of this list.

	Henrik

---

From: Michael Richardson <[email-masked]>
Date: Tue, 29 Oct 2013 09:17:08 -0400

http://stackoverflow.com/questions/4687617/robots-txt-and-google-calendar

suggests:

User-agent: Googlebot
Allow: /*.ics$
Disallow: /

User-agent: *
Disallow: /

(Although, I don't know why not:

User-agent: *
Allow: /*.ics$
Disallow: /

)

--
Michael Richardson <[email-masked]>, Sandelman Software Works

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 29 Oct 2013 15:36:10 +0100

>just needs to fetch the content from that URL...

If Google fetches it repeatedly to capture changes over time then it
certainly should respect the robot exclusion rules.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Michael Richardson <[email-masked]>
Date: Tue, 29 Oct 2013 11:23:47 -0400

Julian Reschke <[email-masked]> wrote:
    >> "Could not fetch the URL because robots.txt prevents us from crawling
    >> the URL"

    > But then why does Google Calendar want to "crawl" in the first place?
    > It just needs to fetch the content from that URL...

My opinion is that they are using the Google crawling engine to fetch
ics files.  The idea would be, I guess, to grab the data once, no matter how
many want it, and also not to continuously hit the web site with the info.

It seems courteous to me, but it's a bit surprising.

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 31 Oct 2013 09:57:54 +0000

On Oct 29, 2013, at 22:11, Henrik Levkowetz <[email-masked]> wrote:
> The robots.txt at the top of the static-files directory says the right
> thing:

robots.txt must be at the top-level of the site, i.e., ietf.org/robots.txt

I don't think per-directory robots.txt files work.

Lars

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 31 Oct 2013 12:54:03 +0100

Hi Lars,

> robots.txt must be at the top-level of the site, i.e., ietf.org/robots.txt

Sorry, I was unclear.  The robots.txt at the top of the static-files directory
in the repository is deployed at the top level of the site.  The trouble in
this case was that the apache config aliased /robots.txt to /static/dev/robots.txt,
which should only be used when running a development site.

I fixed the apache config of the production site to show the correct robots.txt
yesterday, so things should work as intended now.

Best regards,

	Henrik

---

From: Michael Richardson <[email-masked]>
Date: Thu, 31 Oct 2013 10:28:31 -0400

Henrik Levkowetz <[email-masked]> wrote:
    >>> The robots.txt at the top of the static-files directory says the
    >>> right thing:
    >>

    > Sorry, I was unclear.  The robots.txt at the top of the static-files
    > directory in the repository is deployed at the top level of the site.
    > The trouble in this case was that the apache config aliased /robots.txt
    > to /static/dev/robots.txt, which should only be used when running a
    > development site.

    > I fixed the apache config of the production site to show the correct
    > robots.txt yesterday, so things should work as intended now.

And, indeed it does.
(I will use the ticket system in the future)