Subject: Re: #1416 Reinjection - an attempted summary, and a suggested resolution

From: Harald Alvestrand <[email-masked]>
Date: Thu, 01 Mar 2007 10:16:11 +0100

>
> So let us examine the two competing strategies.
>
> I think we are agreed that
>
> . Injecting agents MUST insert Date and Message-ID when absent.
> . Injecting agents MUST insert Injection-Date when Injection-Date is
> absent AND (either Date OR Message-ID is absent).
>   
We have agreement on these 2 points.
> . Injecting agents MUST NOT insert Injection-Date if it is already
> present.
>   
We do not have agreement on this point.
> We disagree on the following two possibilities:
>
> C. Injecting agents MUST insert Injection-Date when it is absent.
>
> R. Injecting agents MUST NOT insert Injection-Date if both (Date AND
>    Message-ID are already present).
>   
I haven't seen anyone arguing for R. I thought that one was settled in 
USEFOR.
> 3.2.7.  Injection-Date
>
>    The Injection-Date header field contains the date and time that the
>    article was injected into the network.  Its purpose is to enable news
>    servers, when checking for "stale" articles, to use a <date-time>
>    that was added by a news server at injection time rather than one
>    added by the user agent at message composition time.
>
>    This header field MUST be inserted whenever an article is injected.
>    However, software that predates this standard does not use this
>    header, and therefore agents MUST accept articles without the
>    Injection-Date header field.
>
>    injection-date  =  "Injection-Date:" SP date-time CRLF
>
>
>    See the remarks under Section 3.1.1 regarding the syntax of
>    <date-time> and the requirements and recommendations to which it is
>    subject.
>
>       NOTE: Since clocks on various agents are not necessarily
>       synchronized, the <date-time> in this header field might not be a
>       later value than that in the Date header field.  Agents MUST NOT
>       alter a pre-existing Date header field when adding an Injection-
>       Date header field.
>
>    This header field is intended to replace the currently-used but
>    undocumented "NNTP-Posting-Date" header field, whose use is now
>    deprecated.
I have not scanned your later argumentation, since I do not agree that 
this question is open.

                 Harald

---

From: Russ Allbery <[email-masked]>
Date: Thu, 01 Mar 2007 05:57:38 -0800

> I haven't seen anyone arguing for R. I thought that one was settled in

I'm arguing for R so that we don't break the guarantees the protocol
currently makes while introducing a new feature.

USEFOR doesn't really settle anything about Injection-Date, I'm afraid.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 1 Mar 2007 18:09:44 GMT

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:

>> OTOH, one would hope that it will be those dedicated relayers who will
>> be the early adopters of Injection-Date.

>Given that development of NNTPRelay has essentially stopped, the Highwinds
>support forums appear to be down, and Highwinds hasn't issued a press
>release on their site for nearly four years, I wouldn't hold my breath.
>Highwinds did just release new builds last November for the first time in
>two years, though, so they don't appear to have disappeared entirely.

Aren't they going to take some action to get into line with RFC 3977?

And for sure there will be other things they ought to be doing to become
compliant with USEFOR/USEPRO. OTOH, if there are such a small number of
such implementations, it is practical for them to be leant on (which is
less true of news servers in general, and virtually impossible for user
agents).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 1 Mar 2007 19:16:12 GMT

>We do not have agreement on this point.

I think Russ and I are agreed on that. It comes with allowing posting
agents to write it, which has benefits in some situations (e.g.
multi-injection, and "posting agents" which are in reality
proxies/servers/other-special cases).

>I haven't seen anyone arguing for R. I thought that one was settled in 

It is my understanding that R is (near enough) what Russ is currently
proposing (so no guessing ad to what 'C' stands for).

>I have not scanned your later argumentation, since I do not agree that 

This (or rather these) questions are precisely what this issue/thread is
about (though the actual nature of the solutions under consideration has
changed somewhat since it started and hence there is less focus on
reinjection and more on other scenarios that might follow).

My "later argumentation" shought to analyse the consequences of following
'C' or 'R' - in some sense it has to be a compromise because you can
always find a situation in which each of them does not do what you might
ideally like.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 1 Mar 2007 18:05:22 GMT

In <FC8A5FB7F478A0F84B09D3E0@[172.28.60.203]> Harald Tveit Alvestrand <[email-masked]> writes:

>--On 24. februar 2007 11:33 -0800 Russ Allbery <[email-masked]> wrote:

>>
>> Frank Ellermann <[email-masked]> writes:
>>
>>> I don't believe in this "identity = ID + timestamp" theory.  The ID is
>>> the identity, the timestamp is a kludge because histories are limited,
>>> and because accepting stale articles isn't in the interest of the
>>> poster.
>>
>> I think that's a wording distinction without any semantic difference.
>> Kludge or identity, history files are limited and therefore the timestamp
>> is a component of the effective identity of the article.

>I still don't believe that this "effective identity" concept is a Good 
>Thing.

As Russ has said, it is not intended as a concept to be mentioned in the
document. It is not even a particularly good terminology. But or the
limited puspose of the current thread it is good enough, so not much
point in changing it.

>- Since some servers will use Date: anyway, injecting-agents should police 
>injection based on Date:

That is the only bit I do not agree with. For a start, they should
certainly NOT be doing any policing if Injection-Date is already present
(though they might police on the content of that Injection-Date).

But, more to the point, injecting agents do not have information as to
what the rest of the net is doing (though they might usefully take note of
the policies of their local relayers and servers). Policing is really a
matter of local policy for each agent. We seem to be agreed that at least
7 or 8 days stale is quite acceptable for injecting, but even that is
longer than the retention of some relayers (OTOH, such relayers will be
flooded around). So we can't lay down hard and fast rules (though we might
provide _advice_ - probably in USEAGE).

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 1 Mar 2007 19:07:05 GMT

In <[email-masked]> Frank Ellermann <[email-masked]> writes:

>Russ Allbery wrote:

>>    3.  It MUST examine the Injection-Date header field or, if absent,
>>        the Date header field, and reject the article if that date
>>        predates the earliest articles of which it keeps record or if
>>        that date is more than 24 hours into the future.  It MAY reject
>>        articles with dates in the future with a smaller margin than 24
>>        hours.

>I'd split that - future timestamps can get their own (small) paragraph:

>3b: It MAY reject articles with dates more than 24 hours in the future.

>Normally magic numbers like "24 hours" need a justification, but here
>it should be obvious that this is about a plausible timezone confusion.
>(If that's not the real reason for "24 hours" better say what it is.)

Essentially, it would be stupid to take action on articles only a few
minutes into the future, because of clock differences, timezone
confusions, etc (my wife's mobile still thinks we are in Britich Summer
Time). Anything over two hours would be quite reasonable.

OTOH, if we allowed things infinitely far into the future, various
scammers would have a field day. But they would need to be days into the
future to have much effect.

So we have to choose sume number between 2 hours and a few days, and for
that "24 hours" seems about right.

>The "articles of which it keeps records" isn't clear.  It's not about
>the oldest available article (that could be months or more), it's the
>oldest unavailable article (only recorded in the history).  Maybe it's
>better if you define and use a term like "retention time" or "cut off
>time" as you've done it elsewhere.  I'm not sure what the difference
>is, "cut off => reject as too old, retention => could be removed from
>the history" (?)

>What I have in mind is a server keeping articles in some groups for
>long periods, e.g. for "answers" groups, and articles in other groups
>only shortly, e.g. in binary groups.

Yes, "of which it keeps record" was meant to imply that the cutoff applied
to articles of that type (e.g. in that group). Essentially, the time for
which records off that article would have been kept if it had arrived on
time.

But if you want the wording clarified (and Russ seems happy to do so),
then that is fine by me. But do note that USEPRO has never included
significant mention of "expiry" at all (though USEAGE does), except in
connection with the Expires header. And we don't want to write USEPRO in
terms of implementation details except where that cannot be avoided.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 01 Mar 2007 21:21:53 -0800

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> Given that development of NNTPRelay has essentially stopped, the
>> Highwinds support forums appear to be down, and Highwinds hasn't issued
>> a press release on their site for nearly four years, I wouldn't hold my
>> breath.  Highwinds did just release new builds last November for the
>> first time in two years, though, so they don't appear to have
>> disappeared entirely.

> Aren't they going to take some action to get into line with RFC 3977?

I doubt it, honestly.  I'm prepared to be favorably surprised, but I think
that Usenet's doing a significant shrinking at the moment and there isn't
a lot of momentum behind such things right now.  My guess is that Usenet
is going to revert to being far more of a hobbiest net than it was, which
has both good and bad aspects, and that momentum on software changes will
only pick up again (if it does) after that's happened.

> less true of news servers in general, and virtually impossible for user

One of the reasons why I want the various Usenet standards to be as
backward compatible as possible is that I'd like to work on
fully-conforming software (insofar as I can find time to do so), and I
don't want to be put into a situation where I have to choose between
following the standard and working well with software that's not been
updated.  Because, unfortunately, I think it's likely that most software
will not be updated.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 03 Mar 2007 15:41:58 +0100

>> I haven't seen anyone arguing for R. I thought that one was settled
>> in USEFOR.

> proposing (so no guessing ad to what 'C' stands for).

I don't get it.  Injection agents are supposed to add a Message-ID when
it's missing.  They can also add a missing Injection-Date.  For strange
"multiple injection" issues with a significant delay between the first
and the last injection this isn't ideal.  But for the far more common
case of messages posted some time after they were composed it's fine.

I'd be more worried about "multiple injections" without a Message-ID,
i.e. something resulting in a "multi-post" after the injection-agents
added their own (different) Message-IDs.

Maybe we could tackle this "multiple injection" problem with a "SHOULD
give up on unavailable servers after some hours of the first successful
injection, and ask for user intervention" (because the user might know
that the unavailable server is really not connected to any parts of the
net where the injections already worked).

The implementors of "multiple injection proxies" need to know that
there's a problem that can't be solved by the injection agents without
dropping the ball for a far more common problem of ordinary 2822 UAs.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 03 Mar 2007 16:05:01 +0100

Russ Allbery wrote:
 
>> Maybe it's better if you define and use a term like "retention time"
>> or "cut off time" as you've done it elsewhere.  I'm not sure what the
>> difference is, "cut off => reject as too old, retention => could be
>> removed from the history" (?)
 
> Right.
 
> I think we're converging on a desire to rephrase things that way and
> introduce those terms.

Fine.  Based on that the definition could note that "retention" isn't
directly related to "expiration", without going into details.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sat, 03 Mar 2007 11:52:30 -0800

Frank Ellermann <[email-masked]> writes:

> it's missing.  They can also add a missing Injection-Date.

I don't agree that they should do this if Date is already present.

> For strange "multiple injection" issues with a significant delay between
> the first and the last injection this isn't ideal.

And, indeed, potentially breaks things.

> But for the far more common case of messages posted some time after they
> were composed it's fine.

It's not acceptable to me to break this fundamental of an existing
protocol guarantee in order to introduce a new feature, without which
Usenet has survived for many years.

> added their own (different) Message-IDs.

We've already discussed this and this already won't be allowed.

> net where the injections already worked).

If we're going to require that user agents change, it would be a far
easier change to just require that they add Injection-Date than add this
more complex logic.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 04 Mar 2007 17:04:28 +0100

Russ Allbery wrote:

  ["missing" Injection-Date added by injection-agent]
>> for the far more common case of messages posted some time after they
>> were composed it's fine.

> Usenet has survived for many years.

It's also not acceptable to redefine the 2822-semantics (and now also
USEFOR-semantics) of Date, something has to give, and we've to figure
out what causes the least damage.

Obviously I don't get what you consider as "fundamental of an existing
protocol guarantee".  It would help to specify what this "guarantee"
is (in the draft), and why it's "fundamental" for the protocol.

One fundamental guarantee I'm aware of is "never ever screw with the
Message-ID", and apparently you think it's okay to make unspecified
exceptions for moderators.

At least I should understand what precisely breaks wrt Injection-Date
when I try hard, so far in vain.  Here are the pieces of this puzzle
as far as I understood it:

- The whole issue is related to "dupes" (articles, i.e. Message-IDs,
  accepted more than once by a server) and "nopes" (articles which
  should arrive but don't make it).

- It's also related to "multiple injections" with a significant delay
  between the first and the last successful injection.  "Significant"
  is a weasel word.  You claim that something fundamental breaks, and
  I need an example what that is, so let's pick a random number like
  "12 hours" for "significant".  Please tune that as needed.

- If a "multiple injection proxy" uses its own unique Injection-Date
  (of say the first successful injection) there should be no problem,
  or rather not the discussed problem, therefore we can ignore this
  case.

- If the "MIP" can inject the article without Injection-Date within
  "12 hours" (or whatever maximal delay you pick as "significant")
  the claimed breakage of different Injection-Dates added by the
  injection-agents probably won't convince me, therefore we can also
  skip this case.

- Finally the "MIP" manages to inject the article without its own
  unique Injection-Date, the injecting-agents add different Injection-
  Dates, and the delay between the first and the last injection is
  more than "12 hours" (or what you pick to show that this is really
  harmful).

- For some serious trouble we have to assume that the first and the
  last injection affect a connected part of the net.  We can ignore
  all other successful injections between first and last, the biggest
  damage should show up with the worst case "12 hours".

- Temporarily we can reduce the problem to three points, the "MIP",
  the server A of the first injection, and the server B of the last
  injection.  For this simplification to work we've to asume that
  A and B are peers.

- AFAIK we can also assume that A relays the article more or less
  immediately to B.  Let's say A's first Injection-Date is T+0, and
  A notes the Message-ID with arrival T+0 in its history.  B does
  the same with T+1.

- "12 hours" (or what you pick) later the "MIP" tries to inject the
  same Message-ID at B.  B has it still in its history and will say
  "no thanks".  No problem.

- Variation:  A was somehow unable to relay the article to B within
  the "12 hours".  Then B will accept it from the "MIP", note its
  arrival time as T+12h in its history, and add a corresponding
  Injection-Date.  After that sooner or later A and B will connect.

- If that's within the retention time of A, say T+7d, then A won't
  accept the Message-ID from B.  Likewise B won't accept the same
  Message-ID from A again within B's retention time, say T+7.5d.

- Because this was a simplification we're actually talking about
  servers A' (where the article arrived via A) and B' (where the
  article arrived via B), and the transports from A to A' and from
  B to B' resp. can take some time.

Now it's your turn (anybody here understanding the problem), what
damage can happen if the A to A' path uses Injection-Date T+0 set
by A, while the B to B' path use T+12h set by B ?

Replacing "12h" and "12 hours" everywhere by any value picked by
Russ to get an instructive example, and using plausible retention
and cut off times at A' and B' (not necessarily identical).

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sun, 04 Mar 2007 11:04:50 -0800

> out what causes the least damage.

Clearly it is acceptable to do that since Usenet has done that since at
least the publication of RFC 1036, which redefines Date to something
different than RFC 2822 and synonymous with our Injection-Date concept.
All the Netnews software written since has been based on that assumption.
I think that answers the question of which is more acceptable.

> protocol guarantee".  It would help to specify what this "guarantee" is
> (in the draft), and why it's "fundamental" for the protocol.

The guarantee is that, if you store a record of message IDs you have seen
for at least N days from the Date in the article and you reject any
article with a date older than N days, you can use the contents of the
history file as an authoritative record for determining whether you can
safely accept an article without creating a duplicate.  (If you base
history retention on the length of time from when you saw the article
rather than on the contents of the Date header, you have to add to your
retention time whatever leeway for future Dates you're allowing to be
fully correct.)

To maintain this guarantee requires the property that if you post a
message and provide the Date and Message-ID, the injecting agent will not
change those header fields and thereby potentially introduce a copy of the
article with a different Date header.

That this is not correctly reflected in the draft is one of the
motivations for this long thread that we're currently having.

>   "12 hours" for "significant".  Please tune that as needed.

The problem exists regardless of the delay.  It is more noticable the
longer the delay is.  A problem that grows continuously with a variable is
not a "weasel word."

>   A notes the Message-ID with arrival T+0 in its history.  B does

The problem occurs when a specific article arrives at A with one Date
header, expires from A, and then arrives at A again with a different,
later Date header between the time the history entry is dropped and A's
cutoff value.  This can happen with any variation in Date headers if the
article is delayed exactly the right length of time in getting to A, but
obviously it's more likely if the difference in Date headers is on the
order of days rather than hours.  It's therefore far more likely to happen
with serial multiple injection than with parallel multiple injection, but
it can still happen with parallel multiple injection if an article gets
stuck in a queue due to a server being down for an extended period and
then shows up later.

The fundamental assumption made by the uniqueness algorithm in a news
server is that if its retention time is at least as long as its cutoff
time, any article it receives will either authoritatively be in the
history file or not or will be discarded by its cutoff time.  Allowing
injecting agents to change the Date header breaks this assumption for
existing implementations and means that the history file can no longer be
treated as authoritative.  This breaks the flood-fill algorithm at the
heart of the Netnews protocol.

Such delays happen routinely.  Servers go down and hold on to articles in
a queue while they're down until an administrator notices and kicks them.
Personal news servers behind suck/rpost feeds pull something down from one
source, hang on to it for several days, and then reinject it at some other
point.  People misconfigure their access permissions and their peers queue
until they fix those permissions and then send them all the old articles
that had been queuing.

You cannot assume that A relays the article more or less immediately to B,
as this is simply the normal case and is in no way guaranteed by the
protocol or by the way that Netnews works in practice.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 5 Mar 2007 20:11:56 GMT

>different than RFC 2822 and synonymous with our Injection-Date concept.

RFC 1036 is, as usual, vague in its wording. It speaks of when the article
was "originally posted to the network"; it does not make the clear
distinction between "posting" and "injection" that we are now used to.

RFC 822 was silent on the matter. Whether RFC 2822 was merely codifying an
existing assumption or introducing a deliberate change I do not know.

>All the Netnews software written since has been based on that assumption.

But that is clearly untrue, since it seems that 70% (admittedly taken from
a smallish sample) of posting agents are already following the RFC 2822
usage. Moreover, our intention when we invented Injection-Date was to
adopt that RFC 2822 usage (and USEFOR reflects that), as well as to
improve matters when injection was delayed after composition.

>safely accept an article without creating a duplicate.

But even if that guarantee occasionally fails, the Sky does mot Fall In,
and in the case we are looking at I cannot see how looping could ensue.
The effect will only be seen by clients of the server concerned, who may
be surprised to see articles (with a different Xref number) that they have
already seen before. The question to be addressed, therefore, is the
disadvantage of the occasional failure of this guarantee against the
advantage of improved propagation of articles injected late with respect
to their composition times.

I tried to analyse these pros and cons in the message in which I
introduced the terms 'C' and 'R' for the two possibilities. It would be
better for people to comment on those detailed arguments rather than for
me to repeat them here.

>longer the delay is.  A problem that grows continuously with a variable is

I do not see how that could be so. If the difference between the injection
times is short (an hour maybe), then no effect will be observed unless
there are severe delays (of the order of 3 days) in propagating one of the
injected copies to some site, and then only if it arrives there within one
hour of that site's cutoff time for its earlier copy of the article.

>cutoff value.

No, the problem only arises if

1. The server in question has been upgraded to use Injection-Date (not Date)
2. The multi-injecting posting agent has not been so upgraded
3. At least one of the injecting agents has been so upgraded

You need to multiply together the probabilites of those events, together
with the probability that the window between the earliest and latest
injections, together with that the delays to that server,
was wide enough to encompass its cutoff point.

Yes, it will happen, buy only occasionally, and less often as more systems
become upgraded.

>a queue while they're down until an administrator notices and kicks them.

We are not concerned with servers that hold articles in a queue (they are
by definition relayers and will soon get flooded around). We are concerned
with multi-injecting posting agents/proxies/whatever that maintain such
queues.

>point.

Then those servers are effectively relaying plus reinjecting; any articles
they receive with Injection-Date in them will be sent on with that
Injection-Date intact.

There may be a problem if they reinject to an agent that now adds
Injection-Date, which may be an argument that injecting agents should
refuse apparent reinjections, or at least omit inserting Injection-Date in
that case.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 5 Mar 2007 21:47:38 GMT

>> net where the injections already worked).

That would indeed be a wise precaution, even without USEPRO.

>easier change to just require that they add Injection-Date than add this

I am open as to whether normal posting agents should be encouraged to add
Injection-Date as routine.

But for sure, "normal" posting agents will be the last things to become
fully compliant with our new standard.

However, those who engage in multi-injection, proxies and the like tend
not to use "normal" posting agents (or they interpose other agents between
those posting agents and the actual network). People who do that sort of
thing are more clueful than the average poster, and therefore more likely
to adopt our new ideas.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 5 Mar 2007 21:41:11 GMT

>case of messages posted some time after they were composed it's fine.

Indeed. The real issue here is the relative frequency of problems
resulting from multi-injection and those resulting from late inmjection. I
tend to share Frank's opinion that the latter will predominate, certainly
so over the long term, although getting hards fact to support either view
will not be easy.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 05 Mar 2007 15:03:37 -0800

Charles Lindsey <[email-masked]> writes:

> We are not concerned with servers that hold articles in a queue (they
> are by definition relayers and will soon get flooded around). We are
> concerned with multi-injecting posting agents/proxies/whatever that
> maintain such queues.

Flooding around them and then having them release articles again later is
*exactly* what creates this problem, so of course we're concerned with
them.  Anything that creates a delay that causes an article to reappear
later can create duplicates unless every copy of the article has
consistent Date information, which will not happen if we grant a license
to injecting agents to change the effective Date information of articles,
either by changing Date or by adding Injection-Date to override it.

> Injection-Date,

Exactly.

> which may be an argument that injecting agents should refuse apparent
> reinjections, or at least omit inserting Injection-Date in that case.

The only way that they can detect this case is to notice that the article
has a Date header already.  That's precisely when I think they should not
be permitted to insert an Injection-Date.  So you have arrived at exactly
my argument, if you also observe that there's no way to tell the
difference between a reinjection and a first injection for existing
suck/rpost setups (completely apart from the separate question of how they
*should* work).

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 06 Mar 2007 07:23:23 +0100

It seems to me that Russ's argument has now come fully around to the 
idea that injecting agents should try to find a single date field that 
stays constant across all copies of the message, and only filter based 
on that.

But making Injection-Date: that field seems to me to be a positively 
weird usage of this field, which is, after all, documented as "a 
replacement for NNTP-Posting-Date". And in the case of multiple 
injections that travel different paths before hitting a news network, 
it's not *possible* to tell the holders of all copies what that 
injection-date should be.

Should the filtering be done on Date:, ignoring Injection-Date: completely?

                  Harald

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 06 Mar 2007 15:55:59 +0100

> existing assumption or introducing a deliberate change I do not know.
I can speak to that, since I was there.

DRUMS debated whether Date: should reflect composing time or injection 
time, agreeing that it was unspecified in 822, agreeing that it should 
be defined as one or the other, and landed with a strong (but not 
unanimous) consensus on composing time.

So 2822 Date: semantics is exactly that - 2822 semantics.

                        Harald

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Tue, 06 Mar 2007 10:03:38 -0500

> Should the filtering be done on Date:, ignoring Injection-Date: completely?

That is what is done now, by existing implementations.

As I understand it, Charles is willing to accept duplicates and inconsistent 
propagation during the transition period, when some servers are checking 
Injection-Date and others Date.

I think this is unacceptable and unnecessary.  If people want the benefits,
there must be a way to get them without degrading existing operation.

I think the benefits that people desire are
    - Better propagation of delayed injected articles
    - Display of the actual authoring date.

I propose:
    Agents MUST use the Date header field and not Injection-Date or the
    deprecated NNTP-Posting-Date to make article staleness checks.  In the case
    of delayed injection, Posting agents MAY provide copies of the proto-article
    with a Date:header field that indicates the time of injection (rather than
    conforming to RFC-2822 time of authoring) only if they also supply a
    Message-ID field.  Posting agents MAY offer proto-articles to multiple
    servers at different times only if both fields are present and unaltered in
    each proto-article posted.  All agents and gateways MUST preserve the
    contents of both header fields, refusing or not propagating an article rather
    than altering either field. User-Agents MAY supply and display an
    "Authoring-Date" header field.

---

From: Dan Schlitt <[email-masked]>
Date: Tue, 6 Mar 2007 12:26:03 -0500

That is the conclusion I was coming to after reading all the back and
forth discussion.

/dan

-- 

Dan Schlitt
[email-masked]

---

From: Russ Allbery <[email-masked]>
Date: Tue, 06 Mar 2007 09:57:37 -0800

>> RFC 1036 is, as usual, vague in its wording. It speaks of when the
>> article was "originally posted to the network"; it does not make the
>> clear distinction between "posting" and "injection" that we are now
>> used to.

What distinction is that?  I use the terms interchangeably.

I have a hard time interpreting this as meaning anything other than
injection time:

    The "Date" line (formerly "Posted") is the date that the message was
    originally posted to the network.

I don't see how, using regular English meanings of those words, one could
interpret that as composition time.  The phrase "to the network" seems to
me to remove all possible doubt.

>> RFC 822 was silent on the matter. Whether RFC 2822 was merely codifying
>> an existing assumption or introducing a deliberate change I do not
>> know.

> So 2822 Date: semantics is exactly that - 2822 semantics.

Which means that the clause in RFC 1036 that says RFC 822 overrides
doesn't come into play.  In this case, RFC 822 left something unspecified
and RFC 1036 chose to make it more specific because of a Usenet-specific
need.  In other words, for once, RFC 1036 played by the rules and RFC 2822
changed the rules out from under it.

Charles is correct that not all *client* software was written with this
understanding, and I exaggerated.  What I meant to say was that all
*server* software of which I'm aware was written assuming the semantics
specified in RFC 1036.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 06 Mar 2007 10:03:37 -0800

> What distinction is that?  I use the terms interchangeably.

Oh, wait, no, I'm sorry.  I see what you're saying.  The same distinction
as between a posting agent and an injection agent.  Although if you use
*exactly* the same meaning with headers, Injection-Date and Injection-Info
are actually Posting-Date and Posting-Info, since they record when and how
the posting agent gave the message to the injecting agent, not when the
injecting agent sent the message to a relaying or serving agent.

Anyway, all of those terms refer to handing the article to other agents
for relaying, not composition times.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 7 Mar 2007 11:51:11 GMT

>*server* software of which I'm aware was written assuming the semantics

Indeed. User agents are increasingly following the RFC 2822 line, and I
think that is both desirable and unstoppable. But one of the consequences
of your rule 'R' is that Injection-Date will now hardly ever be added by
injecting agents, which someewhat defeats the purpose of having it in the
first place.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 7 Mar 2007 11:18:40 GMT

>either by changing Date or by adding Injection-Date to override it.

Eh? I don't follow that argument.

Some server D in the middle of the network put articles into a queue and
does not release them until some days later.

In the meantime, other servers have already flooded the article around D,
so that when some server (say X) receives the article from D, it will
already have received it days earlier via routes that do not involve D.
Therefore, we are concerned only with what happens at X which will see the
article with Date and/or Injection-Date exactly as originally injected
(and noway altered by D).

The case you originally raised did not concern a server causing a delay,
but an ordinary user wih a suck feed who *reinjected* the article some
days later. That case might indeed cause a fresh Injection-Date to be
added.

>suck/rpost setups (completely apart from the separate question of how they

There are several ways in which reinjection can be detected, such as the
Presence of Injection-Info, NNTP-Posting-Host, NNTP-Posting-Date, X-Trace,
'POSTED' in the Path, or a suspiciously long Path.

Whether injecting agents then allow reinjection at all is a matter of
local policy, but if and when they do detect and allow it, then they
clearly MUST NOT insert a fresh Injection-Date themselves.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 7 Mar 2007 11:45:15 GMT

In <[email-masked]> "Forrest J. Cavalier III" <[email-masked]> writes:

>> replacement for NNTP-Posting-Date".

NNTP-Posting-Date was never more than a Trace header. Injection-Date was
intended to be far more than that - nothing 'weird' about what was
intended to be its primary purpose. Removing any need for
NNTP-Posting-Date was a convenient by-product.

> And in the case of multiple 

It needs to be understood that multiple injections are pretty rare, and
are only done by sites with particular problems/situations/needs and by
people who know what they are about (in my case, for example, it was
because I had a choice of two sites where one was unreliable and the other
was restrictive in what it would allow and might possibly munge in a way
that could break PGPVERIFY).

>Injection-Date and others Date.

Indeed. No Pain, No Gain!

And I am claiming the Gain is worth the Pain in this instance, and that is
the issue we should be concentrating on.

>there must be a way to get them without degrading existing operation.

But that turns out to be impossible, so we need to assess the various
compromises.

>    - Display of the actual authoring date.

Correct.

>    "Authoring-Date" header field.

But that either loses you the "Better propagation of delayed injected
articles", or it loses you the "Display of the actual authoring date" (at
least it loses it in Date header which is the only header recipients are
likely to see and understand, and which reading agents are likely to use
as a sorting key).

As I said, there is NO solution which is perfect from all POVs.

The worst that can happen under algorithm 'C' is that readers will
occasionally see duplicates of articles they have already read before. But
no articles will get lost which IMHO would be a worse evil.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 07 Mar 2007 13:14:17 -0800

>> Flooding around them and then having them release articles again later
>> is *exactly* what creates this problem, so of course we're concerned
>> with them.  Anything that creates a delay that causes an article to
>> reappear later can create duplicates unless every copy of the article
>> has consistent Date information, which will not happen if we grant a
>> license to injecting agents to change the effective Date information of
>> articles, either by changing Date or by adding Injection-Date to
>> override it.

> In the meantime, other servers have already flooded the article around
> D, so that when some server (say X) receives the article from D, it will

But the one queued at D may have a different, newer Date header because it
got the article from the other injection path, and therefore it looks like
a new article since the message ID had been dropped from history but the
Date header is newer than cutoff.

> days later. That case might indeed cause a fresh Injection-Date to be

My point is that the two phenomena reinforce each other, which is why the
argument in your previous message was not complete.  If the reinjected
article was only reinjected a day after the first article, it won't cause
problems in practice *unless* some server D grabs the second article and
sits on it for another six days or so and then kicks it out there, at
which point it falls into the window between cutoff and retention at more
servers.  But such servers like D are a routine occurance.

>> The only way that they can detect this case is to notice that the
>> article has a Date header already.  That's precisely when I think they
>> should not be permitted to insert an Injection-Date.  So you have
>> arrived at exactly my argument, if you also observe that there's no way
>> to tell the difference between a reinjection and a first injection for
>> existing suck/rpost setups (completely apart from the separate question
>> of how they *should* work).

> Presence of Injection-Info, NNTP-Posting-Host, NNTP-Posting-Date,
> X-Trace, 'POSTED' in the Path, or a suspiciously long Path.

None of those methods of reinjection detection work with existing
suck/rpost feeds precisely because servers currently try to reject
reinjected articles and therefore rpost removes all of those trace headers
to ensure that its messages don't look like reinjection.

Remember, I'm talking about how this protocol change interacts with
*existing software*, which was not written to our draft.

> clearly MUST NOT insert a fresh Injection-Date themselves.

I've already argued at length, back when I finished my draft, for why I
don't believe that putting the onus on injecting agents to detect and
negotiate permission to do reinjection is workable in practice.  Hopefully
I won't have to repeat all those arguments.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 07 Mar 2007 13:19:31 -0800

Charles Lindsey <[email-masked]> writes:

> think that is both desirable and unstoppable. But one of the
> consequences of your rule 'R' is that Injection-Date will now hardly
> ever be added by injecting agents, which someewhat defeats the purpose
> of having it in the first place.

Yes, what I'm arguing for is that it be added by posting agents in the
normal case and only by injecting agents when it's clearly safe to do so.

I'm returning again to what I consider to be a fundamental rule of
protocol design around existing protocols: put the burden of new features
on the agent that benefits the most from the feature.  Right now, we're
putting the burden of the Injection-Date feature on injecting agents.
Injecting agents don't care at all, don't benefit from the change, and
aren't in a position to know the intent.

This change specifically benefits people who compose off-line and post
later.  Right now, their messages may be rejected or suffer spotty
propagation because of the current Date semantics.  They're feeling the
pain, and the Injection-Date feature is designed for them.  So put the
burden of implementing the feature on them.  Off-line readers that queue
posts for later would be early adopters of the Injection-Date feature,
adding it to their messages if they were concerned that the Date header
may otherwise be stale.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 8 Mar 2007 14:43:41 GMT

>adding it to their messages if they were concerned that the Date header

Yes, they will have some reason to be early adopters. But we also need
relayers to adopt in order to get the benefit, and they are unlikely to do
so unless they see some significant number of articles actually arriving
with this header. And that will not happen unless injecting agents are
routinely adding it.

Would there be some future in saying that injecting agents should add it,
but not if the current time was suspiciously later than the Date (for some
value of "suspiciously" more than 1 day, bnut not much more)?

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 8 Mar 2007 14:35:34 GMT

>Date header is newer than cutoff.

ITYM "newer Injection-Date header" - let us try to be clear which headers
we are talking about, because altered Date headers, if they occur, would
be equally disruptive on the present network.

But in the case you mention, the problem arose because of the delayed
multi-injection. The delay at D (or any other delay due to a slow path
through the network) merely exaccerbated it. But even after that,
propagation of the duplicate article is still going to be patchy. I am not
claiming that no server will ever see it, but that the number off
occasions on which this causes significant upset will be small enough to
tolerate when compared with the better propagation of delayed
single-injections. There is NO 100% solution that will solve ALL of the
possible situations.

>to ensure that its messages don't look like reinjection.

But agents that break the standards (whether new or old) are going to
cause breakages in any system, including the current one (all they have to
do at present is to remove the Date header and all the evils you have been
discussing will promptly ensue).

---

From: Russ Allbery <[email-masked]>
Date: Thu, 08 Mar 2007 11:04:10 -0800

Charles Lindsey <[email-masked]> writes:

> relayers to adopt in order to get the benefit, and they are unlikely to
> do so unless they see some significant number of articles actually
> arriving with this header. And that will not happen unless injecting
> agents are routinely adding it.

Why would a significant number of articles arriving with this header cause
a relayer to change?  I don't see how the situation is any different for
relayers either way.  The only time there will be any pressure on relayers
to adopt this is if they're discarding articles because of not having this
support, and that again limits us down to only those articles posted by
off-line readers.

All the Injection-Date headers that could be added by injecting agents
that are close to existing Date headers are pure noise as far as adoption
goes, since ignoring them and using Date will have the same effective
results in 99.9% of all cases.

> Would there be some future in saying that injecting agents should add
> it, but not if the current time was suspiciously later than the Date
> (for some value of "suspiciously" more than 1 day, bnut not much more)?

Isn't that exactly the case where adding it causes the most damage if
adding it is wrong?

---

From: Russ Allbery <[email-masked]>
Date: Thu, 08 Mar 2007 11:07:25 -0800

Charles Lindsey <[email-masked]> writes:

> propagation of the duplicate article is still going to be patchy. I am
> not claiming that no server will ever see it, but that the number off

Only if you insist on adding this new feature (or bug fix or what have
you) for off-line readers without making off-line readers change.  If you
remove that (IMO artificial) constraint, there is a 100% solution, namely
supporting the existing protocol while still specifying a transition path
for people to adopt the new header which offers improved behavior in that
particular situation once people have adopted it.

>> reinjected articles and therefore rpost removes all of those trace
>> headers to ensure that its messages don't look like reinjection.

> cause breakages in any system, including the current one (all they have
> to do at present is to remove the Date header and all the evils you have
> been discussing will promptly ensue).

What existing standard does that behavior break?

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 9 Mar 2007 17:34:00 GMT

>What existing standard does that behavior break?

Current best practice is certainly that changing anything in articles
other than in variant headers (Path, Xref, etc) is a big No-No. I am not
sure how explicitly RFC 1036 forbids it, but our news drafts certainly lay
it down as a MUST NOT once an article has been officially injected.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 9 Mar 2007 17:44:23 GMT

>support, and that again limits us down to only those articles posted by

If relayers never see any significant number of articles with this "new"
header, they will say "why should we bother to implement it".

>Isn't that exactly the case where adding it causes the most damage if

The longer the cutoff for what is "suspicious", the more damage might
ensue. With most articles, 99.9% of cases there is no difference, as you
say. When the cutoff is 1 day, that might come down to say 97%. Exactly
where a reasonable cutoff would be is an interesting question requiring
discussion, which is why I did not suggest a precise value.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 09 Mar 2007 21:37:01 -0800

>>> cause breakages in any system, including the current one (all they
>>> have to do at present is to remove the Date header and all the evils
>>> you have been discussing will promptly ensue).

> other than in variant headers (Path, Xref, etc) is a big No-No.

Not when doing this sort of gatewaying.  This is how suck/rpost feeds
pretty much universally work.  This type of gatewaying is common practice;
it doesn't make sense to me to say that it's in violation of best
practice.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 09 Mar 2007 21:38:51 -0800

>> Why would a significant number of articles arriving with this header
>> cause a relayer to change?  I don't see how the situation is any
>> different for relayers either way.  The only time there will be any
>> pressure on relayers to adopt this is if they're discarding articles
>> because of not having this support, and that again limits us down to
>> only those articles posted by off-line readers.

> header, they will say "why should we bother to implement it".

I don't believe the mere existence of the header will cause any change.
Lots of new headers show up and no one does anything about them.  What
will make a difference is specifically the feature that the header
enables, which involves the same number of articles whether injecting
agents are adding it routinely or not.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 12 Mar 2007 11:41:19 GMT

>pretty much universally work.  This type of gatewaying is common practice;

Sounds horrible! Especially if done deliberately (I can see it arising
easily from misconfiguration). But it is common for Date to be removed in
that practice?

---

From: Russ Allbery <[email-masked]>
Date: Mon, 12 Mar 2007 10:17:26 -0700

> easily from misconfiguration).

No, it's deliberate.  It's the only way that rpost can work properly.

> But it is common for Date to be removed in that practice?

All the setups that I'm aware of retain Message-ID and Date, although I've
not done a lot of investigation and haven't run such a configuration
personally.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 13 Mar 2007 20:42:50 GMT

>No, it's deliberate.  It's the only way that rpost can work properly.

I just read the manpage for rpost, and indeed it recommends such practices.

But I cannot see why any legitimate ordinary user would need to do such a
thing. Why does he ever need to rpost any article other than one he
generated himself on his own system (i.e. why should he want to relay).

So I still think it is horrible. What is the intended application where it
is needed?

---

From: Russ Allbery <[email-masked]>
Date: Tue, 13 Mar 2007 23:23:15 -0700

> generated himself on his own system (i.e. why should he want to relay).

Given how rpost works, they have to do this in order to successfully
propagate a message that they posted themselves on their own system.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 15 Mar 2007 16:18:09 GMT

>propagate a message that they posted themselves on their own system.

Ah! This is the case of a reinjection from a one-node private network
(which could be regarded as a gateway, as in your Usepro draft). So it is
reasonable to assume such a network has no other hidden 'leaks', and that
is fine. But it will need careful warnings to ensure people do not try
these tricks where they are not in a position to guarantee the absence of
leaks.

And I wish the man page for rpost had made it clearer that this facility
was most certainly NOT to be used for reinjecting articles received from
outside (i.e. such sites should noway be relaying, which is what that man
page seemed to imply).