Subject: [dnsext] draft-ietf-dnsext-rfc1995bis-ixfr -- questions on s3.2.3

From: Alfred Hönes <[email-masked]>
Date: Wed, 18 Apr 2012 23:59:08 +0200 (MESZ)

Note:
  This thread actually has started on the predecessor individual
  draft, but to acoid confusion on the target of the discussion,
  I have changed the Subject to refer to the adopted WG draft,
  draft-ietf-dnsext-rfc1995bis-ixfr-00.

The questions were about the level and detail of the requirements
posed onto an IXFR server in packetizing the conceptual IXFR
response -- in all the different cases that can occur --, and how
thereby the detrimental need for IXFR clients to rely on timeouts
(to determine the outcome/state of an IXFR session, under very
specific conditions) can be avoided.

I have revisited recent messages and previous work, and tried
to arrive at the conclusions that look most reasonable and seem
to correspond to evolving consensus.

Observations and conclusions:

(1)
IXFR is intended as a more efficient method for in-band zone
synchronization than AXFR (for deployment scenarios where it
makes sense -- as explained in the draft); therefore, it seems
fair that the specification is tailored to support timely,
efficient completion of IXFR sessions under most conditions,
and that the requirements for IXFR server implementations be
at least as strong as the comparable requirements for modern
AXFR servers that fully conform to RFC 5936 (excluding the
kludges for backward compatibility still supported by that RFC).

(2)
Updated protocol specifications are not required to still contain
specific requirements, or the lack of specific requirements, if
that has been identified as a weak point in the protocol.

So an updated specification needs to be allowed to specify more
specific requirements than its predecessor, but preferably this
should be done in a manner that interoperability with legacy
implementations is assured as far as it makes sense.

Of course, legacy implementations that do not follow the more
stringent rules of the revised specification cannot claim
conformance to the new specification until they get updated
accordingly.  But that's the fate of implementations at large,
and the DNS world will not be doomed thereby.  :-)

(3)
RFC 5936, Section 2.2 (at the bottom of page 11) specifies:

|  Each AXFR response message SHOULD contain a sufficient number of RRs
|  to reasonably amortize the per-message overhead, up to the largest
|  number that will fit within a DNS message (taking the required
|  content of the other sections into account, as described below).

This was the consensus of the WG two years ago.

I conclude -- and that seems to be in line with feedback from
implementers -- that, to achieve the performance goal of IXFR,
rfc1995bis needs to be *at least* as strong as RFC 5936 in its
requirements in Section 3.2.3.

Therefore, unless there is strong consensus to the contrary
arising quickly, the next draft version will use uppercase
"SHOULD" in this context:

!  For multi-message IXFR responses, the conceptional answer is split
!  into segments that are sent in order.  Each segment is comprised of
!  an integer number of full RRs, and for transport efficiency, the
!> response messages SHOULD be filled up with answer RRs as much as
!  possible for the response message size chosen by the IXFR server,
!  taking into account the space needed for the other sections in the
!  messages.

(4)
Bulk IXFR runs over TCP, with potentially concurrent transactions
for different zones served by the same authoritiative servers.
Since the IXFR protocol allows very different kinds of responses
(as specified in a fully backward-compatible manner in the draft)
the most important kinds of which will consist of multiple response
messages, yet does not contain an overall length indication or a
specific, generic "end-of-response" indication for multi-message
responses, there is a clear need for:
  * rules that ensure that the kind of response can be determined
    instantaneously and without ambiguity upon receipt of the first
    response message; and
  * rules that clearly allow to determine when a received response
    message is the last one in an IXFR session.

As already indicated in the draft, there are a few specific
conditions where the interpreation of an IXFR response cannot be
disambiguated until (and if) the next response message arrives,
unless IXFR servers are required to packetize their responses
in a manner that allows immediate disambiguation at the client.
Waiting (in corner cases) for another response message in a given
IXFR session in order to eventually determine the status of the
response requires the IXFR client to set a timeout and wait -- as
a normal protocol operation.  All such timeouts are generally
regarded as detrimental, and in particular detrimental to the
performance optimization we have as a target goal for IXFR.

Therefore, unless strong consensus to the contrary arises quickly,
the next draft version will contain a more precise description of
what conditions indicate the end of an IXFR response, and how the
different kinds of IXFR responses are to be disambiguated.
But this needs to impose on IXFR servers a very few new, specific,
stronger requirements for the packetization of their responses.
These conditions already are in the draft, but the text will be
revised to distinguish the new requirements and spell out the
rationale.

The core of this clarification will be a new clause added below the
text quoted in (3) above, stating (modulo wordsmithing t.b.d.):

!  If an IXFR server intends to send a conceptual IXFR response that
!  is comprised of at least two answer RRs, the first two RRs of the
!  response MUST be sent in the first response packet to allow the IXFR
!  client to immediately distinguish the kind of response coming in.
!  An IXFR server MUST NOT send over connection-oriented transport the
!  kind of (single-RR) IXFR response that indicates that the server has
!  to send a multi-packet response and hence the IXFR request needs to
!  be retried over connection-orientend transport (currently: TCP);
!  this kind of response is only allowed in response to an IXFR query
!  received over connectionless transport (currently: UDP).
!  See Section 4 for the details of the various kinds of conceptual
!  IXFR responses and how an IXFR client distinguishes these.

The verbiage already present in the draft that indicates the
necessity to anyway set up timeouts as a safeguard to protect
agains fatal conditions will be amended to say that this is also
necessary for seamless interoperation with older IXFR servers not
conforming to these new requirements, but the efficiency of IXFR
operations in specific circumstances will no longer depend on
clever heuristics to choose proper values, once the IXFR servers
used in a given deployment are updated to conform to the revised
specification, since the timeout values then will only apply in
fatal cases and can be chosen rather long for resilience reasons
without negative effects on normal protocol operation in specific
cases.

Finally, a closely related topic / question:

(5)
The example of "fallback to AXFR" in the draft (at the very end of
Section 2) currently suggests that, for efficiency at both the IXFR
server and the IXFR client, for full zone transfer (fallback to AXFR)
an IXFR server (typically) sends the zone RRs grouped by RRset.

It has been correctly observed that this ordering does not correspond
to a requirement contained in the new AXFR specification, RFC 5936.

Should this behavior be recommended/RECOMMENDED in the draft
for new IXFR server implementations?

Kind regards,
  Alfred.

-- 

+------------------------+--------------------------------------------+
| TR-Sys Alfred Hoenes   |  Alfred Hoenes   Dipl.-Math., Dipl.-Phys.  |
| Gerlinger Strasse 12   |  Phone: (+49)7156/9635-0, Fax: -18         |
| D-71254  Ditzingen     |  E-Mail:  [email-masked]                     |
+------------------------+--------------------------------------------+

---

From: Alfred Hönes <[email-masked]>
Date: Thu, 19 Apr 2012 11:57:10 +0200 (MESZ)

Regarding the final point in my previous posting ...

>> ...

Based on the feedback received, some implementations do this grouping
by RRset, but it looks like implementers would prefer the identical
set of RR placement/ordering requirements as in RFC 5936 for AXFR,
and not to have recommendations beyond that in the IXFR document.
I.e., there will be no ordering requirements for the RRs of a zone
beyond the dual (and dual-purpose) placement of SOA RRs (conveying
the SOA RR as such and serving as structure tags or 'sentinels'
for the response).

So the final part of Section 2 in the next version of the rfc1995bis
I-D will avoid mention of RRsets. My working copy of the draft now
says, including the new explanation at the end:

!  In contrast, in the case of fallback to AXFR, the IXFR response would
!  convey, in order:
!
!     *  SOA for sn_n    # first instance
!     *  {other RRs of the zone at sn_n, in arbitrary order}
!     *  SOA for sn_n    # repeated as trailing delimiter
!
!  In these examples, for convenience "other RRs ..." is used as a
!  shorthand for the more precise words "zero or more other RRs ...".

Agreed?

Kind regards,
  Alfred.

---

From: Andreas Gustafsson <[email-masked]>
Date: Fri, 20 Apr 2012 12:00:46 +0300

Alfred,

> performance optimization we have as a target goal for IXFR.

Waiting for additional messages to arrive (with a timeout for error
recovery) is a standard part of any protocol involving multi-message
responses and should not be considered detrimental as such.  The
detrimental case would be if a client could not determine in advance
whether or not the server is going to send another message, and had to
resort to using a timeout as a way of detecting the end of the
response as part of normal protocol operation.  The IXFR protocol as
specified in RFC 1995 does *not* require clients to resort to such
methods; it is always possible for an RFC 1995 client to unambiguously
determine whether or not the server will send an additional message.

Digging in the archives, I found a draft from an earlier effort to
produce an update to the IXFR specification, back in 2000:

  http://tools.ietf.org/html/draft-ietf-dnsext-ixfr-01

Quoting that draft:

   A slave receiving an IXFR response needs to classify it as one of the
   following four cases:

      UDP-overflow     An indication that the transfer will not fit in a
                       UDP packet and should be retried over TCP

      up-to-date       An indication that the serial number of the
                       request is current and no transfer is necessary

      incremental      An incremental transfer

      nonincremental   A full zone transfer

   Performing this classification requires some care.  For example,
   UDP-overflow responses differ from UDP up-to-date responses only in
   the value of the SOA serial number.  Also, to distinguish between a
   nonincremental and an incremental transfer, the slave needs to
   receive the first two response RRs and check whether the second one
   is a SOA.  If the master chose to transmit each RR in a separate TCP
   message, this involves waiting for a second TCP response message.  On
   the other hand, in the case of an up-to-date response, the slave must
   not wait for a second TCP message as doing so would cause it to hang
   waiting for a message the master will never send.  Therefore, the
   slave must examine the first message and eliminate the possibility
   that it is a TCP up-to-date response before it attempts to receive a
   second message.

   Slaves must not attempt to classify a response based on incidental
   information such as the presence or absence of a question section,
   the QTYPE field of a possible question section, or the number of
   response RRs in a TCP response message.

   An example algorithm for classifying IXFR responses appears in
   Appendix A.

> different kinds of IXFR responses are to be disambiguated.

I suggest doing the latter by reusing the above quoted text from
draft-ietf-dnsext-ixfr-01, and/or the pseudocode from its Appendix A.

> revised to distinguish the new requirements and spell out the

I still say there is no actual need to introduce any such new
packetization requirements.

Regards,
-- 
Andreas Gustafsson, [email-masked]

---

From: Alfred Hönes <[email-masked]>
Date: Fri, 20 Apr 2012 12:57:43 +0200 (MESZ)

Andreas,
thanks for your additional feedback.

> determine whether or not the server will send an additional message.

But that has not been described in RFC 1995.

>    An example algorithm for classifying IXFR responses appears in

Interesting quote; thanks! I've picked up that draft now.

But guess what I already have in my working copy
draft-ietf-dnsext-rfc1995bis-ixfr-01(alpha), in Section 4 ...

!  The possible IXFR responses can be classified into various kinds of
!  responses by the number of answer RRs in the conceptual response.
!
!  immediate-error:  empty answer, non-zero RCODE;
!
!  redirect-to-conn:  single SOA RR for sn_n > sn_o, RCODE = NoError;
!
!  you-are-current:  single SOA RR for sn_n = sn_o, RCODE = NoError;
!
!  you-are-more-recent:  single SOA RR for sn_n < sn_o, RCODE = NoError;
!
!  incremental:  multiple RRs, starting and ending with the SOA RR for
!     sn_n > sn_o, which enclose a sequence of one or more change
!     information chunks, the first of which starts with the SOA RR for
!     sn_o (which consequentially is different from sn_n);
!     this is detailed below in Section 4.1
!     (if need arises, a packetized incremental response can be aborted
!     in the second or later response message by sending an empty Answer
!     section and non-zero RCODE);
!
!  fallback-to-axfr:  multiple RRs, starting and ending with the SOA RR
!     for sn_n > sn_o, enclosing a serialization of all other RRs in the
!     zone for sn_n (which hence does not contain any other SOA RR and
!     consequentially cannot start with another SOA RR);
!     this is detailed in [RFC5936]
!     (if need arises, a packetized fallback-to-axfr response can be
!     aborted in the second or later response message by sending an
!     empty Answer section and non-zero RCODE).

... sounds very similar, isn't it?

It's consoling that independent analysis now led to comparable results
as the thinking was more than a decade ago.   :-)

> draft-ietf-dnsext-ixfr-01, and/or the pseudocode from its Appendix A.

Firstly, see above; second, -- excluding the treatment of error cases
that our draft also covers -- the only significant difference between
that pseudocode and the more detailed description in our I-D (already
being expanded upon as well in my working copy) seems to be the
requirement for the second RR to be contained in the first response
packet to avoid the waiting for the second packet before 'incremental'
can be distinguished unambiguously from 'fallback-to-axfr'.

Expect our -01 version to be uploaded by early next week
(my co-authors were busy on a conference this week).

> packetization requirements.

According to feedback from current implementers, they seem to have no
trouble with the suggested small tightening -- in particular as it
effectively needs no additional effort in server implementations
that already follow the "SHOULD" for AXFR in RFC 5936 (to pack as
much answer RRs into the response packets as possible) and reuse
related code for IXFR.

I note that the example in Section 7 of the '2000 draft also shows
answer RR packing, which is needed anyway if the IXFR server intends
to make use of the option to send (packed) incremental (or even
fallback-to-axfr) responses in a single UDP response packet, if
they fit therein.

Best refards,
  Alfred.

> Regards,

Best regards,
  Alfred.

---

From: Andreas Gustafsson <[email-masked]>
Date: Thu, 26 Apr 2012 12:31:50 +0300

Alfred,

Last week, you wrote (referring to the pseudocode in Appendix A
of http://tools.ietf.org/html/draft-ietf-dnsext-ixfr-01):

> [...] the only significant difference between

Yes, that difference is the sticking point.

> much answer RRs into the response packets as possible) and reuse

As I see it, the proposed requirement to send the first two RRs in the
same message isn't so much a "small tightening" as a fundamental
change to the nature of the protocol, from one defined in terms of a
sequence of RRs where message boundaries carry no meaning to one where
the message boundaries are significant.

Also, RFC 5936 doesn't actually require packing "as much answer RRs as
possible" into the response message.  It says:

   Each AXFR response message SHOULD contain a sufficient number of RRs
   to reasonably amortize the per-message overhead, up to the largest
   number that will fit within a DNS message (taking the required
   content of the other sections into account, as described below).

Consider for example a server implementation that chooses to start a
new message whenever adding another RR to the current message would
cause it to grow larger than 16 kB, in order to take maximum advantage
of name compression given that compression pointers can only point
into the first 16 kB of the message.  I would consider such an
implementation compliant with the above SHOULD, but it would not meet
the proposed requirement when the second RR in the transfer is
larger than 16 kB.

Even filling the message with as many answer RRs as possible is not
always sufficient meet the proposed requirement.  When the second RR
is close to 64 kB in size, it may not fit in the same message as the
initial SOA even though it would fit in a separate message.  If
servers will be required to treat this situation as an error instead
of just sending the RRs in separate messages, code changes are needed.

RFC 5936 also says:

   Some old AXFR clients expect each response message to contain only a
   single RR.  To interoperate with such clients, the server MAY
   restrict response messages to a single RR.  As there is no standard
   way to automatically detect such clients, this typically requires
   manual configuration at the server.

There are server implementations where such manual configuration
causes single-RR messages to be sent to both AXFR and IXFR clients.
A client following the decision procedure in the draft will fail to
interoperate with a server configured in this way.

> fallback-to-axfr) responses in a single UDP response packet, if

It's not that existing servers are incapable of packing multiple RRs
per message, or that they won't do it in the typical case, but that
there are atypical cases such as transfers involving large RRs or
non-default configurations where the initial SOA can end up being
sent in a message of its own.

That said, my main objection is not to the draft's "tightening" of
what the server may send, but to the corresponding "loosening" of what
the client must accept.  In other words, whether or not servers are
required to send the first two RRs in the same message, clients should
be required to accept TCP IXFR responses divided into messages at
arbitrary RR boundaries.

Regards,