Subject: [AAA-WG]: Issue 83: Routing/forwarding clarifications

From: Pat Calhoun <[email-masked]>
Date: Fri, 22 Jun 2001 11:05:53 -0700

All,

This is a much bigger issue than it seems. During yesterday's conference
call, we agreed with the items listed in this issue, but these raised
another sub-issue, which is that an explicit disconnect message is necessary
between peers. The reason being that if a node disconnects the transport
connection for a valid reason (e.g. insufficient resources), the last thing
it needs is the peer periodically attempting to re-connect. The way that
the draft is written, with the resilience and failover text, this is 
precisely what a node is to do when a disconnect occurs.

The issue listed the following, and my comments are included:
> - If multiple hosts listed for a given realm (redirect, relay, proxy), use 
> the same one for a given Session (either need to maintain Session state or 
> to use some hash function based on Session-ID) unless error occurs (and 
> then keep that one that works). 
This issue is handled in the transport draft, and does not really need to
be rehashed in the base spec.

> - Clarify if connections to peers are supposed to be opened at startup, or 
> when needed only. 

a new section was created to discuss this issue.

> - Clarify relationship of server discovery (SLP, DNS A, DNS SRV...) with 
> peer table and/or routing table. 

new text was added to the server discovery section.

> - move peer and routing table description earlier in the doc (somewhere in 
> 2.5 or 2.6). 

we agreed with this.

> - Clarify whether "on-demand" connections (e.g. Redirect-Host with 
> certificate) should timeout. 

this one is already taken care of in -06

> - refuse connection to oneself 

the group felt that this was an implementation issue, and must not be ruled out.

Given the above changes, and the inclusion of the disconnect message, a rather 
minor re-org of the document was necessary. In addition to moving some sections
around, I felt that the old sections 7 and 8 really had to do with dealing with
Diameter peers, and all peer related topics should be in a single section, which
is now section 6. I think this makes a whole lot of sense.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Here is the new table of contents (not shown here, but section 4 in the
table of contents was not aligned properly and has been fixed):

      [...]
      2.0  Protocol Overview
            2.1  Transport
                  2.1.1  SCTP Guidelines
            2.2  Securing Diameter Messages
            2.3  Diameter Protocol Extensibility
                  2.3.1  Defining new AVP Values
                  2.3.2  Creating new AVPs
                  2.3.3  Creating a new Diameter Applications
                  2.3.4  Application authentication procedures
            2.4  Diameter Application Compliance
            2.5  Application Identifiers
            2.6  Peer Table
            2.7  Realm-Based Routing Table
            2.8  Role of Diameter Agents
                  2.8.1  Relay Agents
                  2.8.2  Proxy Agents
                  2.8.3  Redirector Agents
                  2.8.4  Translation Agents
      [...]
      5.0  Diameter message processing
            5.1  Diameter request routing overview
                  5.1.1  Originating a Request
                  5.1.2  Sending a Request
                  5.1.3  Processing Local Requests
                  5.1.4  Request Forwarding
                  5.1.6  Request Routing
                  5.1.8  Redirecting requests
                  5.1.9  Relaying and Proxying Requests
            5.2  Diameter Answer Processing
                  5.2.1  Processing received Answers
                  5.2.2  Relaying and Proxying Answers
            5.3  Hiding Network Topology
            5.4  Origin-Host AVP
            5.5  Origin-Realm AVP
            5.6  Destination-Host AVP
            5.7  Destination-Realm AVP
            5.8  Routing AVPs
                  5.8.1  Route-Record AVP
                  5.8.2  Proxy-Info AVP
                  5.8.3  Proxy-Host AVP
                  5.8.4  Proxy-State AVP
            5.9  Auth-Application-Id AVP
            5.10 Acct-Application-Id AVP
            5.11 Vendor-Specific-Application-Id AVP
            5.12 Redirect-Host AVP
            5.13 Redirect-Host-Usage AVP
            5.14 Redirect-Max-Cache-Time AVP
      6.0  Diameter Peers
            6.1  Connecting to Peers
            6.2  Diameter Peer Discovery
            6.3  Capabilities Negotiation
                  6.3.1  Capabilities-Exchange-Request
                  6.3.2  Capabilities-Exchange-Answer
                  6.3.3  Vendor-Id AVP
                  6.3.4  Firmware-Revision AVP
                  6.3.5  Host-IP-Address AVP
                  6.3.6  Supported-Vendor-Id AVP
                  6.3.7  Product-Name AVP
            6.4  Disconnecting Peer connections
                  6.4.1  Disconnect-Peer-Request
                  6.4.2  Disconnect-Peer-Answer
                  6.4.3  Disconnect-Cause AVP
            6.5  Transport Failure Detection
                  6.5.1  Device-Watchdog-Request
                  6.5.2  Device-Watchdog-Answer
                  6.5.3  Failover/Failback Procedures
            6.6  Peer State Machine
                  6.6.1  Incoming connections
                  6.6.2  Events
                  6.6.3  Actions
                  6.6.4  The Election Process
       [...]

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I moved the peer and routing table up into section 2, as requested by this
issue. Furthermore, the addition of some text in the section that discusses
dynamic peer discovery (section 6.2) required the addition of new fields 
in the tables. Here is the next text:

2.6  Peer Table

   The Diameter Peer Table is used in message forwarding, and referenced
   by the Domain Routing Table. A Peer Table entry contains the
   following fields:
      [...]
      - Status. This is the state of the peer entry, and MUST match one
        of the values listed in section 6.6.
      - Role. This field specifies whether a peer is a primary,
        secondary or alternate.
      - Static or Dynamic. Specifies whether a peer entry was statically
        configured, or dynamically discovered.
      - Expiration time. Specifies the time which dynamically discovered
        peer table entries are to be either refreshed, or expired.
      [...]

2.7  Realm-Based Routing Table

   All Realm-Based routing lookups are performed against what is
   commonly known as the Domain Routing Table (see section 16.0). A
   Domain Routing Table Entry contains the following fields:
      [...]
      - Static or Dynamic. Specifies whether a route entry was
        statically configured, or dynamically discovered.
      - Expiration time. Specifies the time which dynamically discovered
        a route table entry expire.
      [...]

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Finally, there have been changes to section 6, which I list here. However,
I do want to bring up the fact that there is one open issue. The state
machine doesn't have a state for "suspect" peers, so the table in 6.1
shows "????" as the state. I believe that we need to have a state
in the state machine for suspect peers. Paul, any ideas?

6.0  Diameter Peers

   This section describes how a Diameter nodes establish connections and
   communicate with peers.

6.1  Peer Connections

   Although a Diameter node may have many possible peers that it is able
   to communicate with, it may not be economical to have an established
   connection to all of them. At a minimum, a Diameter node SHOULD have
   an established connection with a primary and secondary peer, and MAY
   have additional connections, if it is deemed necessary. However, when
   a connection with either the primary or secondary peer becomes
   questionable, a connection to an alternate peer SHOULD be
   established.  The questionable peer should then be removed from the
   primary/secondary list, and the new peer should be added to the list
   (see figure 6).

    +------------------------------+  +------------------------------+
    |  Peer  | Status  |    Role   |  |  Peer  | Status  |    Role   |
    +--------+---------+-----------+  +--------+---------+-----------+
    | peer A | Open    | Primary   |  | peer B | Open    | Primary   |
    | peer B | Open    | secondary |  | peer C | Open    | secondary |
    | peer C | Closed  | alternate |  | peer A | ????    | alternate |
    +------------------------------+  +------------------------------+
         A. prior to failure              B. following failure

    +------------------------------+  +------------------------------+
    |  Peer  | Status  |    Role   |  |  Peer  | Status  |    Role   |
    +--------+---------+-----------+  +--------+---------+-----------+
    | peer B | Open    | Primary   |  | peer B | Open    | Primary   |
    | peer C | Open    | secondary |  | peer C | Open    | secondary |
    | peer A | Open    | alternate |  | peer A | Closed  | alternate |
    +------------------------------+  +------------------------------+
      C1. Connection has stabilized     C2. Connection was terminated

                  Figure 6: Transitions of a Peer Table

   Figure 6 shows the sequence of changes made to a peer table following
   a network failure, causing the connection to peer A to become
   questionable. The following is a description of the steps.
      A.  Peers A and B are the current primary and secondary,
          respectively.  peer C is an alternate server, and no transport
          connection is established with it.
      B.  The connection to peer A becomes questionable, causing a
          connection to be established with peer C. Peer B is moved to
          primary, peer C to secondary and peer A as an alternate. if
          the connection to A stabilizes, the next event is C1,
          otherwise C2.
      C1. Three successful watchdog messages have been exchanged with
          peer A, and the connection is considered stabilized. The
          connection remains open, but the peer is still considered as
          an alternate.
      C2. The connection with peer A was shutdown, and peer A's state is
          changed to disconnected.

6.2  Diameter Peer Discovery

   [...]

   A dynamically discovered peer causes an entry in the Peer Table (see
   section 2.6) to be created. Note that entries created via DNS MUST
   expire (or be refreshed) within the DNS TTL. If a peer is discovered
   outside of the local realm, a routing table entry (see Section 2.7)
   for the peer's realm is created. The routing table entry's expiration
   MUST match the peer's expiration value.

6.3  Capabilities Exchange

   [...]

6.4  Disconnecting Peer connections

   When a Diameter node disconnects one its transport connections, its
   peer cannot know the reason for the disconnect, and will most likely
   assume that a connectivity problem occured, or that the peer has
   rebooted. In these cases, the peer may periodically attempt to
   reconnect, as stated in section 2.1. In the event that the disconnect
   was a result of either a shortage of internal resources, or simply
   that the node in question has no intentions of forwarding any
   Diameter messages to the peer in the foreseeable future, a periodic
   connection request would not be welcomed. The Disconnection-Reason
   AVP contains the reason the Diameter node issued the Disconnect-
   Peer-Request message.

   The Disconnect-Peer-Request message is used by a Diameter node to
   inform its peer of its intent to disconnect the transport layer, and
   that the peer shouldn't reconnect unless it has a valid reason to do
   so (e.g.  message to be forwarded). Upon receipt of the message, the
   Disconnect-Peer-Answer is returned, which SHOULD contain an error if
   messages have recently be forwarded, and are likely in flight, which
   would otherwise cause a race condition.

   The receiver of the Disconnect-Peer-Answer initiates the transport
   disconnect.

6.4.1  Disconnect-Peer-Request

   The Disconnect-Peer-Request (DPR), indicated by the Command-Code set
   to 282 and the Command Flags' 'R' bit set, is sent to a peer to
   inform its intentions to shutdown the transport connection.

   Message Format

      <Disconnect-Peer-Request>  ::= < Diameter Header: 282, REQ >
                                     { Origin-Host }
                                     { Origin-Realm }
                                     { Destination-Host }
                                     { Disconnect-Cause }

6.4.2  Disconnect-Peer-Answer

   The Disconnect-Peer-Answer (DPA), indicated by the Command-Code set
   to 282 and the Command Flags' 'R' bit cleared, is sent as a response
   to the Disconnect-Peer-Request message. Upon receipt of this message,
   the transport connection is shutdown.

   Message Format

      <Disconnect-Peer-Answer>  ::= < Diameter Header: 282 >
                                    { Result-Code }
                                    { Origin-Host }
                                    { Origin-Realm }
                                    { Destination-Host }

6.4.3  Disconnect-Cause AVP

   The Disconnect-Cause AVP (AVP Code 273) is of type Enumerated.  A
   Diameter node MUST include this AVP in the Disconnect-Peer-Request
   message to inform the peer of the reason for its intention to
   shutdown the transport connection. The following values are
   supported:

      REBOOTING                         0
         A scheduled reboot is imminent.

      BUSY                              1
         The peer's internal resources are constrained, and it has
         determined that the transport connection needs to be shutdown.

      DO_NOT_WANT_TO_TALK_TO_YOU        2
         The peer has determined that it does not see a need for the
         transport connection to exist, since it does not expect any
         messages to be exchanged in the foreseeable future.

   [...]

---

From: Jacques Caron <[email-masked]>
Date: Sat, 23 Jun 2001 19:47:40 +0200

Hi,

Good work. Some comments, though:

- The reorg of the different sections looks good, but I would also move the 
new section 6 before section 5, actually. One starts by establishing 
connections to peers before routing packets around.

BTW, another sub-issue (a bit related to the other discussion I started 
about separating more the protocol from the applications) is that protocol 
messages (such as CER, DWR, etc.) should either not be in the pending 
request list, or explicitly not sent to alternate peers in case of failure 
of a connection, of course... Actually, I think most protocol messages 
(local to a connection) don't need the ack (delivery is guaranteed by the 
underlying transport, unless the connection is lost and then the message 
doesn't make sense any more), so don't need the R/A paradigm, and don't 
need to be saved to the pending request queue. I really need to propose 
some text along the lines of that separation of protocol and applications...

- in your description of the peer table, it looks too much like you only 
have one default realm and a set of peers (as would be the case on most 
NASes). There might be multiple different primary/backup peers for 
different realms, and a given peer might serve multiple realms and have 
different roles (primary/backup/etc.) for each realm. I think this 
information belongs in the routing table rather than in the peer table.

- the concept of "questionable" or "suspect" peers with a reliable 
transport underneath is quite intriguing to me?

- for all the stuff that is in the transport draft: I had understood that 
draft was actually just recommendations about what should be present in AAA 
protocols (and specifically Diameter), but that the AAA protocol definition 
was to include whatever is necessary to meet these recommandations?

I'll probably have some more coming up once I digest the changes :-/

Jacques.

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

---

From: Paul Funk <[email-masked]>
Date: Sun, 24 Jun 2001 18:26:08 -0400

Pat,

I assume you mean by "suspect" peer one whose watchdog timer has 
finally elapsed without a response?

If I remember correctly from the interim meeting, when the watchdog 
finally elapses without a response, we don't actually close the connection 
(since the transport will do that if it's truly broken); we just try to use 
alternate peers instead. (Is that right?)

While there is text that describes Tw and when to reset the watchdog 
timer, there is no text that describes when you decide that the connection 
is suspect.

To handle the entire watchdog algorithm in the state machine would 
lead to a lot of complexity. Instead, what if watchdogs were removed 
from the state machine and the "Transport Failure Detection" section were 
expanded to more fully describe the algorithm and when the connection 
is deemed suspect, how it can redeem itself, etc.? 

Another question that needs clarification: Once a connection becomes 
suspect, in addition to connecting to alternate peers, is the Diameter 
node expected to failover pending requests? Or is it supposed to do 
this only upon actual disconnect? Currently, the "Failover/Failback 
Procedures" section speaks of transport failures only.

Paul

Paul Funk
Funk Software, Inc.
617 497-6339
[email-masked]

---

From: Pat Calhoun <[email-masked]>
Date: Mon, 25 Jun 2001 06:49:26 -0700

> connections to peers before routing packets around.

I was considering that when making the changes, but wasn't able to
convince myself. I agree with making this change, and it will be
done for the next revision.

> some text along the lines of that separation of protocol and applications...

actually, a request without the 'P' bit set doesn't get added to the pending
message list.

> information belongs in the routing table rather than in the peer table.

hmmm... they actually belong in both, but I am not sure how to put that down
in ASCII. The basic issue is that for each realm, you want a primary/secondary,
but the actual relationship is:

                           /------- Peer Entry (Primary)
   Route Table Entry +-----
                           \------- Peer Entry (Secondary)

So, my thoughts is that a route table points to a set of peer entries. Each
peer entry is marked with its role, and not the route table. Of course, an
intermediate "shim" may be required between the route table and the peer
entry, but this makes implementations a tad more complex.

> transport underneath is quite intriguing to me?

I am open to suggestions, but that's all I could think of at that time. I
am certain you know what I mean, right?

> was to include whatever is necessary to meet these recommandations?

yes, but I believe that Bernard didn't necessarily want to dup unnecessarily.
However, if you feel strongly that we can put a summary of what's in the
transport draft (and perhaps reference the transport draft), then that would
be acceptable to me.

PatC

---

From: Pat Calhoun <[email-masked]>
Date: Mon, 25 Jun 2001 06:54:13 -0700

> finally elapsed without a response?

yes

> alternate peers instead. (Is that right?)

which is why I showed that a peer went into the "suspect" state, and eventually
either goes back into the open, or the closed state.

> timer, there is no text that describes when you decide that the connection 

right.

> is deemed suspect, how it can redeem itself, etc.? 

I had tried to make the changes to the state machine, but without relying on
a counter in a control block, it makes for a rather nasty, and lengthy,
change in the state machine. So, I agree that some text should be added
somewhere else. However, it may be useful to add a single "suspect" state
in the SM, and have text, as you state, that describes the counter.

> Procedures" section speaks of transport failures only.

we've never discussed this one :(

PatC

---

From: Jacques Caron <[email-masked]>
Date: Mon, 25 Jun 2001 16:34:05 +0200

>actually, a request without the 'P' bit set doesn't get added to the pending

Shouldn't indeed, but that's not what the draft says :-(

>entry, but this makes implementations a tad more complex.

I hard started to write "belongs in the routing table (or rather in the 
peer sub-table associated with each routing table entry)", but that looked 
a bit complex indeed.

I see two ways of describing this:
1. each entry in the routing table has a realm, an application, a priority, 
an action, and a peer, given that multiple entries can have the same realm 
and/or application.

2. an entry in the routing table has a realm, an application, an action, 
and a pointer to a subtable. The said subtable has priorities and pointers 
to the actual peers.

Now, that solves the requirements of data, but not what should be done with 
that, i.e.  when should a connection actually be established or torn 
down... I start to see doubly-linked lists :-(

>am certain you know what I mean, right?

I have a rough idea, though depending on the relationship between the 
watchdog timer and whatever timers the underlying protocol uses might be 
interesting.

>transport draft (and perhaps reference the transport draft), then that would

The transport draft should at least be referenced if there is stuff in 
there that is a MUST for Diameter implementations :-)

Jacques.

---

From: Paul Funk <[email-masked]>
Date: Mon, 25 Jun 2001 13:27:35 -0400

Pat,

>in the SM, and have text, as you state, that describes the counter.

Since watchdog functionality doesn't affect whether a connection is open 
or closed, it really can be handled outside the peer state machine. All 
watchdog activities occur in the Open state, similar to application message 
handling. 

I think that we need to decide how specific we want to be in describing 
watchdog functionality. 

At the low end, we can simply say that a node may use DWR for early 
detection of failed or congested connections, must use jitter, must respond 
to DWR with DWA, but otherwise leave the algorithm to the implementation. 

At the high end, we could provide a separate watchdog state machine or 
pseudo-code, timer definitions, etc., to try to specify exact behavior.

I favor the low end approach, leaving it up to implementations to determine 
when a connection is suspect. After all, exact adherence to a specific 
algorithm is not required for interoperability (as in the peer state machine). 

I also would favor reduction of the proposed "6.1 Peer Connections" section 
to something more simple. None of the elaborate tables and statuses and 
transitions affect interoperability. All that really needs to be stated is
that a 
node need not connect to everyone it knows all at once, but may want to 
manage connections. For example, it may want to connect to only a subset of 
peers in a realm, take down unused connections, and connect to additional 
peers when existing connections are deemed suspect. 

Paul

Paul Funk
Funk Software, Inc.
617 497-6339
[email-masked]

---

From: Pat Calhoun <[email-masked]>
Date: Mon, 25 Jun 2001 17:04:22 -0700

> algorithm is not required for interoperability (as in the peer state machine). 

I also favor the low end approach.

> peers when existing connections are deemed suspect. 

that works for me. I added the table because I thought it would be beneficial
to the reader (and we seem to want to detail as much as possible), but cutting
the section down is something I would also like.

patC

---

From: Pat Calhoun <[email-masked]>
Date: Tue, 26 Jun 2001 06:47:31 -0700

> Shouldn't indeed, but that's not what the draft says :-(

ok, in the process of making this change, I no longer agree with my previous
statement. I think that it is required to receive, and process, the -Answer. 
For watchdog messages, receiving, and processing the -Answer is the difference
between thinking that a peer is up, or not. Furthermore, an implementation may
be looking at the watchdog's RTT to determine whether a peer is in good standing
or not.

So, I disagree with this change, but agree that some text must be added stating
that certain messages MUST NOT be sent to an alternate.

I'll look into this, and figure out the best approach.

PatC

---

From: Pat Calhoun <[email-masked]>
Date: Tue, 26 Jun 2001 07:29:13 -0700

> the section down is something I would also like.

Paul, how about the following text:

5.1  Peer Connections

   have additional connections, if it is deemed necessary.

   When a peer is deemed suspect, which could occur for various reasons,
   including not receiving a DWA within an alloted timeframe, no new
   requests should be forwarded to the peer, but failover procedures are
   not invoked. When an active peer is moved to this mode, additional
   connections SHOULD be established to ensure that the necessary number
   of active connections exists.

   There are two ways that a peer is removed from the suspect peer list:
      1. The peer is no longer reachable, causing the transport
         connection to be shutdown. The peer is moved to the closed
         state.
      2. Three watchdog messages are exchanged with accepted round trip
         times, and the connection to the peer is considered stabilized.

Thanks,

PatC

---

From: Pat Calhoun <[email-masked]>
Date: Tue, 26 Jun 2001 07:37:44 -0700

> I'll look into this, and figure out the best approach.

Jacques,

looking at how to make this change, I believe that the best approach is simply
to add text to the relevant command definitions. Specifically, there are only
three messages that fall within this category, and they are CER, DPR and DWR.

Here is an example of the change (see last sentence):

5.3.1  Capabilities-Exchange-Request

   The Capabilities-Exchange-Request (CER), indicated by the Command-
   Code set to 257 and the Command Flags' 'R' bit set, is sent to inform
   a peer that a reboot has occurred. Upon detection of a transport
   failure, this message MUST NOT be sent to an alternate peer.

Thanks,

PatC

---

From: Pat Calhoun <[email-masked]>
Date: Tue, 26 Jun 2001 07:40:55 -0700

> down... I start to see doubly-linked lists :-(

There is no way around doubly-linked lists in this case.

How about if a routing table entry has a pointer to the primary peer entry,
secondary peer entry, and then additional pointers for alternates. That
eliminates the need for a "shim", and duplicate data structures.

PatC

---

From: Paul Funk <[email-masked]>
Date: Tue, 26 Jun 2001 13:03:50 -0400

Pat,

That's much better.

Paul

Paul Funk
Funk Software, Inc.
617 497-6339
[email-masked]