Subject: RE: [Forces-protocol] Issue 2: Multiple channels/connections

From: "Putzolu, David" <[email-masked]>
Date: Fri, 12 Mar 2004 16:32:29 -0800

Hi Jamal.

My response is inlined below, but before I go into it,
I thought I'd show how one would use multiple logical
connections (sockets) in a clean fashion to provide
differential levels of service to packets on a variety
of implementations:

Case 1: Single Ethernet wire shared by FEs to talk to CE,
    supports QoS via 802.1p.   Each FE does the following:
Step 1) Open socket 1, which is intended for the low priority 
        packets.  Leave the 802.1p settings alone for packets
        going over this socket.
Step 2) Open socket 2, which is intended for the high priority
        packets. Make ioctl() or other call necessary to set
        any packets over this socket to get 802.1p bits set.
Result: High priority packets from any FE take precedence
    over low priority packets from any FE, only compete with
    high priority packets from other FEs.

Case 2: Two Ethernet wires, one for high priority packets
    shared by all FEs, other for low priority packets shared
    by all FEs. Each FE does the following:
Step 1) Open socket 1, which is intended for the low priority
        packets, using IP address on low priority Ethernet.
Step 2) Open socket 2, which is intended for the high priority
        packets, using IP address on high priority Ethernet.
Result: High priority packets from any FE only compete with
    high priority packets from other FEs.

I'd be interested in seeing how one would provide similar
differential treatment of packets using a single socket.
I've guessed at methods of doing it below, but my guesses
where all rather ugly and had the aforementioned layer
violations...

Now on to the response :)

Jamal wrote >
> Let me do a compare with a node with two Ethernet ports
> to put into context what you are saying. I suppose we can assume 
> we can attach IP addresses to these ports. If we are going to run 
> over IP, then such a node is referred to as multihomed.
> Is this a fair corelation?
>
> Lets refer to sockets as the logical connections.

Sure.

Jamal wrote >
> On Fri, 2004-03-12 at 12:49, Putzolu, David wrote:
> > 1) If I build hardware that has two physical channels, but
> >    ForCES carries everything in a single logical connection,
> >    it becomes impossible (short of extremely ugly layer
> >    violations) to use my dual channel hardware, because all
> >    the ForCES messages are coming down in a single stream.
> 
> So you have one socket but two ethernet ports.
> If i understood you corectly, lets say you had 10 ethernet 
> ports, would you then say you want to take advantage of all of them? 
> I suppose you have to draw a line somewhere which seems to me 
> this is a config choice, no?

Not sure what the implication is of this para, so I have no
response :)

Jamal wrote:
> If you are running on top of SCTP it would manage this for you.

SCTP multihoming uses secondary paths for backup purposes -
it transmits all packets on the primary channel, using the backup
only upon losses, eventually switching to backups if losses get
too bad (RFC2960, section 6.4).  An SCTP implementation might
choose to round-robin across the multihomed connections (RFC3286,
section 4), however, that is implementation dependent, and still
would in no way to treat high priority ForCES packets differently
from low priority ForCES packets - they would just be round-robined.

One could use SCTP's unordered option (RFC2960 section 6.6) by 
hacking the SCTP implementation it to put any U flagged packet 
on the backup channel, and all non-U packets on the primary 
channel.  This would prevent other FE's low priority traffic
(running on the primary channel) from congestion crushing the
high priority traffic from this FE, at least until the other FE's
SCTP decides to start using the backup channel (likely if the
primary is overloaded).  Furthermore, this requires a custom
SCTP implementation - modifying SCTP just to support ForCES is 
ugly (and not feasible in all operating systems - we cannot
mandate that anybody who wants to use ForCES must use an open
source OS where they can freely modify transport protocol 
implementations!)

Finally, one could hack SCTP to automatically break open all packets 
handed to it, guess if they are ForCES packets, and then decide 
which ForCES priority classes to map to the backup channel 
and which to map to the primary channel.  In addition to being a
layer violation, this runs into the same problem as above
in terms of hurting anybody who does not have free reign to 
modifying their SCTP implementation.

Jamal wrote >
> An alternative is to use simple lower layers tecniques like
> trunking/bonding. 

Trunking/bonding lower layers provides no differential treatment
of higher priority packets vs. lower priority packets.

> > 2) If I build hardware that has one physical connection, but
> >    has multiple levels of QoS (e.g. Ethernet backplane with
> >    support for 802.1p), but ForCES carries everything in a
> >    single logical connection, it becomes impossible to utilize
> >    my hardware's QoS support (short of those ugly layer 
> >    violations - Ethernet MAC chip cracking the ForCES header 
> >    to look at priority bits)
> 
> So one ethernet, one socket. 
> I know in Linux it would be trivial to map the Forces prio bits to
> either diffserv or 802.1p on the sending side. 

Here are a couple ways of doing this - if 
you have another in mind please let me know!
1) Hack Linux kernel to look at every packet handed over a
   TCP socket via write() and try to guess if it is a ForCES
   packet, then look at the priority field and set the 802.1p
   bits appropriately.  Modifying one's implementation of
   TCP or SCTP just to run ForCES is both a layer violation
   and impractical for anybody who doesn't have an open source
   OS where they can freely modify the transport proto impl.
2) Every time FEd is about to send a packet, before calling write()
   make the appropriate system call (ioctl?) to change the
   802.1p bits associated with the socket to match whatever 
   priority is needed.  This would likely cause packet 
   reordering within a single TCP or SCTP stream - causing 
   the expedited high priority packets to get stuck in SCTP/TCP 
   until the low priority ones make it through.  I guess if this
   was combined with setting the U flag for the packet in
   SCTP at least the reordering issue could be circumvented.

> > Basically, if one puts everything on one logical channel, it
> > precludes some very realistic, beneficial, and deployed 
> > hardware approaches.
> 
> I dont think multiple hardware paths should be precluded.
> What we need to understand is the nature of the beast.
> What differentiates the different "channels" and when to use them.
> I suppose the other question is "how many channels" - or is this a
> config issue? I dont see anything nmagical about the number 2.
> Or D & C.

My goal with this part of the conversation is merely to show
that putting everything over a single logical connection (socket)
severely impinges on the ability to (cleanly) implement 
any sort of prioritization system on the wire.  This was in 
response to the assertion made that separate channels don't 
give us anything.  This part of the discussion is more of a
theoretical discussion - I am trying to show what I
believe to be an objective point which is actually independent
of ForCES itself.

Once that point is cleared up I'll be happy to go into discussing
the (subjective) point of whether ForCES in particular needs the 
ability to treat packets differently, and what kind of packets 
deserve differential treatment.  :)

Cheers,
David

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: Jamal Hadi Salim <[email-masked]>
Date: 13 Mar 2004 10:05:11 -0500

Hi David,

>     high priority packets from other FEs.

sounds reasonable.

>     high priority packets from other FEs.

reasonable - but you need to know these details(there are two ethernet
links therefore must open two sockets and send high priority packets on
link 0) at bootstrap time. 

> where all rather ugly and had the aforementioned layer

I think the problem with your setup is you trying to have
knowledge of the phyiscal link at the Forces level. At least havent said
that you discover the number of physical links dynamically ;->

I have actually thought of this issue but never implemented it.
Implement Forces as a socket family. This means most of the
header is built by the (kernel) socket abstraction code.
And a lot of things are hidden from the application.
This does not preclude that some config is needed.
You have a single socket riding on top of many ethernet pipes.
FEd/CEd sets a default priority to be used in an out of band control
path to the kernel. 
Prioritization (as well as other packet attributes such as reliability
are done per packet). Anytime app wants to override the defaults
it sends out of band metadata(like cmsg interface) along with the
message and the socket abstraction uses that information to configure
appropriately.
The key is that the app is oblivious about how many physical or logical
connections exist and whether they go via the moon or mars.
Although i have spoken in terms of sockets and kernels - consider the
above as a communication abstraction layer. I havent paid attention to
the TIPC doc (its on my todo as is the model draft), i believe this is a
problem they must have already solved.
Jon?

> Not sure what the implication is of this para, so I have no

Same as what i said above; how much should the protocol know?
If there was 8 ethernet links available (to map to the three priority
bits), why not use all of them?
Why only have two? Is simplicty the reason, if yes, why block the 
use of 8 when i want to?;->
So instead of saying lets have a C&D, why not say lets have two links
with different qualities (of service). I gave the criteria as:
reliability, cost, latency and throughput. Its almost sounding like
RFC1349;-> 

[After reading Alexes email on how great PSTN networks are i cringed a
little. I thought i had paid for my sins already for working on SS7 
all those years, but he had to remind me;->].
The bellheads have this dogmatic view that there has to be clear
separation of control and data. Sure they will waste a gigabit pipe to
send 10kbps traffic on it. I thought these guys lost the battle 
already ;-> The design then is driven by "i am going to build this and
it will last 50 years and no replacement needed". Its anti-innovation
(although others may think its pro-robustness).
On a serious side, pick any commodity L2/3 switch blade (i know all my
companys products have this) these days and have at least one management
port thats out of band; it is not unusual to see two or even three.
Then theres the switch fabric which has many ports some of which could
be used for management.
But even given theres two classes of links (and i think people doing
CSIX may have a different view), what has this got to do with Forces
the protocol other than it being a config issue?

Let me cut the email here for readability. Go and get some coffee.

cheers,
jamal

---

From: [email-masked]
Date: Sun, 14 Mar 2004 00:32:45 +0900

Hi,

I have a question.  If there is no control traffic on the data link, 
how will you know if there is a data link failure?  Do you just wait 
for a transport protocol time out? And is that sufficiently timely?  Or 
will you have some other mechanism?

a.

---

From: Jamal Hadi Salim <[email-masked]>
Date: 13 Mar 2004 11:07:22 -0500

> > If you are running on top of SCTP it would manage this for you.
[useful comments removed for brevity]

ok. I was under the impression you could plugin a path selection
algorithm if you wanted (i could almost swear that was possible on
linux, but i may be wrong).
In any case i was looking more at getting to have connection management
to be a separate topic from the protocol.
I like some of the features SCTP has but it scares me on how
feature rich it is.

> source OS where they can freely modify transport protocol 

In Linux, this would be a simple change. Did you say other operating
systems exist?;->

> modifying their SCTP implementation.

On the sending side, the forces level priority is useful for mapping
to link/l3 layer prio. No violations; you pass metadata from the 
(higher) forces layer and it is mapped to the L2/3 level.
On the receive path the L2/3 prio is only useful at those (L2/3)levels.
At the application level, the forces prio becomes relevant for
processing reasons. Example, in a simple strict priority you could have 
8 queues and process the highest priority until theres no more
work left etc.

> of higher priority packets vs. lower priority packets.

A lot of vendors have some form of loadbalancing at this level.
For some glossy-ware, here are a couple of examples:

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml

or:
www.ieee802.org/3/trunk_study/tutorial/pctrunk.pdf

Some of our own products are a lot more flexible than that and you
can set any arbitrary bits in a packet header (within the first
64 bytes) as the selectors on which links gets the packet.

Again my main point is Forces doesnt need to know anything about these
details. Its a transport/communication management issue.

>    OS where they can freely modify the transport proto impl.

Definetely true if you hack at the TCP level.
I was more thinking of a _ForCES_ write() not TCP i.e like i said
earlier - with a connection created by socket(PF_FORCES,...) 
But even a simple abstraction at user space before calling a
forces_write(setprioto4, ,...) should hide all that.

>    SCTP at least the reordering issue could be circumvented.

True this will require probably multi-queues at the send socket level
to work effectively. PF_FORCES will have to take care of this.
But even with single queues hacks are know to exist to prioritize
data (and jump queues)- example is usage ot TCP URG flag. I dont
think a single socket queue is sufficient.

> deserve differential treatment.  :)

Ok, agreed. I do think that if you have x physical links
each with different priority that infact you have a good solid
(although more expensive) setup. 
What bothers me is any mandating of one, two, three, .. eight
such channels.
So far i have NOT heard a good reason.
Again, I dont disagree that 2 links is better than one.
or eight for that matter is better than 2. 
My problem is somehow that Forces needs to know this. 
What bothers me is any mandating of one, two, three, .. eight
such channels.
So far i have NOT heard a good reason why. I claim its an 
implementation/config issue (which is orthogonal to Forces).

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: 13 Mar 2004 12:43:00 -0500

Hi Avri,

> will you have some other mechanism?

I think it is useful for whatever is managing the connections to know
when a link goes down. These typically (should) happen via event
notifications.
The fallback (albeit slower way) is to depend on some control data or
hearbeats.

Not sure if that answers it.

cheers,
jamal