Subject: commented archive
From: Rainer Gerhards <[email-masked]>
Date: Tue, 30 Dec 2003 17:39:57 +0100

Hi WG,

I have created an (partial) archive of WG postings. Other than the full,
official one, this is manually structured according the contents of the
messages (at least as far as *I* think what content they belong to).

I had the idea to do this when I began to track the issues for
syslog-protocol. I have to admit I do not know if it is a good idea or
not. Also, I do not know if I can manage to keep it updated once the
message volume on the list increases. Let's see. If you find it useful
(or not useful;)) or have any other comments - please email my
privately. I would be very interested if you think this is a valuable
effort (most importantly because if all find it is totally useless, I
will probably spend the time somewhere else ;)).

The archive is at

    http://www.syslog.cc/ietf/

I would also like to whish all of you a happy and healthy new year!

Rainer

------------------------------

Date: Tue, 6 Jan 2004 15:37:23 -0800 (PST)
From: Chris Lonvick <[email-masked]>
Subject: syslog-protocol Discussion

Hi Folks,

I trust that everyone has had a good rest and is ready to get things going
again for the new year.  :-)

I'd like to keep us focused on syslog-protocol for the moment.

  http://www.ietf.org/internet-drafts/draft-ietf-syslog-protocol-00.txt

Rainer has proposed adding "metadata" as a field in the protocol (formerly
called "cookies") for specific information to be included in the syslog
packet.  There has been some discussion about this:

  http://www.mail-archive.com/syslog-sec%40employees.org/msg01346.html

with many follow-up responses.

It looks like the consensus is

- - the idea may be a good one.
- - the format may need some tweaking.
- - saying that it "looks" like XML is a bad thing.

I'm going to ask Rainer to add this idea into the syslog-protocol ID at
this time with inputs from the Working Group.  I'd like for this
discussion to continue on the mailing list as well.  Hopefully we can
resolve things on the mailing list.  I'm also going to call for a meeting
in Seoul to give us a deadline for this discussion.  It looks like
syslog-protocol is fairly well finished with the exception of this item
and the TAG issue raised by Albert.  I'd like to have everything resolved
on this just after Seoul (or sooner if we can do it on the mailing list)
so we can put this into WG Last Call rsn and get it to the IESG.

Thanks,
Chris

------------------------------

Date: Thu, 8 Jan 2004 12:04:55 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol: Cookie Format

Hello Anton,

first of all, my apologies for the late reply. I wanted to think a
little more about your message and then the vacation season got into my
way. The plus is that I have some additional references to post.

So now let's finally begin with the answer (I am not removing any of
your text so that it is probably better to get the idea after such a
long time...):

> Rainer:
>
> Sorry if I did not follow earlier discussion and misunderstand
> something obvious. But here is my feedback.
>
> I like the key=value approach. A lot! This would be hugely welcome in
> syslog and this is a great undertaking! To begin with, we could plug
> things like severity and facility because, otherwise, they don't even
> get stored for standard UDP syslog messages! I have dozens of
> practical examples of other things that could be put into these tags.

Yes, I think this is one of the main benefits of the new
"cookie"/metadata format. It allows for many extensions and fits them in
very nicely. Also, non-metadata-aware (and older) syslogds can simply
store the metadata as part of the message content.

> I am not clear why it has to be enclosed specifically in XML-like tags
> though. See what we did below for an alternative.

The orginal idea was that it is relatively easy to get an XML parser AND
XML already has thought-out, practically prooven things like escape
sequences for characters which are used for special purposes (e.g. &gt;
for ">"). However, using XML raises some other questions. I will
actually raise them in a soon-to-be-written reply to chris, because I
think his message was highly focussed just on this.

>
> If "/>" is a special sequence, are there any rules for escaping it if
> one needs to put it in tag values?

"/>" denotes the end of a tag that is not a container. So you may say
"<tag p="aaa"></tag>" or just "<tag p="aaa" />". I am not 100% sure if
the SP before "/" is needed, I don't think so.

> I am also not clear on the use of word "cookie" in general. Can you
> explain where does this name concept come from?  The only association
> that comes to my mind is with web browser cookies, and I don't see the
> connection.  Can we call these key-value pairs "tags" or something
> else?

The term "cookie" stems back to syslog-sign. As it looks now, "metadata"
seems to be an accepted name change (but few comments so far). I am
tracking this issue at http://www.syslog.cc/ietf/protocol/issue3.html
(will receive an update with the new messages soon).

> At Cisco, we are evangelizing a new logging specification for logging
> to file or through syslog. It is called CiscoLog.  It specifies
> optional use of tags. We are looking at delimiting individual tags
> with [] and the tag section with ": ".  Here is an example of a
> message with tags as seen on the wire, but without the syslog PRI
> field:
>
> 12: host.cisco.com: *Jun 13 2003 23:11:52.454 UTC:
> %BACC-4-BAD_REQUEST: RDU:
> [comp=parser][mac=1,6,aa:bb:cc:11:22:33][txn=mytxn123]: Bad request
> received from device [1,6,aa:bb:cc:11:22:33]. Header missing.
>
> Things in square brackets after "RDU: " are tags.  We don't follow
> some of the optional syslog RFC3164 time and other header formats
> partly for legacy reasons and also largely due to the fact that syslog
> server implementations vary in what they store (or how they forward)
> messages while we need a minimum set of data to always be present in
> the log file and in a consistent format. I can give more details about
> our format if needed. I am allowed to disclose it.
>
> It is also useful to differentiate tag syntax from tag meaning.  We
> addressed this with the use of "semantic extensions".  For example,
> tag with key "ip" just means that the value is a properly formatted IP
> related to message.  Tag "ip.orig" means it is IP of host originating
> message.  Tag "ip.client" means it is a tag from a client talking to
> server which originated the message.  You can also see things like
> "ip.from", "ip.to".  Would it be useful to make the syntax/semantic
> distinction in the standard?

This paragraph was one of the main raesons for me not replying
immediately. I thought quite a bit about it. On my initial look, I
thought, yes, this is header data that needs to be specified in the
protocol itself. On second thought, I think it is too verbose to be
actually part of the header. A good indication is that it is optional.
Don't misunderstand me: I think it is an excellent idea - I just think
it is too "upper layer" to be included into -protocol.

In fact, I think this addresses a payload issue, which is currently
outside the scope of the WG (but may come into the scope ones the other
issues are finalized, at least Chris has already allowed on-list
discussion, so I think there is a chance... ;)). I am a strong advocate
for some payload specification (which can also solve some of the "how to
store it" questions around [I think Albert asked, but I am not sure]).
However, I think we really need to finish the lower layers first. Once
this is done, we can focus (and have ressources!) to look at the
payload.

Anyhow, I would be very interested to read your format specification. I
would appreciate if you could post it. Not only for the payload part but
also on ideas that could/should be used in -protocol. One question I
have (for example) is if you have escapes defined for the special
characters ('['/']'/':'].

>
> BTW, we also use tags to mark parts of a multi-part messages.  So,
> there is a lot of uses for them.
>
> >     <cookie MSGNO=234 VENDOR=example vendorparam=name%nnn
> > /> msg.msg.msg
>
> Are spaces allowed for values?  They should be, I think.

Actually, this sample was done to quickly. The message orginally sent by
Chris was just a rough idea and I should have better edited it while
quoting. You need to have parantheses around all values. So the sample
would actually be

    <cookie MSGNO="234" VENDOR="example vendor" param="name%nnn"
/>msg.msg.msg

... and your question had probably never appeared ;)

> Why a different syntax for experimental tags?

I thought to include a class for cross-vendors tags that are "born" in
IETF and other non-vendor specific places. People feel strong about
names. So I thought it doesn't hurt to allow for a class which is neiter
standardized nor vendor specific. I had the following picture on my
mind: On an IETF list, vendeorA, vendorB and vendorC have a bright idea
and create a new cookie to implement it. Technically, everything looks
bright. But then begins the political issue of wheter it should be
placed in the vendorA, vendorB or vendorC namespace... maybe this
expresses what I mean ;)

I still have the feeling that a special category for such
cookies/metadata would at least not hurt. What does the rest of the list
mean?

> Getting back to syslog-international, I think tag values should also
> be allowed to be in foreign languages eventually. Preferably UTF-8.
> Tags could be used to provide any easily parseable parameters. If I
> use "username" tag, for example, then in Japan the value could be in
> Japanese.  We had some specific requirements from some Cisco apps to
> support this. We support foreign languages in tag values and
> specifying UTF-8 encoding for entire message (US-ASCII is preserved as
> is).

YES. This is why -protocol does not enforce 7 bit encoding.
- -international is very easy doable with this. UTF-8 is the solution to
everything as soon as 8 bit encoding is allowed. -international will
become a very thin document ;-)

Rainer

>
> Thanks,
> Anton.
>
>
>
> > -----Original Message-----
> > From: [email-masked]
> > [mailto:[email-masked]]On Behalf Of Rainer Gerhards
> > Sent: Wednesday, December 17, 2003 12:55 PM
> > To: [email-masked]
> > Subject: syslog-protocol: Cookie Format
> >
> >
> > Hi WG,
> >
> > Chris has proposed a XLM-like cookie format in private
> > mail. I received
> > permission to quote him on this. I would appreciate any
> > feedback from
> > the WG.
> >
> > Chris said (Chris, please correct me if you feel I am quoting out of
> > context):
> >
> > ####
> > The thought just struck me about the use of cookies, languages, etc.
> > The
> > use of the special characters is what John Kelsey proposed.
> >  We're not
> > limited to that if you can come up with something better.  I was
> > thinking
> > of something like the following at the start of the MSG:
> >
> >     <cookie MSGNO=123 ENCODING=USASCII /> msg.msg.msg
> > or
> >     <cookie MSGNO=234 VENDOR=example vendorparam=name%nnn
> > /> msg.msg.msg
> > or even
> >     <cookie block=certblock /> certificate.block.msg
> >
> > syslog-protocol would then define the IANA held cookie
> > parameters and
> > vendors would be able to add their own.  Experimental
> > parameters could
> > be
> > done like "vendorparam" (above) where the "%" replaces the
> > "=" of a real
> > parameter.
> > ####
> >
> > Please note: In this quote, "msg.msg.msg" is a syslog
> > message. It is NOT
> > (necessarily) multiple messages (it may for fragmented messages, but
> > that is a separate issue).
> >
> > A key point in Chris suggestion is that this is NOT actual
> > XML. It just
> > looks so. But we could specify the exact sequence of parameters and
> > such, so that no XML parser is needed to process it. After
> > some initial
> > scepticism, I have to say that I like this approach. It
> > looks very clean
> > and extensible.
> >
> > In this mail, I am trying to gather some feedback from the WG if we
> > should move into the proposed direction. If so, I will
> > focus on some of
> > the details.
> >
> > Questions so: Do you like this? Do you think it is useful? Should we
> > proceed into this direction?
> >
> > Rainer
> >
> >
>
>
>

------------------------------

Date: Thu, 8 Jan 2004 12:18:14 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol Discussion

Happy new year to all! I am finally back from an extended holiday season
(January 6th is also a holidy over here) and I fear I will be posting
some responses today... ;)

> Rainer has proposed adding "metadata" as a field in the
> protocol (formerly
> called "cookies")

I count this as a vote for the term "metadata" and will assume this
issue is solved when nobody objects it ;)

> for specific information to be included in
> the syslog
> packet.  There has been some discussion about this:
>
>   http://www.mail-archive.com/syslog-sec%40employees.org/msg01346.html
>
> with many follow-up responses.
>
> It looks like the consensus is
>
> - the idea may be a good one.
> - the format may need some tweaking.
> - saying that it "looks" like XML is a bad thing.
>
> I'm going to ask Rainer to add this idea into the
> syslog-protocol ID at
> this time with inputs from the Working Group.

I agree that some text needs to be written ASAP, but I fear the current
discussion needs to be finished in the regard whether we will actually
use XML or not. I am posting a specific message very soon after this one
- - just to keep the message more focused. I think it is vitally important
to finish that part of the discussion because without that, no good text
can be written - not even a "draft for a draft" simply because I do not
know where we are actually heading to... Read on in the next message ;)

> I'd like for this
> discussion to continue on the mailing list as well.  Hopefully we can
> resolve things on the mailing list.  I'm also going to call
> for a meeting
> in Seoul to give us a deadline for this discussion.

I like this "dealine idea" very much. I unfrotunately can not attend
Seoul personally, but I hope we can prepare many things in advance to
make it a good experience..

> It looks like
> syslog-protocol is fairly well finished with the exception of
> this item
> and the TAG issue raised by Albert.

Actually, I think Albert's response was quite good and this issue is
more or less finshed, too. I will post another reply, to it on that
thread.

Rainer

------------------------------

Date: Thu, 8 Jan 2004 12:26:54 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: TAG incompatibility

> From: Albert Mietus [mailto:[email-masked]]
> Sent: Thursday, December 18, 2003 8:44 AM
>
> Rainer Gerhards writes:
>
>  > It is regarding the syslog TAG. The format described was
> discussed in
>  > the context of syslog-sign-13 but now also applies to
>  > syslog-international-00.
>
> You will understand my vote ;-)
>
>  > So I think we need to make a tradeoff decision: we can either
>  > a) allow colons in the path name
>  > xor ;)
>  > b) allow TAG NOT to be terminated by SP
>
> I vote A.
>
>  > Selecting
>  > a) will break compliance for older clients (how many?)
>
> No it will NOT,
>
> Syslog-rfc* is (now) step by step, faraway by beeing 100% complice to
> old implementations e.g. "long timestamp, DNS hostname, etc. This bit
> will not change this.

I fully agree with you as far as the sender's side is concerned. My
concern is that we will break compatibility of the receiver receiving
existing messages from an old-style syslogd. But, you are right, that
old-style syslogd will also break other things. But then, the difference
is that we could make proper exclusions/exceptions so that a newer
syslogd can successfully parse the message generate by an older system.
Now, however, we face the situation that we can not (easily) detect this
and the message may be mis-parsed. I am not sure how often
"tag:msgmsgsg" actually appears in older messages. I would, however tend
to accept this problem. I think it is minor compared to clean design we
get be requiring TAG to be terminated by SP, too.

So I, too, vote for this. Is there any objection against this? If not, I
would like to say this is solved, allowing us to move quickly on with
- -protocol (as Chris suggested).

Rainer

------------------------------

Date: Thu, 8 Jan 2004 10:51:55 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: syslog-protocol Discussion

> > Rainer has proposed adding "metadata" as a field in the protocol
> > (formerly called "cookies")
>
> I count this as a vote for the term "metadata" and will
> assume this issue is solved when nobody objects it ;)

Meta-data is ok, although I would have preferred referring to these as
just "tags" since this part of the message is not necessarily going to
be "data about data".  It could be just a well-structured part of the
entire message and may carry data no different then the rest of the
message other than formatting.

If you go with meta-data, please note that the correct spelling is with
a hyphen.  The word "Metadata" (one word and with no hyphen) appears to
be trademarked.

http://dictionary.reference.com/search?q=metadata

Anton.

------------------------------

Date: Thu, 8 Jan 2004 18:34:45 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol Discussion

> Meta-data is ok, although I would have preferred referring to these as
> just "tags"

I try to stay away from this, as we have a field (even traditionally)
called TAG in the header. I think a second "tag" causes at least light
confusion ;)

> since this part of the message is not necessarily going to
> be "data about data".  It could be just a well-structured part of the
> entire message and may carry data no different then the rest of the
> message other than formatting.
>
> If you go with meta-data, please note that the correct
> spelling is with
> a hyphen.  The word "Metadata" (one word and with no hyphen)
> appears to
> be trademarked.
>
> http://dictionary.reference.com/search?q=metadata

oops... thanks for spotting this!

Rainer

------------------------------

Date: Thu, 8 Jan 2004 18:31:41 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol: Cookie Format

another message after a too-long vacation season ;) Because I had lots
of time to think, this is a LONG message. It's the reply PLUS a
suggestion for an -international-00 change in regard to the metadata
(formerly cookie) format. I am outling a specific problem and potential
solution, so please bear with me and actually read it until the end ;)

> > I see the argument. Do you think that relying on something already
> > accepted - like BEEP XML in RFC3080 is a solution? As far
> as I can see,
> > RFC3080 is increasingly becoming implemented. So while not
> full XML, it
> > may be a standard that works "as expected"?
>
> I believe that RFC3080 specifies the use of actual XML, not any
> simplified version. It does insist on XML 1.0, and allows dropping of
> the <? ...> headers at the start (since you're not passing an
> independent document but rather an XML fragment inside
> another protocol,
> and that protocol tells you that you're getting XML there).
> But in 2 of
> the 3 implementations I've seen the innards of, it's using
> off-the-shelf
> XML parsers. (And the reason the third one didn't is we didn't want to
> make it dependent on outside libraries, allowing porting to (say)
> routers or hand-held PDAs or whatever).
>
> However, the examples I've seen such as
>      <cookie MSGNO=123 ENCODING=USASCII />
> are not valid XML tags, missing the quote marks around
> attribute values.

I agree. The samples were not meant to serve as a formal spec, just to
get the rough idea. They were good in the informal mail exchange I had
with Chris. I should have edited them before posting here. So the sample
should actually have read:

 <cookie msgno="123" encoding="USASCII" />

>
> Rather than "simplified XML", consider "restricted XML",
> which is what I
> think you're talking about wrt RFC3080. But this is a restriction on
> allowable DTDs, not a simplification on the syntax.

Yes, this is more to what I really meant. I think that if we move
towards XML for the metadata (formerly known as cookie), then we should
supply a DTD. This DTD probably should build on the DTD for BEEP XML as
found in RFC3080, section 7.

If we take this route, this probably means that we also need to include
the ability to create actual containers, things like (line breaks for
readability, only)

<cookie type="whatever">
  <subtag param="value" />
  <subtag param="value> </subtag>
</cookie>

I think we can't call it XML without this. But I have to admit I am not
deep enough in XML to know if it is possible to define a DTD that
strictly requires no-containers, only.

Also, I think we have an issue with properly XMLizing a full message.
Fortunately, we can skip the header as this is not part of the content
(and XML metadata/cookies) should only be used inside the MSG part.
Let's look at a plain example message:

"I am a message"

according to -international-00, it would be as follows (with an example
cookie):

"@#V-example.com-MyTag I am a message"

as far as -international-00 PLUS current discussion goes, this would end
up being (I stripped out some non-important parts:

"<cookie ....>I am a message"

the important things is what is NOT shown: "</cookie>". To be proper
XML, it would need to be

"<cookie ....>I am a message</cookie>"

Simply adding a "closing slash" to the initial tag wouldn't help. I
think this is still not correct XML ("I am a message" is outside any
container):

"<cookie .... />I am a message"

So if we try to use XML based cookies/metadata, we need

a) make sure that the full MSG is XML compliant
b) create a specific portion inside MSG where XML is expected to live
(and only there)

I think the first approach is not desirable here, as it would introduce
a lot of compatiblity issues. In fact, I think it is close to undoable
in a "clean" way. So option b) seem to be the choice ... and I think it
is even a good choice, one that is matching the (simple) spirit syslog.
A quick look at the ABNF as it currently is:

      SYSLOG-MSG      = PRI HEADER MSG [TRAILER]
      MSG             = (COOKIE SP [COOKIE-PARAMS SP] MSG) / PAYLOAD
      PAYLOAD         = *((%d32-126) / (%d128-254))

We could modify it to this:

      SYSLOG-MSG      = PRI HEADER MSG [TRAILER]
      MSG             = [METADATA SP] / PAYLOAD
      PAYLOAD         = *((%d32-126) / (%d128-254))
      METADATA        = "<metadata" 1*PRINTASCII "/>"
                        ; "BEEP XML" stream according
                        ; to DTD (to be defined)
                        ; MUST start with "<metadata"

In this ABNF, I have removed the recursive cookie structure. This in
itself is good news, because recursion tends to be misunderstood.
Instead, I have specified a specific (SP terminated) field holding the
metadata (cookies) in BEEP XML format. That structure, of course, can be
a tree, so there is some recursion again. The METADATA part must start
with "<metadata" simply to detect that it is present. I think this poses
no problem to existing technology (else we may simply think of inserting
another sequence just in front of it).

I have gone through many scenarios and think this approach works well
for most if not all. I am still a bit concerned about fragmented
messages with a lot of metadata - which means that the metadata needs to
be fragmented, too. Aynhow, I thought I post this before finally
thinking it out so that I can gather the WG's feedback (I may be totally
on the wrong path...).

As a side note, structured payload (as Anton has briefly described a
Cisco sample in
http://www.mail-archive.com/syslog-sec%40employees.org/msg01347.html)
could also be transported inside either the metadata XML stream <payload
/> container OR in the traditional PAYLOAD region. I would opt for the
metadata stream, but it is too early to discuss this.

All in all, I think the metadata field, and using restricted XML inside
it, is probably a good idea to meat a number of goals. It has, however,
one big drawback. And that is simplicity and acceptance. David has
provided very good thoughts on market acceptence in his post
http://www.mail-archive.com/syslog-sec%40employees.org/msg01357.html.
While the topic of this post was BEEP, the ideas behind it are general
enough to apply to this discussion. Just to make sure: we are NOT
talking about even mentioning RFC 3080 in -protocol. So our issue is not
the acceptance of BEEP but the acceptance of XML (but in a restricted
but well-formed form [DTD]).

Fortunately, XML is very well accepted. However, this may be a little
different when it comes to syslog vendors/implementors. I am not sure if
this actually is an issue. Most importantly, nobody is forced to use
XML, but you are kind of forced to use it when you would like to use new
features... which in turn could lead to these features not being
(poperly) implemented. If I got the spirit of David's message correct,
he righly said that "It doesn't help if (just) the WG is happy with the
standards. Implementors (and thus vendors) must be happy and see benefit
in them - otherwise the standard won't be implemented.".

On the pro side for XML inside -protocol is that a syslog client who
wants to emit a message - even an XML enhanced one - does not
necessarily need to have an XML parser. I assume that most, if not all,
of our XML usage only requires very basic formatting, so this can be
written with a simple snprintf() - to say it programatically.
Non-technically said, it is more or less the same effort as it was for
the syslog-sign-13 cookies.

I would appreciate if we could try to solve those both issue before I
edit the -protocol draft, simply because it will greatly influence the
actual edit. For a short summary, this mail opens two questions:

#1 Are we ready for moving XML into a core piece of the syslog spec (the
later part of my message)?

#2 If so, is the format described in the first part of my message
acceptable?

I appreciate all comments. I would especially encourage Albert to
comment, as he already implemented -sign and thus may have some good
insight to share (especially on #1).

Sorry for the long mail...

Rainer

------------------------------

Date: Thu, 8 Jan 2004 13:12:06 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: syslog-protocol Discussion

> > Meta-data is ok, although I would have preferred referring
> to these as
> > just "tags"
>
> I try to stay away from this, as we have a field (even
> traditionally) called TAG in the header. I think a second
> "tag" causes at least light confusion ;)

This makes sense. And I don't have a better alternative then meta-data.

Anton.

------------------------------

Date: Wed, 14 Jan 2004 07:07:44 -0800 (PST)
From: Chris Lonvick <[email-masked]>
Subject: Meeting In Seoul

Hi Folks,

I've asked the Agenda people and our ADs (Russ Housley and Steve Bellovin)
for a 2 hour time slot at the 59th IETF.  I'm going to propose the
following agenda.  Your comments on that would be welcome.

I.       Review of Scope and Charter (Chris)                  10 min.
II.      Update of "syslog Protocol" ID (Rainer or proxy)     45 min.
III.     Update of "syslog-sign" ID (Jon or proxy)            15 min.
IV.      Update of "syslog-mib" ID (Glenn)                    30 min.
V.       Wrap-up and review of decisions made (Chris)         10 min.

Since the NetConf people have expressed an interest in making changes when
we update RFC 3195, I'll ask if they would like to be given an opportunity
to present their thoughts.  If so, I'll adjust the agenda.

I'd like for us to focus on the "meta-data" being proposed by Rainer in
syslog-protocol.  Once we get consensus on that, we can submit it to the
IESG.  Once that is defined, Jon can make the necessary revisions to
syslog-sign and we can submit that to the IESG as well.  If we can get
consensus on the mailing list before the meeting then I'll revise the
agenda to include 10 minutes of Group Hug.  :-)

Thanks,
Chris

------------------------------

Date: Wed, 14 Jan 2004 17:05:34 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: Thoughts on Meta-Data

Resend, as there apparently where some server issues...

Below some thoughts...

> Use strict XML such as:
>    <cookie msgno="123" encoding="USASCII" />
>
> Use loose rules so the information can be easily converted to
> XML (if one wishes) such as:
>   <cookie msgno=123 encoding=USASCII>
>
> Define the format and delimiters with little regard to any
> conversion to XML, such as:
>   [cookie msgno=123 encoding=USASCII]

My preference is for this last format.  I think [] brackets make it
clearer it is not XML and visually look more like field separators and
not part of a value (at least to me).

I think Rainer needs to provide more details about the appropriate
character-set/encoding for tag names (probably printable USASCII 32-126
range), and character-set/encoding (probably UTF8) and escaping rules
for tag values.  I think rather than having to escape a space character
in the values, I'd prefer quotes around the value like in example #1 and
define an escape sequence "\" for quote characters. This way we also
don't have to specify escape sequence for the "]" character.  Here is an
example:

[meta msgno="123" username="abra\"kadabra"]

Also, are we going to use "meta-data" or "meta" instead of the "cookie"
like in above examples?  Personally, I'd perefer if we could come up
with a away to use neither such that the placement of the tags and their
general format implied that this is the meta-data portion.  Similar to
how we don't say what the other syslog fields are -- we just provide the
value.

Another alternative for encoding tags would then be something like
[msgno=123][msgpart=2]... In this example, no quotes are needed, but "]"
character would need an escape sequence if it appears in the tag value.

We also need to specify rules or outline lack of preference for:

A) Order of tags. Alphanumeric or any? I am leaning towards allowing any
order in case vendors want to order them a certain way for better visual
display or in order to communicate some precendece.

B) Does order of tags have to preserved on relay? I assume so.

C) Are duplicate tags allowed?  I think they should be for handling
multi-values tags like IP?

> 1) What attributes are going to be defined in syslog-protocol.

I think only tags that are strictly necessary for the syslog-protocol
itself should be defined there.  If there will be no tag definitions at
all in syslog-protocol, it is fine with me.

> 2) How can others define additional attributes in the future.

I see the point about potential vendor conflicts, but vendor extensions
are cumbersome IMO.  Maybe we leave it upto the "marketplace" or future
standards to sort out.  One potential scheme could be a "vendor" tag we
could standardize.  This way, a parser familiar with the specific tags
of a given vendor can interperet them according to the specific rules of
that vendor. But each vendor can still come up with a tag "ip" and use a
different syntax for it if they so choose. At least for now.  Not sure
if "vendor" tag should be required though.

There is a lot more work that can be done in standardizing tags, but
maybe it is not necesssary in the initial standard.  I think it would be
nice to outline rules for representing different things, but yet allow
vendors to add semantic extensions.  For example, we define the syntax
for "ip", but vendors can then define tags like "ip.from", "ip.to",
etc...  I think this is too much of a scope for now though.

Thanks,
Anton.

------------------------------

Date: Wed, 14 Jan 2004 07:53:49 -0800 (PST)
From: Chris Lonvick <[email-masked]>
Subject: Thoughts on Meta-Data

Hi Folks,

(Just stirring the pot; please respond and keep this discussion going. :)

I've seen the following proposals for the Meta-Data field:

Use strict XML such as:
   <cookie msgno="123" encoding="USASCII" />

Use loose rules so the information can be easily converted to XML (if
one wishes) such as:
  <cookie msgno=123 encoding=USASCII>

Define the format and delimiters with little regard to any conversion to
XML, such as:
  [cookie msgno=123 encoding=USASCII]

Please take the time to state your preference and your reasons.

We also need to have a bit of a discussion on "How are the meta-data
attributes defined?".  We need to have two things nailed down in this area
before we can progress syslog-protocol.
1) What attributes are going to be defined in syslog-protocol.
2) How can others define additional attributes in the future.

In (1), Rainer has given some examples in his notes.  This group will need
to agree upon them and get them into the syslog-protocol document.

In (2), we will need to define the name-space and give instructions to
IANA on what has been defined, and how new attributes will be defined and
standardized in the future.  One method is to force non-IETF attributes to
prepend attribute names with a known tag, such as "vend-abcdef" for
"vendor defined" attributes.  Another possibility is to offer no such
guidance.  This would mean that all non-IETF-defined attributes are free
to be used by anyone in any manner.  Speaking personally, this has a
certain attraction since syslog is historically a free-form text format.
:-)  This could however cause some problems in the future.  For example,
Group A could start using an attribute of "counter-1" with a starting
value of "0".  Group B could independently use an attribute of the same
name with a starting value of "1".  The two groups would then have to come
to some agreement and perhaps write an RFC to define the attribute and the
starting value.

Please send your thoughts to the list on these things.

Thanks,
Chris

------------------------------

Date: Wed, 14 Jan 2004 12:31:07 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: Thoughts on Meta-Data

Below some thoughts...

> Use strict XML such as:
>    <cookie msgno="123" encoding="USASCII" />
>
> Use loose rules so the information can be easily converted to
> XML (if one wishes) such as:
>   <cookie msgno=123 encoding=USASCII>
>
> Define the format and delimiters with little regard to any
> conversion to XML, such as:
>   [cookie msgno=123 encoding=USASCII]

My preference is for this last format.  I think [] brackets make it
clearer it is not XML and visually look more like field separators and
not part of a value (at least to me).

I think Rainer needs to provide more details about the appropriate
character-set/encoding for tag names (probably printable USASCII 32-126
range), and character-set/encoding (probably UTF8) and escaping rules
for tag values.  I think rather than having to escape a space character
in the values, I'd prefer quotes around the value like in example #1 and
define an escape sequence "\" for quote characters. This way we also
don't have to specify escape sequence for the "]" character.  Here is an
example:

[meta msgno="123" username="abra\"kadabra"]

Also, are we going to use "meta-data" or "meta" instead of the "cookie"
like in above examples?  Personally, I'd perefer if we could come up
with a away to use neither such that the placement of the tags and their
general format implied that this is the meta-data portion.  Similar to
how we don't say what the other syslog fields are -- we just provide the
value.

Another alternative for encoding tags would then be something like
[msgno=123][msgpart=2]... In this example, no quotes are needed, but "]"
character would need an escape sequence if it appears in the tag value.

We also need to specify rules or outline lack of preference for:

A) Order of tags. Alphanumeric or any? I am leaning towards allowing any
order in case vendors want to order them a certain way for better visual
display or in order to communicate some precendece.

B) Does order of tags have to preserved on relay? I assume so.

C) Are duplicate tags allowed?  I think they should be for handling
multi-values tags like IP?

> 1) What attributes are going to be defined in syslog-protocol.

I think only tags that are strictly necessary for the syslog-protocol
itself should be defined there.  If there will be no tag definitions at
all in syslog-protocol, it is fine with me.

> 2) How can others define additional attributes in the future.

I see the point about potential vendor conflicts, but vendor extensions
are cumbersome IMO.  Maybe we leave it upto the "marketplace" or future
standards to sort out.  One potential scheme could be a "vendor" tag we
could standardize.  This way, a parser familiar with the specific tags
of a given vendor can interperet them according to the specific rules of
that vendor. But each vendor can still come up with a tag "ip" and use a
different syntax for it if they so choose. At least for now.  Not sure
if "vendor" tag should be required though.

There is a lot more work that can be done in standardizing tags, but
maybe it is not necesssary in the initial standard.  I think it would be
nice to outline rules for representing different things, but yet allow
vendors to add semantic extensions.  For example, we define the syntax
for "ip", but vendors can then define tags like "ip.from", "ip.to",
etc...  I think this is too much of a scope for now though.

Thanks,
Anton.

------------------------------

Date: Thu, 15 Jan 2004 17:02:31 +1300
From: Andrew Ross <[email-masked]>
Subject: RE: Thoughts on Meta-Data

Thanks for the ideas Anton,

I'm happy with using the word cookie. It works for me. :-)

To me, there doesn't seem to be a good reason to lean towards using XML
or pseudo XML. I would prefer to avoid any possibility of nested tags
etc. We could use [] or <> for tagging, just something to show opening
and closing would be good. Escape characters need to be defined.

I also like the idea of a vendor tag. It would make for simple parsing
and identifying of log files. Maybe a vendor tag and a sub vendor tag.
i.e. [vendor=Cisco] [vendor.sub=pix6.3(1)]

If we allow duplicate tags, then we need to discuss which tag takes
priority. Is the last tag or first tag the one we use?

What about tag naming conventions? Do we use VendorSub or vendor_sub or
Vendor_Sub or vendor.sub?

If all tags are US ASCII then we can convert the case easily for string
matches. If they are Unicode and not easily convertible to US ASCII then
how to we convert case?

Cheers

Andrew

------------------------------

Date: Thu, 15 Jan 2004 12:15:57 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: Thoughts on Meta-Data

> Below some thoughts...
>
> > Use strict XML such as:
> >    <cookie msgno="123" encoding="USASCII" />
> >
> > Use loose rules so the information can be easily converted to
> > XML (if one wishes) such as:
> >   <cookie msgno=123 encoding=USASCII>
> >
> > Define the format and delimiters with little regard to any
> > conversion to XML, such as:
> >   [cookie msgno=123 encoding=USASCII]
>
> My preference is for this last format.  I think [] brackets make it
> clearer it is not XML and visually look more like field separators and
> not part of a value (at least to me).
>
> I think Rainer needs to provide more details about the appropriate
> character-set/encoding for tag names (probably printable
> USASCII 32-126
> range), and character-set/encoding (probably UTF8) and escaping rules
> for tag values.  I think rather than having to escape a space
> character
> in the values, I'd prefer quotes around the value like in
> example #1 and
> define an escape sequence "\" for quote characters. This way we also
> don't have to specify escape sequence for the "]" character.
> Here is an
> example:
>
> [meta msgno="123" username="abra\"kadabra"]

I have the feeling that Anton's approach is better than using XML. I
still fear that even the word "XML" will make some implementors stay
away from it.

So, if nobody objects today, I will edit -protocol-00 to -01 tomorrow in
this spirit. I will then post -01 - if we don't like it, it is easy to
change it in -02 ;)

There is one thing, though. Even though I like the [...] syntax, I think
we can't just use plain '[' because this is already in widespread use. I
think that would totally confuse things. As such, I will use "[@#....]".
So it starts with "[@#" to make sure it is a -protocol meta-data item.

>
> Also, are we going to use "meta-data" or "meta" instead of
> the "cookie"
> like in above examples?

based on Anton's comment of th meta-data not actually being meta-data. I
evenutally have a better term. What if we simply call this "structured
data"?

> Personally, I'd perefer if we could come up
> with a away to use neither such that the placement of the
> tags and their
> general format implied that this is the meta-data portion.  Similar to
> how we don't say what the other syslog fields are -- we just
> provide the
> value.

I agree placement-ignorant positioning should be doable with the "[@#"
approach. I guess I will see minor issues when I actually edit the draft
;)

>
> Another alternative for encoding tags would then be something like
> [msgno=123][msgpart=2]... In this example, no quotes are
> needed, but "]"
> character would need an escape sequence if it appears in the
> tag value.

I think the other one is better.

>
> We also need to specify rules or outline lack of preference for:
>
> A) Order of tags. Alphanumeric or any? I am leaning towards
> allowing any
> order in case vendors want to order them a certain way for
> better visual
> display or in order to communicate some precendece.

My vote is any order

>
> B) Does order of tags have to preserved on relay? I assume so.

YES

>
> C) Are duplicate tags allowed?  I think they should be for handling
> multi-values tags like IP?

That should be described on a tag-by-tag basis.

>
> > 1) What attributes are going to be defined in syslog-protocol.

Few, actually I think just fragmentation bits. We could also add some
optional bits to reflect RFC3195 cooked information. But this may be
beyond scope (I will not do that in the -01 edit).

>
> I think only tags that are strictly necessary for the syslog-protocol
> itself should be defined there.  If there will be no tag
> definitions at
> all in syslog-protocol, it is fine with me.
>
> > 2) How can others define additional attributes in the future.
>
> I see the point about potential vendor conflicts, but vendor
> extensions
> are cumbersome IMO.  Maybe we leave it upto the "marketplace"
> or future
> standards to sort out.  One potential scheme could be a
> "vendor" tag we
> could standardize.  This way, a parser familiar with the specific tags
> of a given vendor can interperet them according to the
> specific rules of
> that vendor. But each vendor can still come up with a tag
> "ip" and use a
> different syntax for it if they so choose. At least for now.  Not sure
> if "vendor" tag should be required though.

Need to think about this. Eventually I make I proposal right in -01 ...
as I wrote, it is easy to change it in -02. It seems harder to discuss
this without having any text to look at.

>
> There is a lot more work that can be done in standardizing tags, but
> maybe it is not necesssary in the initial standard.  I think
> it would be
> nice to outline rules for representing different things, but yet allow
> vendors to add semantic extensions.  For example, we define the syntax
> for "ip", but vendors can then define tags like "ip.from", "ip.to",
> etc...  I think this is too much of a scope for now though.

Agree ... sounds like payload to me ;)

Rainer

------------------------------

Date: Thu, 15 Jan 2004 12:39:03 -0500
From: "Harrington, David" <[email-masked]>
Subject: RE: Thoughts on Meta-Data

Hi,

I think there could be benefit to using a strict XML format, but there
are also tradeoffs.

Pro:
Compatibility with other protocols for doing manager-device
communications. The Netconf WG is developing an XML-based approach to
configuring devices; using a format that is compatible with netconf
helps tie the information in netconf to the information in syslog. There
are a number of efforts underway that translate SNMP data to XML and
vice-versa.
Compatibility with management applications that utilize XML. There is a
strong trend toward using XML to allow network management applications
and other applications to share data using XML.
Compatibility with applications (management and otherwise) that work in
a web-services environment.
A range of XML toolkits are available, so standardizing parsing is
easier.
XML Parsing engines can check that the message is well-formed.
XML has rules for schema definition, internationalization, localization,
etc.
Using XML means syslog can utilize additional XML features and tools as
they become available, without having to reinvent each feature for
syslog, or redevelop each tool to work with a unique syslog format.
XML tools are available for automating translation between XML and other
formats.

Con:
XML is not as easily-readable by a human as plain text.
Syslog design will be constrained by XML usage rules/guidelines. This
can be vewed as a positive for standardization, but may be considered an
unnecessary annoyance by some.
XML can be wordy when done right, so the maximum size of the syslog
message may be a constraining factor for using XML.
XML has a certain amount of overhead that would use up some of the
available syslog message size.
A standardized XML schema would potentially limit what information could
be passed in a syslog entry.

he primary purpose of syslog is to provide human-readable information

Observations:
if you're embedding XML messages inside another format, only the XML
will be easy to use with other applications.
If the syslog language is well-formed, a translator tool could be
created to translate between syslog format and XML format to gain many
of the benefits of XML format, but this could be a bit cumbersome.

My conclusion:
Using well-formed XML could be a really good idea for improving the
standardization of messages and for improving the ability to use the
information programmatically, which benefits network management in
general. Since syslog primarily serves for local troubleshooting by
providing human-readable log messages with implementation-specific
details, internationalization and application-data-sharing may not be
important goals for syslog.

If it can be shown that XML will not force unnecessary iverhead, and
that the syslog community does want to use syslog for programmatic
management, then XML may be justified.

I think using XML may not be justified since the promised benefits are
not necessaily relevant to the traditional goals of syslog messaging,
and XML may force unnecessary overhead in messages.

I think the answer to this question will determine which format is best:
"what use cases do we want to design syslog to best address?"

Dbh
David Harrington
[email-masked]
Director, Network Management Architecture
Office of the CTO, Enterasys Networks

> -----Original Message-----
> From: Chris Lonvick [mailto:[email-masked]]
> Sent: Wednesday, January 14, 2004 7:54 AM
> To: [email-masked]
> Subject: Thoughts on Meta-Data
>
> Hi Folks,
>
> (Just stirring the pot; please respond and keep this
> discussion going. :)
>
> I've seen the following proposals for the Meta-Data field:
>
> Use strict XML such as:
>    <cookie msgno="123" encoding="USASCII" />
>
> Use loose rules so the information can be easily converted to XML (if
> one wishes) such as:
>   <cookie msgno=123 encoding=USASCII>
>
> Define the format and delimiters with little regard to any
> conversion to
> XML, such as:
>   [cookie msgno=123 encoding=USASCII]
>
> Please take the time to state your preference and your reasons.
>
>
> We also need to have a bit of a discussion on "How are the meta-data
> attributes defined?".  We need to have two things nailed down
> in this area
> before we can progress syslog-protocol.
> 1) What attributes are going to be defined in syslog-protocol.
> 2) How can others define additional attributes in the future.
>
> In (1), Rainer has given some examples in his notes.  This
> group will need
> to agree upon them and get them into the syslog-protocol document.
>
> In (2), we will need to define the name-space and give instructions to
> IANA on what has been defined, and how new attributes will be
> defined and
> standardized in the future.  One method is to force non-IETF
> attributes to
> prepend attribute names with a known tag, such as "vend-abcdef" for
> "vendor defined" attributes.  Another possibility is to offer no such
> guidance.  This would mean that all non-IETF-defined
> attributes are free
> to be used by anyone in any manner.  Speaking personally, this has a
> certain attraction since syslog is historically a free-form
> text format.
> :-)  This could however cause some problems in the future.
> For example,
> Group A could start using an attribute of "counter-1" with a starting
> value of "0".  Group B could independently use an attribute
> of the same
> name with a starting value of "1".  The two groups would then
> have to come
> to some agreement and perhaps write an RFC to define the
> attribute and the
> starting value.
>
>
> Please send your thoughts to the list on these things.
>
> Thanks,
> Chris
>
>

------------------------------

Date: Fri, 16 Jan 2004 18:45:46 +0900
From: Glenn Mansfield Keeni <[email-masked]>
Subject: Syslog MIB Issues and Actions

Hi,
   In the following I will be listing up the SyslogMIB issues
that have come up since the last meeting in March at IETF-56.

    #1. Security Considerations need to be addressed.
    #2. Try to make it more generic (not a BSD syslogMIB)
    #3. Ensure non-duplication with HR-MIB
    #4. Snmp Notifications ? Do we need any ?
    #5. Add a "No-DNS lookup"
    #6. Configuration Stuff - should it be there?
    Among other potential issues
    Editorial issues                : None pending
    MIB Doctor Issues               : None pending
    Any Objects to be added         : None so far
    Any Objects to be deleted       : None so far
    Any doubtful DESCRIPZTIONS      : None so far
    Any useful REFERENCE clauses to be added? : None so far

I will be sending separate mails on each of these issues to
kick-off the discussion on the issues. It would be nice if we
could get closure on the outstanding issues before or during
IETF-59

Thanks and Cheers

      Glenn

------------------------------

Date: Fri, 16 Jan 2004 18:46:40 +0900
From: Glenn Mansfield Keeni <[email-masked]>
Subject: SyslogMIB Issue-#1

Issue #1. Security Considerations need to be addressed.

The Securities Considerations scetion should address
matters relative to READONLY access of the MIB as well
as other threats  too.

Action: The present Security Considerations section
contains the text appended below. The above concerns
are addressed. Have me missed anything ?

Status: Waiting on community input

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

7.  Security Considerations

    Syslog plays a very important role in the computer and network
    security of an organization. SyslogMIB defines several managed
    objects that may be used to monitor configure and control syslog
    processes. As such improper manipulation of the objects represented
    by this MIB may lead to an attack on an important component of the
    computer and network security infrastructure.  The objects in
    syslogParamsTable, syslogAllowedHostsTable, syslogCtlSelectionTable,
    syslogCtlLogActionTable, syslogCtlUserActionTable
    syslogCtlFwdActionTable, syslogCtlPipeActionTable  may be
    misconfigured to cause syslog messages to be diverted or lost. A
    misconfiguration may also result in a DoS attack on a user or
    service.  There are a number of management objects defined in this
    MIB module with a MAX-ACCESS clause of read-write and/or read-create.
    Such objects may be considered sensitive or vulnerable in some
    network environments.  The support for SET operations in a non-secure
    environment without proper protection can have a negative effect on
    network operations.  These are the tables and objects and their
    sensitivity/vulnerability:

        o  syslogParamsTable: the objects in this table describe the
           configuration of the syslog processes. The syslogParamsProcessStatus
           may be used to start stop or suspend the syslog process itself.
        o  syslogAllowedHostsTable: the objects in this table describe the hosts
           from which syslog messages will be accepted. Improper configuration may
           lead to loss of messages from an important source or a flood of messages
           from a, potentially rogue, source.
        o  syslogCtlSelectionTable: the objects in this table describe selection
           rules for messages. Improper configuration may lead to loss of relevant
           messages or the collection of useless, potentially ill-intentioned,
           messages.
        o  syslogCtlLogActionTable: the objects in this table describe the actions
           that will be carried on a received syslog message. Misconfiguration may
           lead to loss of important messages or misdirection of messages.
        o  syslogCtlUserActionTable: Objects in this table describe the users that
           will be notified. It may be misconfigured to prevent a user from
           receiving an important message or to spam a user's console.
        o  syslogCtlFwdActionTable: Objects in this table describe the forwarding
           action that will carried out on messages. It may be misconfigured to
           prevent important messages from reaching their destinations or to direct
           a DoS attack on a specific destination. It may also be misconfigured to
           send syslog messages to an improper destination - resulting in a breach
           of user's privacy.
        o  syslogCtlPipeActionTable: objects in this table describe the commands
           that will be invoked to process a log message. This may be misconfigured
           to cause arbitrary programs to be invoked on the syslog receiver.

    Some of the readable objects in this MIB module (i.e., objects with a
    MAX-ACCESS other than not-accessible) may be considered sensitive or
    vulnerable in some network environments.  It is thus important to
    control even GET and/or NOTIFY access to these objects and possibly
    to even encrypt the values of these objects when sending them over
    the network via SNMP.  These are the tables and objects and their
    sensitivity/vulnerability:

        o  syslogProcTable: objects in this table carry sensitive information.  The
           counters may reveal information about the deployment and effectiveness
           of the relevant security systems. The counters may be analyzed to tell
           whether the security systems are able to detect an event or not.

    SNMP versions prior to SNMPv3 did not include adequate security.
    Even if the network itself is secure (for example by using IPSec),
    even then, there is no control as to who on the secure network is
    allowed to access and GET/SET (read/change/create/delete) the objects
    in this MIB module.

    It is RECOMMENDED that implementers consider the security features as
    provided by the SNMPv3 framework (see [RFC3410], section 8),
    including full support for the SNMPv3 cryptographic mechanisms (for
    authentication and privacy).

    Further, deployment of SNMP versions prior to SNMPv3 is NOT
    RECOMMENDED.  Instead, it is RECOMMENDED to deploy SNMPv3 and to
    enable cryptographic security.  It is then a customer/operator
    responsibility to ensure that the SNMP entity giving access to an
    instance of this MIB module is properly configured to give access to
    the objects only to those principals (users) that have legitimate
    rights to indeed GET or SET (change/create/delete) them.

------------------------------

Date: Fri, 16 Jan 2004 18:47:05 +0900
From: Glenn Mansfield Keeni <[email-masked]>
Subject: SyslogMIB Issue-#2

Issue # 2 Try to make it more generic not a BSD-centric

Action: The present document is based on an attempt to design
a generic Syslog-MIB using BSD-syslog as an example. We will
need input  on other mechanisms.
Input like "in XYZ-Syslog we need to monitor ABC" or, in more
severe cases "in XYZ-syslog this MIB cannot be used because
the knobs and dials are totally incompatible".

Status: Waiting on community input

------------------------------

Date: Fri, 16 Jan 2004 14:30:50 -0000
From: Tom Petch <[email-masked]>
Subject: Re: Thoughts on Meta-Data

I have lost sight of the requirements; what is metadata setting out to
achieve that I should evaluate XML as a solution against?

My engineering instinct is always to reuse an existing technology if it is
good enough, not too simple and not too complex for the task in hand.  XML
seems to be being considered on the too complex end of the spectrum but
what is it setting out to do?

Strict subsets of existing technologies are also fine by me but bending it
a bit to suit I am dead against.

And does XML need refining as a label for a technology? is there v1.13 and
rev2.3 etc etc
.
Tom Petch

+(44) 192 575 3018
- -----Original Message-----
From: Chris Lonvick <[email-masked]>
To: [email-masked] <[email-masked]>
Date: 14 January 2004 16:46
Subject: Thoughts on Meta-Data

>Hi Folks,
>
>(Just stirring the pot; please respond and keep this discussion going. :)
>
>I've seen the following proposals for the Meta-Data field:
>
>Use strict XML such as:
>   <cookie msgno="123" encoding="USASCII" />
>
>Use loose rules so the information can be easily converted to XML (if
>one wishes) such as:
>  <cookie msgno=123 encoding=USASCII>
>
>Define the format and delimiters with little regard to any conversion to
>XML, such as:
>  [cookie msgno=123 encoding=USASCII]
>
>Please take the time to state your preference and your reasons.
>
>
>We also need to have a bit of a discussion on "How are the meta-data
>attributes defined?".  We need to have two things nailed down in this area
>before we can progress syslog-protocol.
>1) What attributes are going to be defined in syslog-protocol.
>2) How can others define additional attributes in the future.
>
>In (1), Rainer has given some examples in his notes.  This group will need
>to agree upon them and get them into the syslog-protocol document.
>
>In (2), we will need to define the name-space and give instructions to
>IANA on what has been defined, and how new attributes will be defined and
>standardized in the future.  One method is to force non-IETF attributes to
>prepend attribute names with a known tag, such as "vend-abcdef" for
>"vendor defined" attributes.  Another possibility is to offer no such
>guidance.  This would mean that all non-IETF-defined attributes are free
>to be used by anyone in any manner.  Speaking personally, this has a
>certain attraction since syslog is historically a free-form text format.
>:-)  This could however cause some problems in the future.  For example,
>Group A could start using an attribute of "counter-1" with a starting
>value of "0".  Group B could independently use an attribute of the same
>name with a starting value of "1".  The two groups would then have to come
>to some agreement and perhaps write an RFC to define the attribute and the
>starting value.
>
>
>Please send your thoughts to the list on these things.
>
>Thanks,
>Chris
>

------------------------------

Date: Fri, 16 Jan 2004 18:47:57 +0900
From: Glenn Mansfield Keeni <[email-masked]>
Subject: SyslogMIB Issue-#3

Issue #3. Ensure non-duplication with HR-MIB

Action:
The SyslogMIB MUST not attempt to monitor the generic S/W
process related details (cpu, memory start/stop etc.)

1. provide a mapping from the S/W process in hrSWRunTable
    to the syslog process in hrSWRunTable

2. remove syslogParamsProcessStatus : This will be served by
           syslogParamsProcessStatus

Status: TBD in the next ID.

------------------------------

Date: Fri, 16 Jan 2004 18:48:16 +0900
From: Glenn Mansfield Keeni <[email-masked]>
Subject: SyslogMIB Issue-#4

Issue #4. Snmp Notifications ? Do we need any ?

Action: Do we need any?
         Note- Syslog itself is a notification.
         If there a clear requirement for a notification
         that can added.

Status: Waiting on community input

------------------------------

Date: Fri, 16 Jan 2004 18:48:42 +0900
From: Glenn Mansfield Keeni <[email-masked]>
Subject: SyslogMib Issue-#5

Issue #5. Add a "No-DNS lookup"

Action:  "DoNotUseDNSLookup" will be added to the enumerations
           of syslogParamsDNSLookup.

           syslogParamsDNSLookup OBJECT-TYPE
               SYNTAX      INTEGER {
                           doNotUseDNSLookUp   (1)
                           useLocalCache       (2),
                           doNotUseLocalCache  (3),

                           }
               MAX-ACCESS  read-write
               STATUS      current
               DESCRIPTION
                   "IF doNotUseDNSLookup is on. (syslogParamsDNSLookup = 1
                    DNS lookup will not be carried for addresses encountered.
                    If doNotUseLocalCache is on, fresh DNS lookups will
                    be carried out everytime a hostname is encountered.
                    Else, DNS lookups will be carried out only once for
                    each hostname.
                   "
               DEFVAL { useLocalCache }
               ::= { syslogParamsEntry 9 }

Status: TBD in the next ID

------------------------------

Date: Fri, 16 Jan 2004 18:49:21 +0900
From: Glenn Mansfield Keeni <[email-masked]>
Subject: SyslogMIB Issue-#6

Issue #6. Configuration Stuff - should it be there ?

If the MIB is made read-only we cannot use it for configuration
of syslog processes. That is a disadvantage if we wanted to
configure our syslog processes in the first place. Otherwise,
the consequences are
      - implementations would be simpler
      - security implications are lesser
      - passage through the IESG-review process may be smoother (?)

So, do we want to be able to configure Syslog processes? Community
input is required here.

Status: Waiting on community input

------------------------------

Date: Tue, 20 Jan 2004 11:33:04 -0500
From: "Harrington, David" <[email-masked]>
Subject: RE: SyslogMIB Issue-#4

Hi Glenn,

The benefit of having an SNMP trap is that there becomes some
coordination between the syslog messaging and SNMP trap manager
applications.

Operators often use trap management tools to alert them to problems, and
these applications often have the ability to filter out uninteresting
traps, to sort by priority, to do fault isolation, and so on. Trap
management utilities can also feed into SNMP-based topology mapping
tools (like HPOV) which can use colored icons to represent serious
problems, and so on.

The problem with a syslog notification is - what would it contain? You
could write one syslog notification for reporting all syslog events by
simply having a varbind with the complete syslog message. That means
that an SNMP trap application cannot do much in terms of sorting by
priority. You could create a notification that ocntains some
standardized values common to syslog messages, such as severity, related
subsystem, vendor, and then include the human-readable string in an
octet string. This might be useful, but it might be less useful than
notifications specific to a technology.

I think the WG needs to discuss the nature of potential notification
designs.

Personally I think such a generalized trap might be a good thing for
improving coordination between syslog and SNMP, but it will depend a
great deal on the actual design and how easily that design can be
utilized by applications.

dbh

> -----Original Message-----
> From: Glenn Mansfield Keeni [mailto:[email-masked]]
> Sent: Friday, January 16, 2004 4:48 AM
> To: [email-masked]
> Subject: SyslogMIB Issue-#4
>
>
> Issue #4. Snmp Notifications ? Do we need any ?
>
>
> Action: Do we need any?
>          Note- Syslog itself is a notification.
>          If there a clear requirement for a notification
>          that can added.
>
> Status: Waiting on community input
>
>
>
>
>
>
>
>
>
>
>

------------------------------

Date: Tue, 20 Jan 2004 15:45:06 -0500
From: [email-masked]
Subject: I-D ACTION:draft-ietf-syslog-protocol-01.txt

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Security Issues in Network Event Logging Working Group of the IETF.

	Title		: The syslog Protocol
	Author(s)	: R. Gerhards
	Filename	: draft-ietf-syslog-protocol-01.txt
	Pages		: 40
	Date		: 2004-1-20

This document describes the syslog protocol. The syslog protocol has
been used throughout the years to convey event notifications. This
documents describes a layered architecture for a backwards-compatible
and easily extensible syslog protocol.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-syslog-protocol-01.txt

To remove yourself from the IETF Announcement list, send a message to
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-syslog-protocol-01.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-syslog-protocol-01.txt".

NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

------------------------------

Date: Wed, 21 Jan 2004 09:53:15 +0100
From: Rainer Gerhards <[email-masked]>
Subject: syslog-protocol-01 posted & comments

Hi WG,

as you may have seen, the draft editor has posted protocol-01:

http://www.ietf.org/internet-drafts/draft-ietf-syslog-protocol-01.txt

First things first: this was a "quick" edit (while not as quick as I
hoped... ;)). My main objective was to get out some text as quickly as
possible. There are probably some typos and some other minor
inconsistencies. Also, some of the descriptions may not be as good as
they should be. As the format issue was quite controversal, I try to
save a little bit work by providing ONE POSSIBLE text to handle it. But
further discussion may go into a different direction. So I tried to make
it as understandable as possible while not putting total finishing
efforts into it. Once we have decided the final direction, I will either
revamp totally or polish the current text.

To the content:

#1
As said on the list, I used Anton's non-XML proposal. Weighting all the
arguments received, I really think we do not actually need XML in
syslog, even though syslog is no longer just for human review but also
for automatted processes (this in answer to David's question).

After finishing the text, I am far more convinced that the simple
tagging approach is not only sufficiently enough for transport, but it
also is a good solution for syslog in general. In the long term, it can
also be used to define payload dictionaries, which may be very useful
(should we manage to do this;)).

Regarding integrating this into XML-based systems, I think a mapping of
what I described now and XML is fairly easy, at least as long as you
assume that the message is orginally generated by a syslog device and
the brought over to some XML system.

#2
I would also like to drag your attention to the fragmentation that I now
described. There are some specific implications in regard to
syslog-sign. I would appreciate if those deeply involved in -sign could
cross-check that this format could actually do the job for -sign.

#3
The section on fragmentation is currently missing a recommendation to
use a reliable transport. This will be moved in once we have a general
concensus.

I appreciate comments on these points as well as -protocol-01 in
general. I am prepared to do a quick re-edit.

Thanks,
Rainer

------------------------------

Date: Wed, 21 Jan 2004 07:22:42 -0800 (PST)
From: Chris Lonvick <[email-masked]>
Subject: Updated Web Page

Hi Folks,

As Rainer has pointed out, we have a new version of the syslog-protocol.
http://www.ietf.org/internet-drafts/draft-ietf-syslog-protocol-01.txt

Please take the time to comment on it.  Any issues we have with this MUST
be resolved during the 59th IETF meeting if not before.

I have updated the web page here:
  http://www.employees.org/~lonvick/index.shtml
This now includes
- - a pointer to Rainers' email archive (thanks :-)
- - a pointer to the issues raised by Glenn about the syslog-mib work.
- - a pointer to our plans to meet at the 59th IETF.

Also, please take the time to review Glenn's list of issues and comment
upon them.  These items were raised during our last meeting.
  http://www.employees.org/~lonvick/issues-mib-04-a.txt
(I'll convert that to html with links to the emails rsn.)

Thanks,
Chris

------------------------------

Date: Wed, 21 Jan 2004 14:48:09 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Rainer:

Good draft!  I have a laundry list of suggestions/questions below.  Feel
free to respond to them in different emails at your convenience.  The
issues are in more or less section order.

1. First, I have to say I don't understand the backward compatibility
aspects. RFC 3164 compliant syslog collector or relay is supposed to
accept ANY messages as stated in section 4.2 of that RFC.  Can you
explain in practical terms what backward-compatibility we are trying to
achieve?  I think it is an important question we need to settle as this
affects the whole draft.

I assume the new draft RFC will provide hard requirements for syslog in
contrast to the informational nature of RFC 3164.  As such, this RFC
cannot be fully backward compatible with RFC 3164 which allowed free
form messages.  If we make various selected RFC 3164 aspects required in
the draft (like old timestamp), we are essentially putting a stamp of
approval on a previously informational recommendation and making it a
requirement instead of obsoleting it.  How are we ever going to obsolete
the old format this way?

2. Section 2 refers to "machines" and "devices" which is misleading.  I
think we need to talk about "applications". After all a sender and
collector can both be on the same machine.

3. Section 4. HOSTNAME.  I think "." and "-" characters are allowed in
FQDN (except no trailing "-") per RFC 1123. Also, the limit of 64
characters is inappropriate. It should be 255 per same RFC.

4. Section 4. The time-secfrac field should be specified as 1*4DIGIT.
This is the only number of digits that would be allowed given the 32
character limit you specified for TIMESTAMP field. This just makes it
more explicit and actually removes the need to specify the length of the
TIMESTAMP field.

5. Section 4. MSG. I think the character set specified here is not
consistent with specifying that UTF-8 is supported.  UTF-8 character can
consist of multiple bytes and each byte can be any 8-byte value. Also
you refer to "PRINTABLE" in the comment, which is not defined anywhere.

6. Section 4.1. PRI field.  First, I support Albert's proposal of a new
format which increases the number of facilities and provides a format
that is easier to handle.  I just don't know why stop at 999 facilities
and not allow say 2bln (signed 32-bit).  An alternative (less optimal
for performance) is defining a structured content parameter "facility"
or "channel" and assuming new syslog collectors/relays will use it in
configurations.

7. Section 4.1. PRI field.  I think naming facilities 16-23 "local" is
misleading.  In fact, remote logging uses those almost exclusively. So,
how are they local? I'd call them "custom facility 1,2,..." or something
like that.

8. Section 4.1. Note 1.  I think here and in many other places in this
draft RFC we should avoid using language such as "...have been seen...",
etc.  This is not intended to be an informational RFC like 3164. I think
it would be more appropriate to be talking about what SHOULD or MUST
happen instead of what has been seen to happen.

9. Section 4.2. Here and thereafter you use the term "visible (printing)
characters".  Although you clarify everywhere the specific character
range, I think this term is imprecise.  A Chinese character encoded in
UTF-8 will be visible if you have the right viewer and not visible if
you don't.  Maybe you should refer to "non-control characters" instead.

10.  Section 4.2 Last 3 sentences. Again you mention "has usually been
seen".  Do we actually want to recommend the use of one IP or the other
or at least the consistent use of one?

11. Section 4.2.1. In the note, you mention "single syntax".  In fact,
use of second fractions is optional. Yes, technically it is one ABNF
syntax. But then so is the RFC 3339 which you claim provides "multiple
syntaxes".

12. Section 4.2.1. My feeling is we should not support the old timestamp
format in this RFC.  If some collector wants to support it, they can be
both RFC3164 and RFC.new compatible, right?  Why give more prominence to
the old legacy timestamp which we know is bad?

13. Section 4.2.1. Bullet point talking about time-secfrac should
mention that performance considerations is another condition for the
recommendation, not just availability of clock accuracy.

14. Section 4.2.2.  Again, I am not convinced that supporting the legacy
of just the hostname instead of FQDN is the good reason to have. We may
still want just the hostname option for local logging though.

15. Section 4.2.2.  Do we want to make a recommendation as to what is
preferred hostname-only or IP?

16. Section 4.2.2. Where we mention IPv6 RFC 2373, we should mention
specifically the section on "Textual representation" of that RFC -
section 2.

17. Section 4.2.3.  We never say what the purpose of the TAG field is
nor give any guidance to what should be put there. This field of the
syslog specification is, to me, very strange.  I understand the legacy,
but see my concerns about backward-compatibility.  The fact that no
spaces are allowed is not optimal.  Recommendation of a trailing ":" can
only mislead casual observers into believing it is used as a separator
character while it is not.  Then, what's the purpose of this
recommendation?

18. Section 4.2.3.  We never explain what's the difference between
static and dynamic portions of TAG.  The last sentence talks about use
of "consistent tag value", but I don't understand what it means.
Consistent between what and what? This needs clarification.

19.  Section 4.3. The phrase "traditionally and most frequently used"
should be replaces with SHOULD, MUST or RECOMMENDED I think.

20. Section 4.3. The last two paragraphs are talking about some "code
sets".  I think if we are talking about UTF-8, we are talking about
*one* code set -- UNICODE -- and one encoding -- UTF-8.  I thought
UNICODE and UTF-8 obsoleted all that code set business, or am I wrong?

21. Section 4.4. TRAILER. This says that some receivers may require a
trailer.  Aren't we supposed to specify here what compatible receiver is
allowed to require and what not? Why are we allowing this?  I think
nobody should require trailer and we should drop this from format.

22. Section 4.5. Sentence "..locally defined facility (local4)...".
Again, I am confused by term "locally-defined".

23. Section 5 & beyond. Why is there a need to specified structured data
*anywhere* within the message.  I thought we will designate a special
field like TAG for the structured data.  This way we won't need a
special sequence to identify it.  Also, I think allowing it everywhere
gives too much unnecessary freedom. Harder to evolve protocol later.

24. Section 5.1.  Like with the MSG, I think the character set of
parameter value is any non-control character with some characters being
escaped. We are supporting UTF-8 within the parameter values, right?

25. Section 5.1. I think the fact that each structured data item which
has a different IANA dictionary needs to be in a different block is
somewhat cumbersome and limiting. For example, if I want to put the
msgid parameter in all of my messages regardless of use of
fragmentation, then when I do use fragmentation, do I have to put this
parameter twice?

26. Section 5.1.  I think dictionary identifier can be made into a just
another key-value parameter. This would be more consistent with
providing a general mechanism key-value pairs and idea of using []
brackets to group related tags.

27. Section 5.1. Can the SD-ID be optional for experimental parameters.
This way I don't have to put "x-cisco" in front of all tags.  I don't
see any value in this.  We can just assume experimental tags.  If a
given vendor needs to identify his tags they can do this with their own
parameters like "vendor", "product", "version", or whatever else the
vendor wants.  Vendor tags are for vendor use only, right? General
syslog collector won't use them anyway, correct?

28. Section 5.1. I would also consider the following approach which
eliminates dictionaries.  If we only need parameter namespace so we can
avoid conflicts between current & future syslog RFCs and vendor
parameters, then we can just define some prefix for current and future
syslog protocol parameters. For example "sys.msgno", "sys.fragcount",
etc.  Then, we will control the tags in this namespace using IANA or
RFCs.  If some vendor wants to re-use the "sys.msgno" tag because the
definition of the tag suites them for a different use case, then they
don't need to duplicate it.

29. Section 5.1. I think we should require a space character after each
structured data block closing bracket.  This will make it more readable
while eliminating the ambiguity as to whether or not the space is part
of the message.  Even you examples will look nicer. I think we can make
the space optional between two structured blocks of data.

30. Section 6. Paragraph 3 call for not using fragmentation when message
can fit in a single message.  I think, in general, we assume the use of
fragmentation *only* for splitting long messages.  We had some
discussion on this a long time ago, but I don't remember the conclusion.
The other use case for fragmentation (or better named multi-part
messages in this case) is when the message is inherently multi-line.
For example, a stack trace:

LockConflictException
 at com.cisco.csrc.db.LockTable.obtainUpdateLock(LockTable.java:199)
 at
com.cisco.csrc.db.indexes.OidIndex.obtainUpdateLock(OidIndex.java:448)
 at
com.cisco.csrc.db.PObjectImpl.obtainUpdateLock(PObjectImpl.java:1184)

How can I send such message with current draft?  I would have to come up
with some new parameters likely.  I think this needs to be standardized.
The distinction here is that the original message is not a single line.
Rather the original message is a multi-part message with each part being
a separate line.

To handle the above we need to differentiate the case when message does
not need to be assembled.

31. Section 6.  Again we had discussion on this before... It would be
useful if message parts could be sent before the total length of the
message is know.  We have one message in our system which is about 2000
lines long. It dumps all kinds of properties on crash.  It would be nice
if I could send parts of this message without knowing the total message
part count.  Otherwise, I would need to assemble the whole message
before sending it.  This can be problematic if I am crashing due to out
of memory condition, for example.  To address this, we simply need to
sate that recount parameter is optional in all fragments except for the
last message.  This will designate the end of the fragmented message.

32. Section 6.2.  The above suggestions would mean that you can't sign
the whole message, only parts.  You suggest that signing all parts is
not as safe as signing the whole message.  Why?  We know exactly the
message to which each part belongs and this information is  signed,
right?

33. General. What do we do with non-conforming messages.  Do we want to
recommend that collectors/relay agent fire some diagnostic message which
embeds the offending message?

34. Do we want to introduce more standard parameters? Good candidates
are "facility" and "severity".  Yes, this will duplicate information,
but we can make them optional.  At least this will overcome the problem
of syslog servers only storing the message and not the PRI field which
leads to then not knowing what facility or severity the message had if
you store multiple facilities/severities in the same log file.

I did not review section 7 and beyond yet.  It seems a lot of it is
identical to old RFC.

Thanks,
Anton.

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Rainer Gerhards
> Sent: Wednesday, January 21, 2004 3:53 AM
> To: [email-masked]
> Subject: syslog-protocol-01 posted & comments
>
>
> Hi WG,
>
> as you may have seen, the draft editor has posted protocol-01:
>
> http://www.ietf.org/internet-drafts/draft-ietf-syslog-protocol-01.txt
>
> First things first: this was a "quick" edit (while not as
> quick as I hoped... ;)). My main objective was to get out
> some text as quickly as possible. There are probably some
> typos and some other minor inconsistencies. Also, some of the
> descriptions may not be as good as they should be. As the
> format issue was quite controversal, I try to save a little
> bit work by providing ONE POSSIBLE text to handle it. But
> further discussion may go into a different direction. So I
> tried to make it as understandable as possible while not
> putting total finishing efforts into it. Once we have decided
> the final direction, I will either revamp totally or polish
> the current text.
>
> To the content:
>
> #1
> As said on the list, I used Anton's non-XML proposal.
> Weighting all the arguments received, I really think we do
> not actually need XML in syslog, even though syslog is no
> longer just for human review but also for automatted
> processes (this in answer to David's question).
>
> After finishing the text, I am far more convinced that the
> simple tagging approach is not only sufficiently enough for
> transport, but it also is a good solution for syslog in
> general. In the long term, it can also be used to define
> payload dictionaries, which may be very useful (should we
> manage to do this;)).
>
> Regarding integrating this into XML-based systems, I think a
> mapping of what I described now and XML is fairly easy, at
> least as long as you assume that the message is orginally
> generated by a syslog device and the brought over to some XML system.
>
> #2
> I would also like to drag your attention to the fragmentation
> that I now described. There are some specific implications in
> regard to syslog-sign. I would appreciate if those deeply
> involved in -sign could cross-check that this format could
> actually do the job for -sign.
>
> #3
> The section on fragmentation is currently missing a
> recommendation to use a reliable transport. This will be
> moved in once we have a general concensus.
>
> I appreciate comments on these points as well as -protocol-01
> in general. I am prepared to do a quick re-edit.
>
> Thanks,
> Rainer
>
>
>

------------------------------

Date: Wed, 21 Jan 2004 21:36:38 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Anton:

Thanks for the great feedback. I would just like to reply to one thing
now, simply because it is 9:30p over here now - but that one is
important for all others reviewing the draft. I should have mentioned it
in my mail:

> I did not review section 7 and beyond yet.  It seems a lot of it is
> identical to old RFC.
>

Actually, it *is* the same. I did not touch it yet because almost all
things in there will rely on the format specified. So I intended to add
this once we are in for a more specific format. Of course, that could
raise some new format questions, but at least I find it easier taking
that round.

I will try to reply to most of the other points tomorrow. I hope we will
also see some other commments by then ;)

Rainer

------------------------------

Date: Thu, 22 Jan 2004 18:16:29 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Anton,

with this mail, I am still just adressing one question, I think the
major one on why compatibility. By adressing only this, I hope to save
some time. I would like to get some feedback on the WG if the proposed
format (with non-xml brackets) is generally acceptable. I think this is
one major question - because if the answer would be no, all other work
done would be lost.

So if someone does not like this format, please speak up now. I would
really appreciate this. I'll wait until monday before I do any major
edit. If I got no objection by then, I assume the format in general is
accepted and we "just" need to look at the "minor" issues of the draft.

Now to backwards compatibility: Of course, we create a new standard for
new syslog implementation. However, I am *very* sure that existing RFC
3164 compliant and NON-compliant implementation will be out for many
years to come. Obviously, all real-world syslogds will need to support
those older clients - or they will not receive market acceptance. So my
goal is to both specify a new standard but leave a window open so that a
newer syslogd can still - in a standard way - support the older
protocols. In fact, I think the older emitors can get quite a lot from
this, as the first relay is permitted to change this.

However, a cleaner solution may be to assign new IP ports to
syslog-protocol, saying that it is actually *not* syslog. But I fear
this, too, causes some loss of acceptance. And as I do not see real evil
in how it is specified, I would like to stick with this approach ...
until a good argument against it comes up.

(To adress a specific issue: the charset will be specified via a
low-footprint -international structured data element - this is why I
left it open in the traditional sense. And, yes, some wording needs to
be changed, too.).

I appreciate all comments.

Rainer

> -----Original Message-----
> From: Anton Okmianski [mailto:[email-masked]]
> Sent: Wednesday, January 21, 2004 8:48 PM
> To: Rainer Gerhards; [email-masked]
> Subject: RE: syslog-protocol-01 posted & comments
>
> Rainer:
>
> Good draft!  I have a laundry list of suggestions/questions
> below.  Feel
> free to respond to them in different emails at your convenience.  The
> issues are in more or less section order.
>
> 1. First, I have to say I don't understand the backward compatibility
> aspects. RFC 3164 compliant syslog collector or relay is supposed to
> accept ANY messages as stated in section 4.2 of that RFC.  Can you
> explain in practical terms what backward-compatibility we are
> trying to
> achieve?  I think it is an important question we need to
> settle as this
> affects the whole draft.
>
> I assume the new draft RFC will provide hard requirements for
> syslog in
> contrast to the informational nature of RFC 3164.  As such, this RFC
> cannot be fully backward compatible with RFC 3164 which allowed free
> form messages.  If we make various selected RFC 3164 aspects
> required in
> the draft (like old timestamp), we are essentially putting a stamp of
> approval on a previously informational recommendation and making it a
> requirement instead of obsoleting it.  How are we ever going
> to obsolete
> the old format this way?
>
> 2. Section 2 refers to "machines" and "devices" which is
> misleading.  I
> think we need to talk about "applications". After all a sender and
> collector can both be on the same machine.
>
> 3. Section 4. HOSTNAME.  I think "." and "-" characters are allowed in
> FQDN (except no trailing "-") per RFC 1123. Also, the limit of 64
> characters is inappropriate. It should be 255 per same RFC.
>
> 4. Section 4. The time-secfrac field should be specified as 1*4DIGIT.
> This is the only number of digits that would be allowed given the 32
> character limit you specified for TIMESTAMP field. This just makes it
> more explicit and actually removes the need to specify the
> length of the
> TIMESTAMP field.
>
> 5. Section 4. MSG. I think the character set specified here is not
> consistent with specifying that UTF-8 is supported.  UTF-8
> character can
> consist of multiple bytes and each byte can be any 8-byte value. Also
> you refer to "PRINTABLE" in the comment, which is not defined
> anywhere.
>
> 6. Section 4.1. PRI field.  First, I support Albert's
> proposal of a new
> format which increases the number of facilities and provides a format
> that is easier to handle.  I just don't know why stop at 999
> facilities
> and not allow say 2bln (signed 32-bit).  An alternative (less optimal
> for performance) is defining a structured content parameter "facility"
> or "channel" and assuming new syslog collectors/relays will use it in
> configurations.
>
> 7. Section 4.1. PRI field.  I think naming facilities 16-23 "local" is
> misleading.  In fact, remote logging uses those almost
> exclusively. So,
> how are they local? I'd call them "custom facility 1,2,..."
> or something
> like that.
>
> 8. Section 4.1. Note 1.  I think here and in many other places in this
> draft RFC we should avoid using language such as "...have
> been seen...",
> etc.  This is not intended to be an informational RFC like
> 3164. I think
> it would be more appropriate to be talking about what SHOULD or MUST
> happen instead of what has been seen to happen.
>
> 9. Section 4.2. Here and thereafter you use the term "visible
> (printing)
> characters".  Although you clarify everywhere the specific character
> range, I think this term is imprecise.  A Chinese character encoded in
> UTF-8 will be visible if you have the right viewer and not visible if
> you don't.  Maybe you should refer to "non-control
> characters" instead.
>
> 10.  Section 4.2 Last 3 sentences. Again you mention "has usually been
> seen".  Do we actually want to recommend the use of one IP or
> the other
> or at least the consistent use of one?
>
> 11. Section 4.2.1. In the note, you mention "single syntax".  In fact,
> use of second fractions is optional. Yes, technically it is one ABNF
> syntax. But then so is the RFC 3339 which you claim provides "multiple
> syntaxes".
>
> 12. Section 4.2.1. My feeling is we should not support the
> old timestamp
> format in this RFC.  If some collector wants to support it,
> they can be
> both RFC3164 and RFC.new compatible, right?  Why give more
> prominence to
> the old legacy timestamp which we know is bad?
>
> 13. Section 4.2.1. Bullet point talking about time-secfrac should
> mention that performance considerations is another condition for the
> recommendation, not just availability of clock accuracy.
>
> 14. Section 4.2.2.  Again, I am not convinced that supporting
> the legacy
> of just the hostname instead of FQDN is the good reason to
> have. We may
> still want just the hostname option for local logging though.
>
> 15. Section 4.2.2.  Do we want to make a recommendation as to what is
> preferred hostname-only or IP?
>
> 16. Section 4.2.2. Where we mention IPv6 RFC 2373, we should mention
> specifically the section on "Textual representation" of that RFC -
> section 2.
>
> 17. Section 4.2.3.  We never say what the purpose of the TAG field is
> nor give any guidance to what should be put there. This field of the
> syslog specification is, to me, very strange.  I understand
> the legacy,
> but see my concerns about backward-compatibility.  The fact that no
> spaces are allowed is not optimal.  Recommendation of a
> trailing ":" can
> only mislead casual observers into believing it is used as a separator
> character while it is not.  Then, what's the purpose of this
> recommendation?
>
> 18. Section 4.2.3.  We never explain what's the difference between
> static and dynamic portions of TAG.  The last sentence talks about use
> of "consistent tag value", but I don't understand what it means.
> Consistent between what and what? This needs clarification.
>
> 19.  Section 4.3. The phrase "traditionally and most frequently used"
> should be replaces with SHOULD, MUST or RECOMMENDED I think.
>
> 20. Section 4.3. The last two paragraphs are talking about some "code
> sets".  I think if we are talking about UTF-8, we are talking about
> *one* code set -- UNICODE -- and one encoding -- UTF-8.  I thought
> UNICODE and UTF-8 obsoleted all that code set business, or am I wrong?
>
> 21. Section 4.4. TRAILER. This says that some receivers may require a
> trailer.  Aren't we supposed to specify here what compatible
> receiver is
> allowed to require and what not? Why are we allowing this?  I think
> nobody should require trailer and we should drop this from format.
>
> 22. Section 4.5. Sentence "..locally defined facility (local4)...".
> Again, I am confused by term "locally-defined".
>
> 23. Section 5 & beyond. Why is there a need to specified
> structured data
> *anywhere* within the message.  I thought we will designate a special
> field like TAG for the structured data.  This way we won't need a
> special sequence to identify it.  Also, I think allowing it everywhere
> gives too much unnecessary freedom. Harder to evolve protocol later.
>
> 24. Section 5.1.  Like with the MSG, I think the character set of
> parameter value is any non-control character with some
> characters being
> escaped. We are supporting UTF-8 within the parameter values, right?
>
> 25. Section 5.1. I think the fact that each structured data item which
> has a different IANA dictionary needs to be in a different block is
> somewhat cumbersome and limiting. For example, if I want to put the
> msgid parameter in all of my messages regardless of use of
> fragmentation, then when I do use fragmentation, do I have to put this
> parameter twice?
>
> 26. Section 5.1.  I think dictionary identifier can be made
> into a just
> another key-value parameter. This would be more consistent with
> providing a general mechanism key-value pairs and idea of using []
> brackets to group related tags.
>
> 27. Section 5.1. Can the SD-ID be optional for experimental
> parameters.
> This way I don't have to put "x-cisco" in front of all tags.  I don't
> see any value in this.  We can just assume experimental tags.  If a
> given vendor needs to identify his tags they can do this with
> their own
> parameters like "vendor", "product", "version", or whatever else the
> vendor wants.  Vendor tags are for vendor use only, right? General
> syslog collector won't use them anyway, correct?
>
> 28. Section 5.1. I would also consider the following approach which
> eliminates dictionaries.  If we only need parameter namespace
> so we can
> avoid conflicts between current & future syslog RFCs and vendor
> parameters, then we can just define some prefix for current and future
> syslog protocol parameters. For example "sys.msgno", "sys.fragcount",
> etc.  Then, we will control the tags in this namespace using IANA or
> RFCs.  If some vendor wants to re-use the "sys.msgno" tag because the
> definition of the tag suites them for a different use case, then they
> don't need to duplicate it.
>
> 29. Section 5.1. I think we should require a space character
> after each
> structured data block closing bracket.  This will make it
> more readable
> while eliminating the ambiguity as to whether or not the space is part
> of the message.  Even you examples will look nicer. I think
> we can make
> the space optional between two structured blocks of data.
>
> 30. Section 6. Paragraph 3 call for not using fragmentation
> when message
> can fit in a single message.  I think, in general, we assume
> the use of
> fragmentation *only* for splitting long messages.  We had some
> discussion on this a long time ago, but I don't remember the
> conclusion.
> The other use case for fragmentation (or better named multi-part
> messages in this case) is when the message is inherently multi-line.
> For example, a stack trace:
>
> LockConflictException
>  at com.cisco.csrc.db.LockTable.obtainUpdateLock(LockTable.java:199)
>  at
> com.cisco.csrc.db.indexes.OidIndex.obtainUpdateLock(OidIndex.java:448)
>  at
> com.cisco.csrc.db.PObjectImpl.obtainUpdateLock(PObjectImpl.java:1184)
>
> How can I send such message with current draft?  I would have
> to come up
> with some new parameters likely.  I think this needs to be
> standardized.
> The distinction here is that the original message is not a
> single line.
> Rather the original message is a multi-part message with each
> part being
> a separate line.
>
> To handle the above we need to differentiate the case when
> message does
> not need to be assembled.
>
> 31. Section 6.  Again we had discussion on this before... It would be
> useful if message parts could be sent before the total length of the
> message is know.  We have one message in our system which is
> about 2000
> lines long. It dumps all kinds of properties on crash.  It
> would be nice
> if I could send parts of this message without knowing the
> total message
> part count.  Otherwise, I would need to assemble the whole message
> before sending it.  This can be problematic if I am crashing
> due to out
> of memory condition, for example.  To address this, we simply need to
> sate that recount parameter is optional in all fragments
> except for the
> last message.  This will designate the end of the fragmented message.
>
> 32. Section 6.2.  The above suggestions would mean that you can't sign
> the whole message, only parts.  You suggest that signing all parts is
> not as safe as signing the whole message.  Why?  We know exactly the
> message to which each part belongs and this information is  signed,
> right?
>
> 33. General. What do we do with non-conforming messages.  Do
> we want to
> recommend that collectors/relay agent fire some diagnostic
> message which
> embeds the offending message?
>
> 34. Do we want to introduce more standard parameters? Good candidates
> are "facility" and "severity".  Yes, this will duplicate information,
> but we can make them optional.  At least this will overcome
> the problem
> of syslog servers only storing the message and not the PRI field which
> leads to then not knowing what facility or severity the message had if
> you store multiple facilities/severities in the same log file.
>
> I did not review section 7 and beyond yet.  It seems a lot of it is
> identical to old RFC.
>
> Thanks,
> Anton.
>
>
> > -----Original Message-----
> > From: [email-masked]
> > [mailto:[email-masked]] On Behalf Of Rainer Gerhards
> > Sent: Wednesday, January 21, 2004 3:53 AM
> > To: [email-masked]
> > Subject: syslog-protocol-01 posted & comments
> >
> >
> > Hi WG,
> >
> > as you may have seen, the draft editor has posted protocol-01:
> >
> >
> http://www.ietf.org/internet-drafts/draft-ietf-syslog-protocol-01.txt
> >
> > First things first: this was a "quick" edit (while not as
> > quick as I hoped... ;)). My main objective was to get out
> > some text as quickly as possible. There are probably some
> > typos and some other minor inconsistencies. Also, some of the
> > descriptions may not be as good as they should be. As the
> > format issue was quite controversal, I try to save a little
> > bit work by providing ONE POSSIBLE text to handle it. But
> > further discussion may go into a different direction. So I
> > tried to make it as understandable as possible while not
> > putting total finishing efforts into it. Once we have decided
> > the final direction, I will either revamp totally or polish
> > the current text.
> >
> > To the content:
> >
> > #1
> > As said on the list, I used Anton's non-XML proposal.
> > Weighting all the arguments received, I really think we do
> > not actually need XML in syslog, even though syslog is no
> > longer just for human review but also for automatted
> > processes (this in answer to David's question).
> >
> > After finishing the text, I am far more convinced that the
> > simple tagging approach is not only sufficiently enough for
> > transport, but it also is a good solution for syslog in
> > general. In the long term, it can also be used to define
> > payload dictionaries, which may be very useful (should we
> > manage to do this;)).
> >
> > Regarding integrating this into XML-based systems, I think a
> > mapping of what I described now and XML is fairly easy, at
> > least as long as you assume that the message is orginally
> > generated by a syslog device and the brought over to some
> XML system.
> >
> > #2
> > I would also like to drag your attention to the fragmentation
> > that I now described. There are some specific implications in
> > regard to syslog-sign. I would appreciate if those deeply
> > involved in -sign could cross-check that this format could
> > actually do the job for -sign.
> >
> > #3
> > The section on fragmentation is currently missing a
> > recommendation to use a reliable transport. This will be
> > moved in once we have a general concensus.
> >
> > I appreciate comments on these points as well as -protocol-01
> > in general. I am prepared to do a quick re-edit.
> >
> > Thanks,
> > Rainer
> >
> >
> >
>
>
>
>

------------------------------

Date: Thu, 22 Jan 2004 15:31:50 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Rainer:

Sorry, but I am still not getting backwards compatibility business.
Please help me understand it.

Clients which fire messages in RFC.new format are always going to be
compatible with RFC 3164 servers regardless of what we put into RFC.new.
This is because RFC 3164 allows anything. The way we have the draft now,
any RFC 3164 server can accept/relay RFC 3164 or RFC.new messages.  This
is not an issue. We get this backwards compatibility "for free". Right?

Now, will clients which fire messages in *any* RFC 3164 format be
compatible with syslog server which implements *only* RFC.new? I hope
note because it would defeat the purpose of a new RFC.  I think what you
spec'ed is at best only partially compatible -- only if they follow some
selected recommendations of RFC 3164 which are compatible with RFC.new.
Right?

> Now to backwards compatibility: Of course, we create a new
> standard for new syslog implementation. However, I am *very*
> sure that existing RFC 3164 compliant and NON-compliant
> implementation will be out for many years to come. Obviously,
> all real-world syslogds will need to support those older
> clients - or they will not receive market acceptance.

I think nothing prevents a syslog server implementation from complying
with both RFC 3164 and RFC.new. And whatever else they want to comply
with.  None of the RFCs prevent an implementation from processing
messages in any other format, right?  But our job is to define a precise
new format which will be useful to new generations of applications.

We are not saying that RFC.new obsoletes RFC 3164, right?

I would rather leave RFC 3164 as open as it is.  But in RFC.new define a
strict format we want new applications follow. So, if somebody says that
their syslog client is RFC.new-compliant then it means it fires messages
in one precise format we know is desirable. And not, maybe a new format,
but maybe also some old one like with old timestamp.  Otherwise, we are
allowing an implementation to use an old format and claim it is
RFC.new-compatible.  I think it is not desirable. This would require me,
for example, saying to our development groups that being compliant with
RFC.new is not enough.  I would also have to ask for implementations of
a specific timestamp format, for example.  I'd rather not have to do
that wherever possible. It creates ambiguity around the RFC.

Note: if we only support the new timestamp RFC.new messages will *still*
be RFC 3164 compatible.  So, I don't quite grasp the issue here.

Thanks,
Anton.

------------------------------

Date: Sat, 24 Jan 2004 21:32:43 +1300
From: Andrew Ross <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Like Anton, I would like to ensure that RFC.new uses a new, specific
message format, that does not use the RFC3164 timestamp.

The [] or <> bracketing system with some sort of escaping will work for
me.

My expectation is that if a syslog relay/collector receives a RFC3164
(or any non-RFC.new format) message, that it will modify it and send it
on (or log it to disk) in the RFC.new format.

It sounds like we are all on the same wavelength, but saying it in a
different way.

Cheers

Andrew

>Anton wrote...
>I would rather leave RFC 3164 as open as it is.  But in RFC.new define
a
>strict format we want new applications follow. So, if somebody says
that
>their syslog client is RFC.new-compliant then it means it fires
messages
>in one precise format we know is desirable. And not, maybe a new
format,
>but maybe also some old one like with old timestamp.  Otherwise, we are
>allowing an implementation to use an old format and claim it is
>RFC.new-compatible.  I think it is not desirable. This would require
me,
>for example, saying to our development groups that being compliant with
>RFC.new is not enough.  I would also have to ask for implementations of
>a specific timestamp format, for example.  I'd rather not have to do
>that wherever possible. It creates ambiguity around the RFC.

------------------------------

Date: Sun, 25 Jan 2004 23:04:57 +0900
From: Glenn Mansfield Keeni <[email-masked]>
Subject: Re: SyslogMIB Issue-#4

Hi Dave,
    Thanks for the comments. I agree with you. Traps are useful.
They provide a uniform mechanism/console for the network manager
to manage the network, related devices and processes.
    There are two levels at which we may want SNMP traps in the
Syslog Messaging context.
    a. To provide another option of how a message will be
       processed. Presently we have
          - log (syslogCtlLogActionTable)
          - display on users console (syslogCtlUserActionTable)
          - relay/forward (syslogCtlFwdActionTable)
          - pipe (syslogCtlPipeActionTable)
       We can add snmp trap to this with relative ease. It is
       just a matter of adding another table
       e.g. syslogCtlSnmpTrapActionTable
       I would propose that the contents of the trap would be
              o the syslog message itself. Perhaps the length will
                need to be constrained.
              o SyslogSeverity,
              o syslogFacility,
              o message source
              o timestamp.
    b. To notify the syslog/System administrator of notable events
       e.g.   o failure to process a message
                    relay, log, display etc
              o too many rejected messages
              o anything else ?

  Any comments, suggestions ?

  Cheers

  Glenn

Harrington, David wrote:
> Hi Glenn,
>
> The benefit of having an SNMP trap is that there becomes some
> coordination between the syslog messaging and SNMP trap manager
> applications.
>
> Operators often use trap management tools to alert them to problems, and
> these applications often have the ability to filter out uninteresting
> traps, to sort by priority, to do fault isolation, and so on. Trap
> management utilities can also feed into SNMP-based topology mapping
> tools (like HPOV) which can use colored icons to represent serious
> problems, and so on.
>
> The problem with a syslog notification is - what would it contain? You
> could write one syslog notification for reporting all syslog events by
> simply having a varbind with the complete syslog message. That means
> that an SNMP trap application cannot do much in terms of sorting by
> priority. You could create a notification that ocntains some
> standardized values common to syslog messages, such as severity, related
> subsystem, vendor, and then include the human-readable string in an
> octet string. This might be useful, but it might be less useful than
> notifications specific to a technology.
>
> I think the WG needs to discuss the nature of potential notification
> designs.
>
> Personally I think such a generalized trap might be a good thing for
> improving coordination between syslog and SNMP, but it will depend a
> great deal on the actual design and how easily that design can be
> utilized by applications.
>
> dbh
>
>
>>-----Original Message-----
>>From: Glenn Mansfield Keeni [mailto:[email-masked]]
>>Sent: Friday, January 16, 2004 4:48 AM
>>To: [email-masked]
>>Subject: SyslogMIB Issue-#4
>>
>>
>>Issue #4. Snmp Notifications ? Do we need any ?
>>
>>
>>Action: Do we need any?
>>         Note- Syslog itself is a notification.
>>         If there a clear requirement for a notification
>>         that can added.
>>
>>Status: Waiting on community input
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>

------------------------------

Date: Mon, 26 Jan 2004 11:09:47 +1300
From: Andrew Ross <[email-masked]>
Subject: RE: SyslogMIB Issue-#4

Hi Glenn,

That system looks good. Something to map syslog messages into SNMP traps
and another one for notifications.

Let's make the timestamp high resolution like the new proposed syslog
RFC that Rainer is working on.

Leave a string binding in the notification trap for error message
descriptions that we can't plan for.

Maybe set a binding in the syslogCtlSnmpTrapActionTable that indicates a
multi part message. Or just constrain the length to 1024 again??

Regards

Andrew

PS. More comments on your other 6 points will follow when I get my act
together. :-)

- -----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Glenn Mansfield
Keeni
Sent: Monday, 26 January 2004 3:05 a.m.
To: Harrington, David
Cc: [email-masked]
Subject: Re: SyslogMIB Issue-#4

Hi Dave,
    Thanks for the comments. I agree with you. Traps are useful.
They provide a uniform mechanism/console for the network manager
to manage the network, related devices and processes.
    There are two levels at which we may want SNMP traps in the
Syslog Messaging context.
    a. To provide another option of how a message will be
       processed. Presently we have
          - log (syslogCtlLogActionTable)
          - display on users console (syslogCtlUserActionTable)
          - relay/forward (syslogCtlFwdActionTable)
          - pipe (syslogCtlPipeActionTable)
       We can add snmp trap to this with relative ease. It is
       just a matter of adding another table
       e.g. syslogCtlSnmpTrapActionTable
       I would propose that the contents of the trap would be
              o the syslog message itself. Perhaps the length will
                need to be constrained.
              o SyslogSeverity,
              o syslogFacility,
              o message source
              o timestamp.
    b. To notify the syslog/System administrator of notable events
       e.g.   o failure to process a message
                    relay, log, display etc
              o too many rejected messages
              o anything else ?

  Any comments, suggestions ?

  Cheers

  Glenn

Harrington, David wrote:
> Hi Glenn,
>
> The benefit of having an SNMP trap is that there becomes some
> coordination between the syslog messaging and SNMP trap manager
> applications.
>
> Operators often use trap management tools to alert them to problems,
and
> these applications often have the ability to filter out uninteresting
> traps, to sort by priority, to do fault isolation, and so on. Trap
> management utilities can also feed into SNMP-based topology mapping
> tools (like HPOV) which can use colored icons to represent serious
> problems, and so on.
>
> The problem with a syslog notification is - what would it contain? You
> could write one syslog notification for reporting all syslog events by
> simply having a varbind with the complete syslog message. That means
> that an SNMP trap application cannot do much in terms of sorting by
> priority. You could create a notification that ocntains some
> standardized values common to syslog messages, such as severity,
related
> subsystem, vendor, and then include the human-readable string in an
> octet string. This might be useful, but it might be less useful than
> notifications specific to a technology.
>
> I think the WG needs to discuss the nature of potential notification
> designs.
>
> Personally I think such a generalized trap might be a good thing for
> improving coordination between syslog and SNMP, but it will depend a
> great deal on the actual design and how easily that design can be
> utilized by applications.
>
> dbh
>
>
>>-----Original Message-----
>>From: Glenn Mansfield Keeni [mailto:[email-masked]]
>>Sent: Friday, January 16, 2004 4:48 AM
>>To: [email-masked]
>>Subject: SyslogMIB Issue-#4
>>
>>
>>Issue #4. Snmp Notifications ? Do we need any ?
>>
>>
>>Action: Do we need any?
>>         Note- Syslog itself is a notification.
>>         If there a clear requirement for a notification
>>         that can added.
>>
>>Status: Waiting on community input
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>

------------------------------

Date: Mon, 26 Jan 2004 11:25:00 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Hi,

this is a reply both to Andrew's as well as Anton's post.

I have the following scenario on my mind:

A company has installed devices speaking both old-style syslog (not
necessarily RFC 3164) as well as -protocol. All of their messages should
be centralized in a single store.

My current approach is to keep -protocol open enough to allow reception
of these old-style messages. That is, the syslogd would receive the
message and while parsing detect if it is actually the new format or
not. If not, it would know these are old style messages (there would
definitely be need for some additional relay rules in the raft).

I am doing it in this spirit, because my experience tells me if you
don't allow this, every vendor will go ahead and create its own version
of the (I-have-a-single-daemon-and-do-all-within-it approach).

So I am mostly concerned when receiving messages. Anton's comments seem
very wisely to address the sender part. There, of course, I would like
to a see a fixed (or at least very limited) format, too.

Would it be a solution to to specify rules that, when sendinging, there
are only MUSTs in regard to the format but there are SHOULDs when
receiving?

We can also drop the backwards compatibility totally. But then we should
be ready to go the full route, that is we should create a new UDP
transport mapping and have a port other than 514 assigend. So you would
need both a traditional, port 514 receiver for the old stuff and a new
port whatever receive for the new stuff. While I think this would
technically be the best way to handle it, I am a bit concrend of what
this costs us in terms of "ooohhh, this is not syslog, this is a new
protocol" missing acceptance by the average user. Still, we would need
to define some rules for relays.

Rainer

> -----Original Message-----
> From: Andrew Ross [mailto:[email-masked]]
> Sent: Saturday, January 24, 2004 9:33 AM
> To: 'Anton Okmianski'; Rainer Gerhards; [email-masked]
> Subject: RE: syslog-protocol-01 posted & comments
>
>
> Like Anton, I would like to ensure that RFC.new uses a new, specific
> message format, that does not use the RFC3164 timestamp.
>
> The [] or <> bracketing system with some sort of escaping
> will work for
> me.
>
> My expectation is that if a syslog relay/collector receives a RFC3164
> (or any non-RFC.new format) message, that it will modify it
> and send it
> on (or log it to disk) in the RFC.new format.
>
> It sounds like we are all on the same wavelength, but saying it in a
> different way.
>
> Cheers
>
> Andrew
>
>
>
> >Anton wrote...
> >I would rather leave RFC 3164 as open as it is.  But in
> RFC.new define
> a
> >strict format we want new applications follow. So, if somebody says
> that
> >their syslog client is RFC.new-compliant then it means it fires
> messages
> >in one precise format we know is desirable. And not, maybe a new
> format,
> >but maybe also some old one like with old timestamp.
> Otherwise, we are
> >allowing an implementation to use an old format and claim it is
> >RFC.new-compatible.  I think it is not desirable. This would require
> me,
> >for example, saying to our development groups that being
> compliant with
> >RFC.new is not enough.  I would also have to ask for
> implementations of
> >a specific timestamp format, for example.  I'd rather not have to do
> >that wherever possible. It creates ambiguity around the RFC.
>
>
>
>
>
>

------------------------------

Date: Mon, 26 Jan 2004 11:07:40 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: SyslogMIB Issue-#4

Hi all,

I have to admit I am not following that closely with SNMP due being busy
with -protocol.

As it looks SNMP trap may be useful, I would suggest that this is
implemented as a transport mapping for -protocol. -protocol is transport
ignorant by design. Some of the structured data elements may be mapped
to SNMP properties, as are the header fields. So we already have a
number of well-defined values.

The ability of mapping syslog message to different transports was a key
thing that made us create -protocol, so we could utilize it here.
Eventually, it would make sense to create some few additional (option)
structure data elements, like the IP address of the original sender.
This may be useful for mapping onto SNMP (if element is provided by
emitor).

Rainer

> -----Original Message-----
> From: Andrew Ross [mailto:[email-masked]]
> Sent: Sunday, January 25, 2004 11:10 PM
> To: 'Glenn Mansfield Keeni'; 'Harrington, David'
> Cc: [email-masked]
> Subject: RE: SyslogMIB Issue-#4
>
>
> Hi Glenn,
>
> That system looks good. Something to map syslog messages into
> SNMP traps
> and another one for notifications.
>
> Let's make the timestamp high resolution like the new proposed syslog
> RFC that Rainer is working on.
>
> Leave a string binding in the notification trap for error message
> descriptions that we can't plan for.
>
> Maybe set a binding in the syslogCtlSnmpTrapActionTable that
> indicates a
> multi part message. Or just constrain the length to 1024 again??
>
> Regards
>
> Andrew
>
> PS. More comments on your other 6 points will follow when I get my act
> together. :-)
>
>
>
>
> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Glenn Mansfield
> Keeni
> Sent: Monday, 26 January 2004 3:05 a.m.
> To: Harrington, David
> Cc: [email-masked]
> Subject: Re: SyslogMIB Issue-#4
>
>
> Hi Dave,
>     Thanks for the comments. I agree with you. Traps are useful.
> They provide a uniform mechanism/console for the network manager
> to manage the network, related devices and processes.
>     There are two levels at which we may want SNMP traps in the
> Syslog Messaging context.
>     a. To provide another option of how a message will be
>        processed. Presently we have
>           - log (syslogCtlLogActionTable)
>           - display on users console (syslogCtlUserActionTable)
>           - relay/forward (syslogCtlFwdActionTable)
>           - pipe (syslogCtlPipeActionTable)
>        We can add snmp trap to this with relative ease. It is
>        just a matter of adding another table
>        e.g. syslogCtlSnmpTrapActionTable
>        I would propose that the contents of the trap would be
>               o the syslog message itself. Perhaps the length will
>                 need to be constrained.
>               o SyslogSeverity,
>               o syslogFacility,
>               o message source
>               o timestamp.
>     b. To notify the syslog/System administrator of notable events
>        e.g.   o failure to process a message
>                     relay, log, display etc
>               o too many rejected messages
>               o anything else ?
>
>
>   Any comments, suggestions ?
>
>   Cheers
>
>   Glenn
>
> Harrington, David wrote:
> > Hi Glenn,
> >
> > The benefit of having an SNMP trap is that there becomes some
> > coordination between the syslog messaging and SNMP trap manager
> > applications.
> >
> > Operators often use trap management tools to alert them to problems,
> and
> > these applications often have the ability to filter out
> uninteresting
> > traps, to sort by priority, to do fault isolation, and so on. Trap
> > management utilities can also feed into SNMP-based topology mapping
> > tools (like HPOV) which can use colored icons to represent serious
> > problems, and so on.
> >
> > The problem with a syslog notification is - what would it
> contain? You
> > could write one syslog notification for reporting all
> syslog events by
> > simply having a varbind with the complete syslog message. That means
> > that an SNMP trap application cannot do much in terms of sorting by
> > priority. You could create a notification that ocntains some
> > standardized values common to syslog messages, such as severity,
> related
> > subsystem, vendor, and then include the human-readable string in an
> > octet string. This might be useful, but it might be less useful than
> > notifications specific to a technology.
> >
> > I think the WG needs to discuss the nature of potential notification
> > designs.
> >
> > Personally I think such a generalized trap might be a good thing for
> > improving coordination between syslog and SNMP, but it will depend a
> > great deal on the actual design and how easily that design can be
> > utilized by applications.
> >
> > dbh
> >
> >
> >>-----Original Message-----
> >>From: Glenn Mansfield Keeni [mailto:[email-masked]]
> >>Sent: Friday, January 16, 2004 4:48 AM
> >>To: [email-masked]
> >>Subject: SyslogMIB Issue-#4
> >>
> >>
> >>Issue #4. Snmp Notifications ? Do we need any ?
> >>
> >>
> >>Action: Do we need any?
> >>         Note- Syslog itself is a notification.
> >>         If there a clear requirement for a notification
> >>         that can added.
> >>
> >>Status: Waiting on community input
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
>
>
>
>
>
>
>

------------------------------

Date: Mon, 26 Jan 2004 18:33:17 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Anton,

one more comment:

> > implementation will be out for many years to come. Obviously,
> > all real-world syslogds will need to support those older
> > clients - or they will not receive market acceptance.
>
> I think nothing prevents a syslog server implementation from complying
> with both RFC 3164 and RFC.new.

I think there is indeed a *big* issue. I may be totally missing the
obvious, but how can a syslogd - IN A STANDARD WAY - differntiate
between legacy syslog and -protocol? Its an honest question.

Hoewever, the only answers I have seen to similar issues so far is "look
at the headers and guess" - I think this is not good. Maybe we should
REQUIRE a specifc structured data element to be present, to say "hey,
this is new format" - this could solve it. Is that an idea?

A sample for a REQUIRED structured data element:

[#@syslog version="1"]

Everything that does not have this structued data element, MUST be
considered "old style", everthing that has it MUST be fully compliant
with -protocol. A syslogd just implementing -protocol is free to drop
all packets that do not have this element.

How does this sound?

Another approach may be to change the message format, eg. instead of

<PRI>......

- -protcol may start with

[VERSION, FACILITY, SEVERITY]....

deliberately breaking compatibility.

Comments are highly appreciated.

Rainer

------------------------------

Date: Mon, 26 Jan 2004 10:12:46 -0800 (PST)
From: Chris Lonvick <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Hi,

I believe that we originally put in the "backwards compatibility" wording
in syslog-sign so that the mechanism could be used by classic syslog (3164
style) and also in what we were defining as the protocol in that ID (now
syslog-protocol).  I also think that those thoughts just got migrated into
syslog-protocol because they were in syslog-sign.  I still like the idea
of allowing that in syslog-sign so perhaps syslog-sign should be modified
to say "you really should be using syslog-protocol so do it _this_ way.
However, if you want this type of authentication applied to what you
currently send as syslog messages, then do it _that_ way."

In syslog-protocol, perhaps it would be better to just say "MUST" about
the new fields.  Rainer makes a good point that we would really like to
build syslog-protocol on the vast amount of momentum currently being held
by classic syslog. To do that in the least disruptive way, I think we
should still use udp/514 and still remain somewhat similar to the classic
fields in use.  Would it be suitable to state that intention that in the
Abstract or Introduction in syslog-protocol?  Perhaps it would also be
good to restate "be conservitive in what you send, and liberal in what you
receive" to note that syslog servers should be capable of receiving both
syslog-protocol messages as well as classic style messages.

Thanks,
Chris

On Mon, 26 Jan 2004, Rainer Gerhards wrote:

> Hi,
>
> this is a reply both to Andrew's as well as Anton's post.
>
> I have the following scenario on my mind:
>
> A company has installed devices speaking both old-style syslog (not
> necessarily RFC 3164) as well as -protocol. All of their messages should
> be centralized in a single store.
>
> My current approach is to keep -protocol open enough to allow reception
> of these old-style messages. That is, the syslogd would receive the
> message and while parsing detect if it is actually the new format or
> not. If not, it would know these are old style messages (there would
> definitely be need for some additional relay rules in the raft).
>
> I am doing it in this spirit, because my experience tells me if you
> don't allow this, every vendor will go ahead and create its own version
> of the (I-have-a-single-daemon-and-do-all-within-it approach).
>
> So I am mostly concerned when receiving messages. Anton's comments seem
> very wisely to address the sender part. There, of course, I would like
> to a see a fixed (or at least very limited) format, too.
>
> Would it be a solution to to specify rules that, when sendinging, there
> are only MUSTs in regard to the format but there are SHOULDs when
> receiving?
>
> We can also drop the backwards compatibility totally. But then we should
> be ready to go the full route, that is we should create a new UDP
> transport mapping and have a port other than 514 assigend. So you would
> need both a traditional, port 514 receiver for the old stuff and a new
> port whatever receive for the new stuff. While I think this would
> technically be the best way to handle it, I am a bit concrend of what
> this costs us in terms of "ooohhh, this is not syslog, this is a new
> protocol" missing acceptance by the average user. Still, we would need
> to define some rules for relays.
>
> Rainer
>
> > -----Original Message-----
> > From: Andrew Ross [mailto:[email-masked]]
> > Sent: Saturday, January 24, 2004 9:33 AM
> > To: 'Anton Okmianski'; Rainer Gerhards; [email-masked]
> > Subject: RE: syslog-protocol-01 posted & comments
> >
> >
> > Like Anton, I would like to ensure that RFC.new uses a new, specific
> > message format, that does not use the RFC3164 timestamp.
> >
> > The [] or <> bracketing system with some sort of escaping
> > will work for
> > me.
> >
> > My expectation is that if a syslog relay/collector receives a RFC3164
> > (or any non-RFC.new format) message, that it will modify it
> > and send it
> > on (or log it to disk) in the RFC.new format.
> >
> > It sounds like we are all on the same wavelength, but saying it in a
> > different way.
> >
> > Cheers
> >
> > Andrew
> >
> >
> >
> > >Anton wrote...
> > >I would rather leave RFC 3164 as open as it is.  But in
> > RFC.new define
> > a
> > >strict format we want new applications follow. So, if somebody says
> > that
> > >their syslog client is RFC.new-compliant then it means it fires
> > messages
> > >in one precise format we know is desirable. And not, maybe a new
> > format,
> > >but maybe also some old one like with old timestamp.
> > Otherwise, we are
> > >allowing an implementation to use an old format and claim it is
> > >RFC.new-compatible.  I think it is not desirable. This would require
> > me,
> > >for example, saying to our development groups that being
> > compliant with
> > >RFC.new is not enough.  I would also have to ask for
> > implementations of
> > >a specific timestamp format, for example.  I'd rather not have to do
> > >that wherever possible. It creates ambiguity around the RFC.
> >
> >
> >
> >
> >
> >
>
>
>

------------------------------

Date: Mon, 26 Jan 2004 11:26:17 -0800
From: Devin Kowatch <[email-masked]>
Subject: Re: syslog-protocol-01 posted & comments

On Mon, Jan 26, 2004 at 06:33:17PM +0100, Rainer Gerhards wrote:
> Anton,
>
> one more comment:
>
> > > implementation will be out for many years to come. Obviously,
> > > all real-world syslogds will need to support those older
> > > clients - or they will not receive market acceptance.
> >
> > I think nothing prevents a syslog server implementation from complying
> > with both RFC 3164 and RFC.new.
>
> I think there is indeed a *big* issue. I may be totally missing the
> obvious, but how can a syslogd - IN A STANDARD WAY - differntiate
> between legacy syslog and -protocol? Its an honest question.
>
> Hoewever, the only answers I have seen to similar issues so far is "look
> at the headers and guess" - I think this is not good. Maybe we should
> REQUIRE a specifc structured data element to be present, to say "hey,
> this is new format" - this could solve it. Is that an idea?
>
> A sample for a REQUIRED structured data element:
>
> [#@syslog version="1"]
>
> Everything that does not have this structued data element, MUST be
> considered "old style", everthing that has it MUST be fully compliant
> with -protocol. A syslogd just implementing -protocol is free to drop
> all packets that do not have this element.

I was actually about to suggest that you add a version field to the new
protocol (maybe start it at 2?).  So that changes could be made eaiser
in the future.

Also, I _really_ like the idea of making the new format differ from the
old format in a very well defined way, which is early in message.  This
would make shifting the message to either an old style parser or a
- -protocol parser easier.

>
> How does this sound?
>
> Another approach may be to change the message format, eg. instead of
>
> <PRI>......
>
> -protcol may start with
>
> [VERSION, FACILITY, SEVERITY]....
>
> deliberately breaking compatibility.
>
> Comments are highly appreciated.

Okay, rather than go back and reply to the various messages that have
brought up compatability, I'll lump my comments in this one.

The behaviour I have observed from linux syslogd, and solaris syslogd is
the following.  They will send messages formatted as:
    <PRI> TAG: MSG

I don't recall if there is a timestamp in the message as sent.  The
missing information is added by the reciever.  This information is also
added before writing to a file.  Under solaris at least, the syslogd
server will always add a host name after the timestamp.  So if the
syslog sender, spits out:
    <PRI> TS HOST TAG: MSG
solaris syslogd will write (to it's file):
    TS HOST HOST TAG: MSG

I guess that the point to all of this is that we are almost garenteed to
break backwards compatibility.  The philosophy that I have designed my
syslogd implementation with is to try and recieve all the old style
messages, but not worry too much about behaving like an old style
client.  As such I would very much like to have an easy time
distinguishing between old style and new style messages to make the
parsing easier.  But Don't care too much about being able to send new
style messages to stock syslogd implementations.

- -- 
Devin Kowatch
[email-masked]

------------------------------

Date: Mon, 26 Jan 2004 15:09:31 -0500
From: "Harrington, David" <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Hi,

I agree that a version number would be helpful; it will be even more
helpful when the **next** version of a syslog standard is published.

I suggest that you should consider adding two fields to a standardized
syslog format: the (IANA-assigned) enterprise ID and a version number.
(The IETF should have an enterprise ID, and I know other IETF work has
considered 0 to be the IETF enterpriseID, but check with Joyce Reynolds
whether 0 should be used for this or not.)

This would allow vendors to migrate their existing proprietary versions
of syslog to an intermediate standardized message format, without
forcing them to go all the way to fully standardized content. Some
syslog messages may be encoded in hardware, or in licensed binary
software, or in code the vendor cannot economically justify changing all
at once, and will be difficult to migrate the content to a
fully-standard syslog format. However, the embedded software that
creates the final messages for the wire may be able to wrap the content
in a standarized message format, improving manageability and
interoperability.

Applications that read syslog messages may be able to support some
vendor formats and not others, and the enterprise ID and the version
number provide quick and useful filtering of messages.

dbh

> -----Original Message-----
> From: Rainer Gerhards [mailto:[email-masked]]
> Sent: Monday, January 26, 2004 12:33 PM
> To: Anton Okmianski; [email-masked]
> Subject: RE: syslog-protocol-01 posted & comments
>
> Anton,
>
> one more comment:
>
> > > implementation will be out for many years to come. Obviously,
> > > all real-world syslogds will need to support those older
> > > clients - or they will not receive market acceptance.
> >
> > I think nothing prevents a syslog server implementation
> from complying
> > with both RFC 3164 and RFC.new.
>
> I think there is indeed a *big* issue. I may be totally missing the
> obvious, but how can a syslogd - IN A STANDARD WAY - differntiate
> between legacy syslog and -protocol? Its an honest question.
>
> Hoewever, the only answers I have seen to similar issues so
> far is "look
> at the headers and guess" - I think this is not good. Maybe we should
> REQUIRE a specifc structured data element to be present, to say "hey,
> this is new format" - this could solve it. Is that an idea?
>
> A sample for a REQUIRED structured data element:
>
> [#@syslog version="1"]
>
> Everything that does not have this structued data element, MUST be
> considered "old style", everthing that has it MUST be fully compliant
> with -protocol. A syslogd just implementing -protocol is free to drop
> all packets that do not have this element.
>
> How does this sound?
>
> Another approach may be to change the message format, eg. instead of
>
> <PRI>......
>
> -protcol may start with
>
> [VERSION, FACILITY, SEVERITY]....
>
> deliberately breaking compatibility.
>
> Comments are highly appreciated.
>
> Rainer
>
>
>

------------------------------

Date: Tue, 27 Jan 2004 23:19:54 +1300
From: Andrew Ross <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Rainer,

We do need some way to positively identify the -protocol messages as
being different to the old format messages.

Either of the ways you have suggested would work.

Anyone favour one method more than the other?

Regards

Andrew

Rainer wrote...

I think there is indeed a *big* issue. I may be totally missing the
obvious, but how can a syslogd - IN A STANDARD WAY - differntiate
between legacy syslog and -protocol? Its an honest question.

Hoewever, the only answers I have seen to similar issues so far is "look
at the headers and guess" - I think this is not good. Maybe we should
REQUIRE a specifc structured data element to be present, to say "hey,
this is new format" - this could solve it. Is that an idea?

A sample for a REQUIRED structured data element:

[#@syslog version="1"]

Everything that does not have this structued data element, MUST be
considered "old style", everthing that has it MUST be fully compliant
with -protocol. A syslogd just implementing -protocol is free to drop
all packets that do not have this element.

How does this sound?

Another approach may be to change the message format, eg. instead of

<PRI>......

- -protcol may start with

[VERSION, FACILITY, SEVERITY]....

deliberately breaking compatibility.

Comments are highly appreciated.

Rainer

------------------------------

Date: Tue, 27 Jan 2004 17:43:27 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Hi all,

many thanks for all the great comments. Let me try to wrap up the
concensus I see:

- - the proposed format (with brackets and structured data
  elements) is acceptable
- - -protocol should REQUIRE senders to emit a specific format
- - a version is good to have and will also solve compatibility issues
  (current & future)
- - adding the enterprise ID can further facilitate message parsing
- - having the version identifier early in the message is a plus
- - -protocol should leverage existing syslog momentum, thus
  UDP transport should use destination port 514

and previous discussion already had the consent that -protocol is
sufficiently different from RFC3164 that messages will not be seen as
validly formatted by an old-style syslogd. This just as a reminder.

As such, I am about to change the initial message header from just

<PRI>

to

[VERSION enterpriseID FACILITY SEVERITY]  (separated by SP)

so you can recognise new format by the presence of a "[" instead of a
"<" in the first byte (plus, you MUST check that the rest of the header
is syntaxtically correct in order to avoid misinterpreting a malformed
old-style message).

So I am NOT going to propose a structured data element for this
information. It goes straight into the REQUIRED header.

When you think about a new-format message being received by a RFC3164
syslogd, this is probably the best way to handle it, too. Because a
RFC3164 syslogd sees this as a message without header, it will not try
to parse the rest of the message, which I think is the best that can be
done under the circumstances. A sufficiently smart new syslogd may even
be able to re-extract the -protocol message from a message that was
handled by 3164 syslogds in the middle of a relay chain.

So I think this discussion has brought us much value. At least I am very
happy with the outcome.

I will begin to change the draft in this spirit. I will try to finish it
by tomorrow - let's see how much is actually involved.

If someone objects my conclusions, I would appreciate if you could do
this as quickly as possible - not only to save me some time for the
edit, but also in the hope to do things quickly before the meeting ;)

Thanks,
Rainer

> -----Original Message-----
> From: Andrew Ross [mailto:[email-masked]]
> Sent: Tuesday, January 27, 2004 11:20 AM
> To: Rainer Gerhards; [email-masked]
> Subject: RE: syslog-protocol-01 posted & comments
>
>
> Rainer,
>
> We do need some way to positively identify the -protocol messages as
> being different to the old format messages.
>
> Either of the ways you have suggested would work.
>
> Anyone favour one method more than the other?
>
> Regards
>
> Andrew
>
>
>
>
> Rainer wrote...
>
> I think there is indeed a *big* issue. I may be totally missing the
> obvious, but how can a syslogd - IN A STANDARD WAY - differntiate
> between legacy syslog and -protocol? Its an honest question.
>
> Hoewever, the only answers I have seen to similar issues so
> far is "look
> at the headers and guess" - I think this is not good. Maybe we should
> REQUIRE a specifc structured data element to be present, to say "hey,
> this is new format" - this could solve it. Is that an idea?
>
> A sample for a REQUIRED structured data element:
>
> [#@syslog version="1"]
>
> Everything that does not have this structued data element, MUST be
> considered "old style", everthing that has it MUST be fully compliant
> with -protocol. A syslogd just implementing -protocol is free to drop
> all packets that do not have this element.
>
> How does this sound?
>
> Another approach may be to change the message format, eg. instead of
>
> <PRI>......
>
> -protcol may start with
>
> [VERSION, FACILITY, SEVERITY]....
>
> deliberately breaking compatibility.
>
> Comments are highly appreciated.
>
> Rainer
>
>
>
>
>
>

------------------------------

Date: Wed, 28 Jan 2004 12:53:51 +0100
From: Rainer Gerhards <[email-masked]>
Subject: -international: trailer

Hi WG,

as I am editing -international, I noticed that the TRAILER specified may
be worth looking at. Now that we do not need to remain
backward-compatible, I would opt to have a simple tailer in the message.
It was defined as:

 SYSLOG-MSG      = PRI HEADER MSG [TRAILER]
 TRAILER         = [CR] LF

While the trailer is not strictly necessary to form the message, I think
it is quite a good sanity check if the message looks OK.

Of course, I will continue to edit the draft as this is a minor issue.

Any comments?
Rainer

------------------------------

Date: Wed, 28 Jan 2004 16:09:50 +0100
From: Rainer Gerhards <[email-masked]>
Subject: Advice on Trademark Issues requested

Hi WG,

I thought that "structured data element" is a bit lengthy and was tempted to replace it by the abbreviation "SDE" in the protcol-02 darft. However, I found that this is a trademark. Than, I did some trademark searchon TAG, and voilá, it too is trademarked. I didn't bother to dig for the other abbreviations, but I think some of them will be the same as existing trademarks.

I would appreciate advise from someone with experience on this issue. Is it really an issue? Can I use "SDE" or need I to stick with the longer form?

Any help is deeply appreciated,
Rainer

------------------------------

Date: Wed, 28 Jan 2004 08:39:53 -0800 (PST)
From: Chris Lonvick <[email-masked]>
Subject: Re: Advice on Trademark Issues requested

Hi Rainer,

These documents are the current state of thought in the IETF on IPR,
trademarks, etc.

http://www.ietf.org/internet-drafts/draft-ietf-ipr-wg-guidelines-05.txt
http://www.ietf.org/internet-drafts/draft-ietf-ipr-submission-rights-08.txt
http://www.ietf.org/internet-drafts/draft-ietf-ipr-technology-rights-12.txt

I don't think that we're violating any trademarks by assigning a common
name to a defined field in a protocol.  However, I'll send a note around
(off list) to make sure.  I'll provide a summary back to the list.

Thanks,
Chris

On Wed, 28 Jan 2004, Rainer Gerhards wrote:

> Hi WG,
>
> I thought that "structured data element" is a bit lengthy and was tempted to replace it by the abbreviation "SDE" in the protcol-02 darft. However, I found that this is a trademark. Than, I did some trademark searchon TAG, and voilá, it too is trademarked. I didn't bother to dig for the other abbreviations, but I think some of them will be the same as existing trademarks.
>
> I would appreciate advise from someone with experience on this issue. Is it really an issue? Can I use "SDE" or need I to stick with the longer form?
>
> Any help is deeply appreciated,
> Rainer
>
>
>
>

------------------------------

Date: Wed, 28 Jan 2004 17:23:43 +0100
From: Rainer Gerhards <[email-masked]>
Subject: Facilities and Severity

Hi WG,

again on the edit: I am replacing the few fixed facilities by a 4-digit
number right now. I wonder if it would still make sense to mention the
traditional facilities in -protocol. Same goes for severity, where we
now also allow additional values.

I would appreciate comments on this. For now, I will remove any
references. This does not imply that I think they should be removed. I
am actually half-half.

If that is concensus will be that the traditional values should be
mentioned, I'll move them back in either in -02 (if comments arrive
before end of edit) or -03 (if not).

Thanks,
Rainer

------------------------------

Date: Wed, 28 Jan 2004 12:19:14 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: -international: trailer

Rainer:

> as I am editing -international, I noticed that the TRAILER
> specified may be worth looking at. Now that we do not need to
> remain backward-compatible, I would opt to have a simple
> tailer in the message. It was defined as:
>
>  SYSLOG-MSG      = PRI HEADER MSG [TRAILER]
>  TRAILER         = [CR] LF
>
> While the trailer is not strictly necessary to form the
> message, I think it is quite a good sanity check if the
> message looks OK.

IMO, if we think the trailer is useful, then it should be required.
Making it optional only opens possibilities for misinterpretation.
Also, having to deal with two potential variations of trailers is not
optimal although I understand the OS differences. Yet another reason to
drop it.

BTW, previous syslog server implementations (Solaris, for instance)
allowed line breaks inside messages.  I think there was some discussion
about this before, but I am not sure what we concluded.  Should we allow
them in -protocol?

If not, I think we need some other mechanism to handle such messages to
ease migration to -protocol.  For example, we could extend the scope of
message fragmentation for this purpose.

Here is Solaris example of what appears in the log file if sending "\n":

Oct 11 22:14:15 myhost2 su: Message with line break\nbefore end

Now, example when sending "\r\n":

Oct 11 22:14:15 myhost2 su: Message with line break^M\nbefore end

Thanks,
Anton.

------------------------------

Date: Wed, 28 Jan 2004 18:43:23 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: -international: trailer

> -----Original Message-----
> From: Anton Okmianski [mailto:[email-masked]]
> Sent: Wednesday, January 28, 2004 6:19 PM
> To: Rainer Gerhards; [email-masked]
> Subject: RE: -international: trailer
>
> Rainer:
>
> > as I am editing -international, I noticed that the TRAILER
> > specified may be worth looking at. Now that we do not need to
> > remain backward-compatible, I would opt to have a simple
> > tailer in the message. It was defined as:
> >
> >  SYSLOG-MSG      = PRI HEADER MSG [TRAILER]
> >  TRAILER         = [CR] LF
> >
> > While the trailer is not strictly necessary to form the
> > message, I think it is quite a good sanity check if the
> > message looks OK.
>
> IMO, if we think the trailer is useful, then it should be required.
> Making it optional only opens possibilities for misinterpretation.
> Also, having to deal with two potential variations of trailers is not
> optimal although I understand the OS differences. Yet another
> reason to
> drop it.

Agree...

how about this

SYSLOG-MSG      = PRI HEADER MSG TRAILER
TRAILER         = LF

I see the benefit in a way to verify (a little) that we actually have
received the full message. And don't take the LF as something I insist
on. It may be any value that is not allowed in MSG itself. As it looks,
LF may not be a good candidate for this...

> BTW, previous syslog server implementations (Solaris, for instance)
> allowed line breaks inside messages.

oops ... really?

> I think there was some
> discussion
> about this before, but I am not sure what we concluded.

I have used the search engine on my private archive (which seems to have
an issue with returning to the right threading view). I think this was
the discussion:

http://www.syslog.cc/ietf/autoarc/msg00865.html

I went quickly over it, and it was not resolved. However, I said that LF
in messages are a bad thing and nobody objected ;) I am still of this
feeling, because I see a lot of issues on the log parser side if we
allow multiple lines for a single message.

> Should we allow
> them in -protocol?
>
> If not, I think we need some other mechanism to handle such
> messages to
> ease migration to -protocol.  For example, we could extend
> the scope of
> message fragmentation for this purpose.
>
> Here is Solaris example of what appears in the log file if
> sending "\n":
>
> Oct 11 22:14:15 myhost2 su: Message with line break\nbefore end

ah, ok, so solaris does this processing, obviously when writing to the
file. That would mean that we would need to apply escaping to the free
form part of the text, too. Not a bad idea... Especially as I think the
\x form will seldomly hit sombody. Maybe we should also allow \#HX with
HX being the byte to be represented in hex. Doing so opens up the
ability to actually send anything as printable text in a standards way.
Sounds like a good thing...

Any comments on that?

Rainer

------------------------------

Date: Thu, 29 Jan 2004 10:43:15 +1300
From: Andrew Ross <[email-masked]>
Subject: RE: -international: trailer - escaping chrs

Since syslog messages are usually logged to a flat file, and each line
in the file is terminated by CRLF (or just LF or CR), I think we should
escape all control chrs in the message text. Currently Kiwi Syslog does
this by converting it to <013> type format. It would be good if we could
reach a consensus on the best way to do this for -protocol. Having
control chrs like CR and LF in the message text would break a lot of log
file parsers.

We could use \#HX or \#HXHX for double byte chr sets. The advantage to
using <013> is that you have a start and end delimiter. So if we used
\#HX then it would either need to be fixed length, or followed by a
known delimiter (space). This means that the message can be converted
back at a later date if required.

Do we mandate that -protocol messages MUST not contain ASCII values less
than &H20 ? Or do we say that the receiving syslog daemon is expected to
escape all invalid chrs. Personally, I can't see a reason why you would
want to use chrs < &H20 in a message. Can anyone think of a good reason?

Cheers

Andrew

- -----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Rainer Gerhards
Sent: Thursday, 29 January 2004 6:43 a.m.
To: Anton Okmianski; [email-masked]
Subject: RE: -international: trailer

> -----Original Message-----
> From: Anton Okmianski [mailto:[email-masked]]
> Sent: Wednesday, January 28, 2004 6:19 PM
> To: Rainer Gerhards; [email-masked]
> Subject: RE: -international: trailer
>
> Rainer:
>
> > as I am editing -international, I noticed that the TRAILER
> > specified may be worth looking at. Now that we do not need to
> > remain backward-compatible, I would opt to have a simple
> > tailer in the message. It was defined as:
> >
> >  SYSLOG-MSG      = PRI HEADER MSG [TRAILER]
> >  TRAILER         = [CR] LF
> >
> > While the trailer is not strictly necessary to form the
> > message, I think it is quite a good sanity check if the
> > message looks OK.
>
> IMO, if we think the trailer is useful, then it should be required.
> Making it optional only opens possibilities for misinterpretation.
> Also, having to deal with two potential variations of trailers is not
> optimal although I understand the OS differences. Yet another
> reason to
> drop it.

Agree...

how about this

SYSLOG-MSG      = PRI HEADER MSG TRAILER
TRAILER         = LF

I see the benefit in a way to verify (a little) that we actually have
received the full message. And don't take the LF as something I insist
on. It may be any value that is not allowed in MSG itself. As it looks,
LF may not be a good candidate for this...

> BTW, previous syslog server implementations (Solaris, for instance)
> allowed line breaks inside messages.

oops ... really?

> I think there was some
> discussion
> about this before, but I am not sure what we concluded.

I have used the search engine on my private archive (which seems to have
an issue with returning to the right threading view). I think this was
the discussion:

http://www.syslog.cc/ietf/autoarc/msg00865.html

I went quickly over it, and it was not resolved. However, I said that LF
in messages are a bad thing and nobody objected ;) I am still of this
feeling, because I see a lot of issues on the log parser side if we
allow multiple lines for a single message.

> Should we allow
> them in -protocol?
>
> If not, I think we need some other mechanism to handle such
> messages to
> ease migration to -protocol.  For example, we could extend
> the scope of
> message fragmentation for this purpose.
>
> Here is Solaris example of what appears in the log file if
> sending "\n":
>
> Oct 11 22:14:15 myhost2 su: Message with line break\nbefore end

ah, ok, so solaris does this processing, obviously when writing to the
file. That would mean that we would need to apply escaping to the free
form part of the text, too. Not a bad idea... Especially as I think the
\x form will seldomly hit sombody. Maybe we should also allow \#HX with
HX being the byte to be represented in hex. Doing so opens up the
ability to actually send anything as printable text in a standards way.
Sounds like a good thing...

Any comments on that?

Rainer

------------------------------

Date: Thu, 29 Jan 2004 11:01:17 +1300
From: Andrew Ross <[email-masked]>
Subject: RE: Facilities and Severity

In order for us to map older syslog into the new -protocol, we should at
least reference the existing facilities and levels that are mentioned in
3164. We need a cross reference table in the spec so we can map old to
new and back again. Also, make note of how to map new to old. For
example, if the new code does not directly map to an old value (via the
cross reference table) it must be set to Local7.Info (or what ever).

Andrew

- -----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Rainer Gerhards
Sent: Thursday, 29 January 2004 5:24 a.m.
To: [email-masked]
Subject: Facilities and Severity

Hi WG,

again on the edit: I am replacing the few fixed facilities by a 4-digit
number right now. I wonder if it would still make sense to mention the
traditional facilities in -protocol. Same goes for severity, where we
now also allow additional values.

I would appreciate comments on this. For now, I will remove any
references. This does not imply that I think they should be removed. I
am actually half-half.

If that is concensus will be that the traditional values should be
mentioned, I'll move them back in either in -02 (if comments arrive
before end of edit) or -03 (if not).

Thanks,
Rainer

------------------------------

Date: Wed, 28 Jan 2004 15:26:54 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: Facilities and Severity

Rainer:

1. I would propose leaving severities as is & with descriptions.  I
don't think deviating from widely adopted convention gives much of a
benefit here.  Did somebody ask for a change?

2. I do agree that legacy facilities need to go. Original legacy
facilities came out of system-local use (kernel, printer, system
daemons, user-level messages, etc).  Essentially, it was a set of
standard facilities for one specific product - BSD OS.  Whereas we are
defining a protocol which is much more broadly applicable.

3. I think defining any *standard* facilities (legacy or not) will be
very difficult. For example, for one product it may be acceptable to use
just one mail facility, while another mail server product may wish to
send messages from different subsystems to different facilities. So,
facilities can't be required, only suggested anyway.

A broader question... What is a facility? And how do we recommend people
use it? I did not see anything about this in the -protocol.

Is this something that could be more appropriately called a "logging
channel"? This removes the vague idea people may have that this is
something like a product or a subsystem.  It could be any arbitrary
grouping of messages.  Essentially, it does not have an inherent meaning
- -- it looks like it is just another tag, only restricted numerics and
placed at the front of the message for easy parsing.

Thanks,
Anton.

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Rainer Gerhards
> Sent: Wednesday, January 28, 2004 11:24 AM
> To: [email-masked]
> Subject: Facilities and Severity
>
>
> Hi WG,
>
> again on the edit: I am replacing the few fixed facilities by
> a 4-digit number right now. I wonder if it would still make
> sense to mention the traditional facilities in -protocol.
> Same goes for severity, where we now also allow additional values.
>
> I would appreciate comments on this. For now, I will remove
> any references. This does not imply that I think they should
> be removed. I am actually half-half.
>
> If that is concensus will be that the traditional values
> should be mentioned, I'll move them back in either in -02 (if
> comments arrive before end of edit) or -03 (if not).
>
> Thanks,
> Rainer
>
>

------------------------------

Date: Wed, 28 Jan 2004 16:29:59 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Hi WG,

while editing -protocol to 02, I also work through Anton's early
comments with a lot of good points. I hadn't commented on them so far
because we were on the broader topics. Most of the comments now smoothly
go into the new version, but I found this here that I would like to
bring to the list:

> 23. Section 5 & beyond. Why is there a need to specified
> structured data
> *anywhere* within the message.  I thought we will designate a special
> field like TAG for the structured data.  This way we won't need a
> special sequence to identify it.  Also, I think allowing it everywhere
> gives too much unnecessary freedom. Harder to evolve protocol later.

I did this to allow app software layers/libraries an easy way to add its
information. But we could also specify that it MUST be immediatley after
the tag, with no SPs between the structured data elements.

On the special sequence, I think that is actually still mandatory.
Otherwise it may be mistaken by a non structured data element which
co-incidently begins at that location. An alternative, of course, would
be to supply a special sequence (eg "[]") to indicate that no structured
data elements are present.

I will leave it as is in -02, except if we reach concensus on this topic
quicker than I edit...

Comments are appreciated.

Rainer

------------------------------

Date: Thu, 29 Jan 2004 12:21:20 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

David,

I am not so much in SNMP, so please bear with me...

> I suggest that you should consider adding two fields to a standardized
> syslog format: the (IANA-assigned) enterprise ID and a version number.
> (The IETF should have an enterprise ID, and I know other IETF work has
> considered 0 to be the IETF enterpriseID, but check with
> Joyce Reynolds
> whether 0 should be used for this or not.)
>
> This would allow vendors to migrate their existing
> proprietary versions
> of syslog to an intermediate standardized message format, without
> forcing them to go all the way to fully standardized content. Some
> syslog messages may be encoded in hardware, or in licensed binary
> software, or in code the vendor cannot economically justify
> changing all
> at once, and will be difficult to migrate the content to a
> fully-standard syslog format. However, the embedded software that
> creates the final messages for the wire may be able to wrap
> the content
> in a standarized message format, improving manageability and
> interoperability.

If a message is fully compliant with the spec, should we then use the
IETF enterprise ID or should we still use the vendor's enterprise ID? I
guess still the vendor enterprise ID, because if it weren't fully
compliant, the syslogd MUST assume an error and not process the message.
Am I right here?

Rainer

------------------------------

Date: Thu, 29 Jan 2004 15:25:25 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: Facilities and Severity

Andrew,

> In order for us to map older syslog into the new -protocol,
> we should at
> least reference the existing facilities and levels that are
> mentioned in
> 3164.

based on the discussion, I now think it is not strictly necessary to
specify the table.

I think one of Anton's core ideas is that the facility should not be
tied to any specific meaning. A facility is just a quick way to filter
messages based on a simple, numerical ID.

From what I see at customer installations, this is probably a good
description (but I also see that the traditional facilities are widely
used in local sysadmin, e.g. for the mail subsystem).

With the current version I have under edit, I have not assigned any
specific semantical meaning to the facilities, but I have recommended to
stay away from the traditional facilities (< 24) unless specifically
instructed by the user.

> We need a cross reference table in the spec so we can map old to
> new and back again. Also, make note of how to map new to old. For
> example, if the new code does not directly map to an old
> value (via the
> cross reference table) it must be set to Local7.Info (or what ever).

Actually, there is no mapping to be done. We no longer have this PRI
type of thing, that is a single value which specifies two things
(facility and severity). Instead, we have two fields, one being the
facility (in current in-edit draft unsigned int 32) and one being the
severity (in current in-edit draft only values 0..7 - the traditional
ones).

So there is is need to map it specifically (well, a relay receiving a
3164 message may map a little, but I assume it will simply split the
fields - that will go into yet to be written "relay operations"
section).

Rainer

------------------------------

Date: Thu, 29 Jan 2004 09:28:03 -0800
From: Darren New <[email-masked]>
Subject: Re: Advice on Trademark Issues requested

Rainer Gerhards wrote:
 > was tempted to replace it by the abbreviation "SDE" in the protcol-02
darft. However, I found that this is a trademark.

I'm not a lawyer. However, it's a question of what it's trademarking.
Trademarks are intended to distinguish where the product comes from, and
trademark infringement comes when someone's using a trademark for their
product that might confuse consumers into thinking it's someone else's
product. If there's a trademarked laundry soap called SDE, it's unlikely
to be confused with the letters SDE in a protocol description for a
computer program. This is how you have Apple Computers and Apple
Records, and the problem only came up when Apple Computers started
selling computers that could play music.

There's also a situation with "Well known" trademarks. You cannot, for
example, call your car company McDonalds, even tho an automobile is
unlikely to be confused with a burger joint.

Of course, all this is a layman's understanding of US trademark law.
YMMV. Consult a real lawyer before getting sued.

- -- 
Darren New, San Diego CA USA (PST)
   I am in geocentric orbit, supported by
   a quantum photon exchange drive....

------------------------------

Date: Thu, 29 Jan 2004 12:36:00 +0100
From: Rainer Gerhards <[email-masked]>
Subject: change of facilities

Hi WG,

just a by-note, so that everybody is aware of this: we are changing the
facility from a few defined values to a very broad range (unsigned int
32, if nobody objects). Please note that some existing implementations
of syslog (namely at least the stock linux versions) use a table of PRI
values to guide there processing.

Adapting them to the new scheme will require considerate redesign of
that part of the code. As the whole syslogd is just around ~3000 lines
of not-so-complex code, I think that should not stop us from making
these changes.

I just wanted to make sure that everybody is aware of this implication.

Rainer

------------------------------

Date: Thu, 29 Jan 2004 13:21:02 -0800
From: Devin Kowatch <[email-masked]>
Subject: Re: -international: trailer - escaping chrs

For escaping characters I would suggest using a comment escape syntax.
For example, C does character escapes as \(o|x|)DDD, or if you want a
begining and ending, perhaps the XML escape &#XXXX; (I think).  Anyway
the point use an escape sequence that is common rather than making up
our own.  It just confuses people less.

Also, using <XXX> will make some wonder what XML/HTML is doing in their
log messages :)

On Thu, Jan 29, 2004 at 10:43:15AM +1300, Andrew Ross wrote:
>
> Since syslog messages are usually logged to a flat file, and each line
> in the file is terminated by CRLF (or just LF or CR), I think we should
> escape all control chrs in the message text. Currently Kiwi Syslog does
> this by converting it to <013> type format. It would be good if we could
> reach a consensus on the best way to do this for -protocol. Having
> control chrs like CR and LF in the message text would break a lot of log
> file parsers.
>
> We could use \#HX or \#HXHX for double byte chr sets. The advantage to
> using <013> is that you have a start and end delimiter. So if we used
> \#HX then it would either need to be fixed length, or followed by a
> known delimiter (space). This means that the message can be converted
> back at a later date if required.
>
> Do we mandate that -protocol messages MUST not contain ASCII values less
> than &H20 ? Or do we say that the receiving syslog daemon is expected to
> escape all invalid chrs. Personally, I can't see a reason why you would
> want to use chrs < &H20 in a message. Can anyone think of a good reason?
>
> Cheers
>
> Andrew
>
>
>
>
>
> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Rainer Gerhards
> Sent: Thursday, 29 January 2004 6:43 a.m.
> To: Anton Okmianski; [email-masked]
> Subject: RE: -international: trailer
>
>
> > -----Original Message-----
> > From: Anton Okmianski [mailto:[email-masked]]
> > Sent: Wednesday, January 28, 2004 6:19 PM
> > To: Rainer Gerhards; [email-masked]
> > Subject: RE: -international: trailer
> >
> > Rainer:
> >
> > > as I am editing -international, I noticed that the TRAILER
> > > specified may be worth looking at. Now that we do not need to
> > > remain backward-compatible, I would opt to have a simple
> > > tailer in the message. It was defined as:
> > >
> > >  SYSLOG-MSG      = PRI HEADER MSG [TRAILER]
> > >  TRAILER         = [CR] LF
> > >
> > > While the trailer is not strictly necessary to form the
> > > message, I think it is quite a good sanity check if the
> > > message looks OK.
> >
> > IMO, if we think the trailer is useful, then it should be required.
> > Making it optional only opens possibilities for misinterpretation.
> > Also, having to deal with two potential variations of trailers is not
> > optimal although I understand the OS differences. Yet another
> > reason to
> > drop it.
>
> Agree...
>
> how about this
>
> SYSLOG-MSG      = PRI HEADER MSG TRAILER
> TRAILER         = LF
>
> I see the benefit in a way to verify (a little) that we actually have
> received the full message. And don't take the LF as something I insist
> on. It may be any value that is not allowed in MSG itself. As it looks,
> LF may not be a good candidate for this...
>
> > BTW, previous syslog server implementations (Solaris, for instance)
> > allowed line breaks inside messages.
>
> oops ... really?
>
> > I think there was some
> > discussion
> > about this before, but I am not sure what we concluded.
>
> I have used the search engine on my private archive (which seems to have
> an issue with returning to the right threading view). I think this was
> the discussion:
>
> http://www.syslog.cc/ietf/autoarc/msg00865.html
>
> I went quickly over it, and it was not resolved. However, I said that LF
> in messages are a bad thing and nobody objected ;) I am still of this
> feeling, because I see a lot of issues on the log parser side if we
> allow multiple lines for a single message.
>
> > Should we allow
> > them in -protocol?
> >
> > If not, I think we need some other mechanism to handle such
> > messages to
> > ease migration to -protocol.  For example, we could extend
> > the scope of
> > message fragmentation for this purpose.
> >
> > Here is Solaris example of what appears in the log file if
> > sending "\n":
> >
> > Oct 11 22:14:15 myhost2 su: Message with line break\nbefore end
>
> ah, ok, so solaris does this processing, obviously when writing to the
> file. That would mean that we would need to apply escaping to the free
> form part of the text, too. Not a bad idea... Especially as I think the
> \x form will seldomly hit sombody. Maybe we should also allow \#HX with
> HX being the byte to be represented in hex. Doing so opens up the
> ability to actually send anything as printable text in a standards way.
> Sounds like a good thing...
>
> Any comments on that?
>
> Rainer
>
>
>
>

- -- 
Devin Kowatch
[email-masked]

------------------------------

Date: Thu, 29 Jan 2004 17:24:57 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: -international: trailer

Rainer:

>> Oct 11 22:14:15 myhost2 su: Message with line break\nbefore end

> ah, ok, so solaris does this processing, obviously when writing
> to the file. That would mean that we would need to apply escaping
> to the free form part of the text, too. Not a bad idea...
> Especially as I think the \x form will seldomly hit sombody. Maybe
> we should also allow \#HX with HX being the byte to be represented
> in hex. Doing so opens up the ability to actually send anything as
> printable text in a standards way. Sounds like a good thing...

> Any comments on that?

Can we just allow any ASCII control characters and any UTF-8 encoded
characters and leave it up to a different standard to define the storage
format?

I think storage format must be standardized soon, but it is out of scope
for -protocol, right? People will be looking to -protocol and try to
infer the log file format so they can process messages.  We should make
it clear in -protocol that syslog servers are free to store messages in
whatever format they choose and that storage format is out of scope. Or
explicitly make it part of the scope.

There is a benefit to leaving it up to the server to do the extra
reformatting.  This way clients running inside applications whose
primary responsibility is not logging can be made dumber and hence run
less of risk of performance issues or a failure during logging of a
critical message.

If we do restrict new lines though, I feel we must provide some
alternative.

Anton.

------------------------------

Date: Fri, 30 Jan 2004 20:29:29 +0900
From: Glenn Mansfield Keeni <[email-masked]>
Subject: Re: SyslogMIB Issue-#4

Hi Andrew,
   Getting the high resolution timestamp should not be a problem. And
I think that we can agree to constrain the length of the message that
will be carried in the trap.

   Glenn
> Hi Glenn,
>
> That system looks good. Something to map syslog messages into SNMP traps
> and another one for notifications.
>
> Let's make the timestamp high resolution like the new proposed syslog
> RFC that Rainer is working on.
>
> Leave a string binding in the notification trap for error message
> descriptions that we can't plan for.
>
> Maybe set a binding in the syslogCtlSnmpTrapActionTable that indicates a
> multi part message. Or just constrain the length to 1024 again??
>
> Regards
>
> Andrew
>
> PS. More comments on your other 6 points will follow when I get my act
> together. :-)
>
>
>
>
> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Glenn Mansfield
> Keeni
> Sent: Monday, 26 January 2004 3:05 a.m.
> To: Harrington, David
> Cc: [email-masked]
> Subject: Re: SyslogMIB Issue-#4
>
>
> Hi Dave,
>     Thanks for the comments. I agree with you. Traps are useful.
> They provide a uniform mechanism/console for the network manager
> to manage the network, related devices and processes.
>     There are two levels at which we may want SNMP traps in the
> Syslog Messaging context.
>     a. To provide another option of how a message will be
>        processed. Presently we have
>           - log (syslogCtlLogActionTable)
>           - display on users console (syslogCtlUserActionTable)
>           - relay/forward (syslogCtlFwdActionTable)
>           - pipe (syslogCtlPipeActionTable)
>        We can add snmp trap to this with relative ease. It is
>        just a matter of adding another table
>        e.g. syslogCtlSnmpTrapActionTable
>        I would propose that the contents of the trap would be
>               o the syslog message itself. Perhaps the length will
>                 need to be constrained.
>               o SyslogSeverity,
>               o syslogFacility,
>               o message source
>               o timestamp.
>     b. To notify the syslog/System administrator of notable events
>        e.g.   o failure to process a message
>                     relay, log, display etc
>               o too many rejected messages
>               o anything else ?
>
>
>   Any comments, suggestions ?
>
>   Cheers
>
>   Glenn
>
> Harrington, David wrote:
>
>>Hi Glenn,
>>
>>The benefit of having an SNMP trap is that there becomes some
>>coordination between the syslog messaging and SNMP trap manager
>>applications.
>>
>>Operators often use trap management tools to alert them to problems,
>
> and
>
>>these applications often have the ability to filter out uninteresting
>>traps, to sort by priority, to do fault isolation, and so on. Trap
>>management utilities can also feed into SNMP-based topology mapping
>>tools (like HPOV) which can use colored icons to represent serious
>>problems, and so on.
>>
>>The problem with a syslog notification is - what would it contain? You
>>could write one syslog notification for reporting all syslog events by
>>simply having a varbind with the complete syslog message. That means
>>that an SNMP trap application cannot do much in terms of sorting by
>>priority. You could create a notification that ocntains some
>>standardized values common to syslog messages, such as severity,
>
> related
>
>>subsystem, vendor, and then include the human-readable string in an
>>octet string. This might be useful, but it might be less useful than
>>notifications specific to a technology.
>>
>>I think the WG needs to discuss the nature of potential notification
>>designs.
>>
>>Personally I think such a generalized trap might be a good thing for
>>improving coordination between syslog and SNMP, but it will depend a
>>great deal on the actual design and how easily that design can be
>>utilized by applications.
>>
>>dbh
>>
>>
>>
>>>-----Original Message-----
>>>From: Glenn Mansfield Keeni [mailto:[email-masked]]
>>>Sent: Friday, January 16, 2004 4:48 AM
>>>To: [email-masked]
>>>Subject: SyslogMIB Issue-#4
>>>
>>>
>>>Issue #4. Snmp Notifications ? Do we need any ?
>>>
>>>
>>>Action: Do we need any?
>>>        Note- Syslog itself is a notification.
>>>        If there a clear requirement for a notification
>>>        that can added.
>>>
>>>Status: Waiting on community input
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
>
>
>
>

------------------------------

Date: Fri, 30 Jan 2004 22:08:35 +0900
From: Glenn Mansfield Keeni <[email-masked]>
Subject: Re: SyslogMIB Issue-#4

Hi Rainer,
   This pretty much matches with the image I have in mind.
Till there is no "original sender" in the syslog message
itself I guess that we will have to do with the "last sender".

    Glenn
Rainer Gerhards wrote:

> Hi all,
>
> I have to admit I am not following that closely with SNMP due being busy
> with -protocol.
>
> As it looks SNMP trap may be useful, I would suggest that this is
> implemented as a transport mapping for -protocol. -protocol is transport
> ignorant by design. Some of the structured data elements may be mapped
> to SNMP properties, as are the header fields. So we already have a
> number of well-defined values.
>
> The ability of mapping syslog message to different transports was a key
> thing that made us create -protocol, so we could utilize it here.
> Eventually, it would make sense to create some few additional (option)
> structure data elements, like the IP address of the original sender.
> This may be useful for mapping onto SNMP (if element is provided by
> emitor).
>
> Rainer
>
>
>>-----Original Message-----
>>From: Andrew Ross [mailto:[email-masked]]
>>Sent: Sunday, January 25, 2004 11:10 PM
>>To: 'Glenn Mansfield Keeni'; 'Harrington, David'
>>Cc: [email-masked]
>>Subject: RE: SyslogMIB Issue-#4
>>
>>
>>Hi Glenn,
>>
>>That system looks good. Something to map syslog messages into
>>SNMP traps
>>and another one for notifications.
>>
>>Let's make the timestamp high resolution like the new proposed syslog
>>RFC that Rainer is working on.
>>
>>Leave a string binding in the notification trap for error message
>>descriptions that we can't plan for.
>>
>>Maybe set a binding in the syslogCtlSnmpTrapActionTable that
>>indicates a
>>multi part message. Or just constrain the length to 1024 again??
>>
>>Regards
>>
>>Andrew
>>
>>PS. More comments on your other 6 points will follow when I get my act
>>together. :-)
>>
>>
>>
>>
>>-----Original Message-----
>>From: [email-masked]
>>[mailto:[email-masked]] On Behalf Of Glenn Mansfield
>>Keeni
>>Sent: Monday, 26 January 2004 3:05 a.m.
>>To: Harrington, David
>>Cc: [email-masked]
>>Subject: Re: SyslogMIB Issue-#4
>>
>>
>>Hi Dave,
>>    Thanks for the comments. I agree with you. Traps are useful.
>>They provide a uniform mechanism/console for the network manager
>>to manage the network, related devices and processes.
>>    There are two levels at which we may want SNMP traps in the
>>Syslog Messaging context.
>>    a. To provide another option of how a message will be
>>       processed. Presently we have
>>          - log (syslogCtlLogActionTable)
>>          - display on users console (syslogCtlUserActionTable)
>>          - relay/forward (syslogCtlFwdActionTable)
>>          - pipe (syslogCtlPipeActionTable)
>>       We can add snmp trap to this with relative ease. It is
>>       just a matter of adding another table
>>       e.g. syslogCtlSnmpTrapActionTable
>>       I would propose that the contents of the trap would be
>>              o the syslog message itself. Perhaps the length will
>>                need to be constrained.
>>              o SyslogSeverity,
>>              o syslogFacility,
>>              o message source
>>              o timestamp.
>>    b. To notify the syslog/System administrator of notable events
>>       e.g.   o failure to process a message
>>                    relay, log, display etc
>>              o too many rejected messages
>>              o anything else ?
>>
>>
>>  Any comments, suggestions ?
>>
>>  Cheers
>>
>>  Glenn
>>
>>Harrington, David wrote:
>>
>>>Hi Glenn,
>>>
>>>The benefit of having an SNMP trap is that there becomes some
>>>coordination between the syslog messaging and SNMP trap manager
>>>applications.
>>>
>>>Operators often use trap management tools to alert them to problems,
>>
>>and
>>
>>>these applications often have the ability to filter out
>>
>>uninteresting
>>
>>>traps, to sort by priority, to do fault isolation, and so on. Trap
>>>management utilities can also feed into SNMP-based topology mapping
>>>tools (like HPOV) which can use colored icons to represent serious
>>>problems, and so on.
>>>
>>>The problem with a syslog notification is - what would it
>>
>>contain? You
>>
>>>could write one syslog notification for reporting all
>>
>>syslog events by
>>
>>>simply having a varbind with the complete syslog message. That means
>>>that an SNMP trap application cannot do much in terms of sorting by
>>>priority. You could create a notification that ocntains some
>>>standardized values common to syslog messages, such as severity,
>>
>>related
>>
>>>subsystem, vendor, and then include the human-readable string in an
>>>octet string. This might be useful, but it might be less useful than
>>>notifications specific to a technology.
>>>
>>>I think the WG needs to discuss the nature of potential notification
>>>designs.
>>>
>>>Personally I think such a generalized trap might be a good thing for
>>>improving coordination between syslog and SNMP, but it will depend a
>>>great deal on the actual design and how easily that design can be
>>>utilized by applications.
>>>
>>>dbh
>>>
>>>
>>>
>>>>-----Original Message-----
>>>>From: Glenn Mansfield Keeni [mailto:[email-masked]]
>>>>Sent: Friday, January 16, 2004 4:48 AM
>>>>To: [email-masked]
>>>>Subject: SyslogMIB Issue-#4
>>>>
>>>>
>>>>Issue #4. Snmp Notifications ? Do we need any ?
>>>>
>>>>
>>>>Action: Do we need any?
>>>>        Note- Syslog itself is a notification.
>>>>        If there a clear requirement for a notification
>>>>        that can added.
>>>>
>>>>Status: Waiting on community input
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
>
>

------------------------------

Date: Fri, 30 Jan 2004 14:22:24 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: SyslogMIB Issue-#4

> Hi Rainer,
>    This pretty much matches with the image I have in mind.
> Till there is no "original sender" in the syslog message
> itself I guess that we will have to do with the "last sender".

I assume we are talking about a syslog relay chain BEFORE the realy that
issues the trap here. Actually, in this case, it is the original sender,
because a relay is not allowed to modify the message - so the hostname
will always be that of the original sender.

Of course, if a syslog message is transformed to some other transport
and the retransformed (at some later stage) to a syslog message, the
sender might have changed. However, this has more implications, eg
message signatures would be broken. So any architecture converting
syslog messages should try hard NOT to alter the message.

But in the context of this discussion /transport mapping to snmp/ I
think we can assume that it is the original sender.

Rainer

------------------------------

Date: Fri, 30 Jan 2004 15:20:17 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: SyslogMIB Issue-#6

Sorry for the late follow up...

I am not sure if we really desire this. As of my knowledge, most
existing syslogds have far more sophisticated rules than the stock
syslogd implementtions with just filtering on facility and severity. And
they have, because there is a customer demand.

You now can argue that by providing the generic configuration way, an
administrator can at least configure all devices (in a vendor-ignorant
way) to some desired common basic configuration. I just doubt that this
"basic configuration" would just be too limited to be actually useful
for the administrator. So I am not sure if it is really worth putting
this effort in....

I have no strong opinion on this, though. But I am a bit on the "we
should stick with read-only" side...

Rainer

> -----Original Message-----
> From: Glenn Mansfield Keeni [mailto:[email-masked]]
> Sent: Friday, January 16, 2004 10:49 AM
> To: [email-masked]
> Subject: SyslogMIB Issue-#6
>
> Issue #6. Configuration Stuff - should it be there ?
>
> If the MIB is made read-only we cannot use it for configuration
> of syslog processes. That is a disadvantage if we wanted to
> configure our syslog processes in the first place. Otherwise,
> the consequences are
>       - implementations would be simpler
>       - security implications are lesser
>       - passage through the IESG-review process may be smoother (?)
>
> So, do we want to be able to configure Syslog processes? Community
> input is required here.
>
> Status: Waiting on community input

------------------------------

Date: Fri, 30 Jan 2004 16:10:43 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: SyslogMIB Issue-#4

>    Getting the high resolution timestamp should not be a problem. And
> I think that we can agree to constrain the length of the message that
> will be carried in the trap.

In -protocol, the max length is sill FIXED at 1024 per message fragment.
If a message is large, it MUST be fragmented into multiple fragments.
Each of these fragments MUST be 1024 chars max. So you can safely set
the max length to 1024 - otherwise we have an invalid syslog message.
The same applies in principle to old-style syslog and is defined in RFC
3164 (though some implementations do not abide to this limit).

Rainer

------------------------------

Date: Fri, 30 Jan 2004 11:03:16 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Rainer:

I hope I have annoyed you too much with my non-stop comments, but I
think we are making good progress!

> > 23. Section 5 & beyond. Why is there a need to specified
> > structured data
> > *anywhere* within the message.  I thought we will designate
> a special
> > field like TAG for the structured data.  This way we won't need a
> > special sequence to identify it.  Also, I think allowing it
> everywhere
> > gives too much unnecessary freedom. Harder to evolve
> protocol later.
>
> I did this to allow app software layers/libraries an easy way
> to add its information. But we could also specify that it
> MUST be immediatley after the tag, with no SPs between the
> structured data elements.
>
> On the special sequence, I think that is actually still
> mandatory. Otherwise it may be mistaken by a non structured
> data element which co-incidently begins at that location. An
> alternative, of course, would be to supply a special sequence
> (eg "[]") to indicate that no structured data elements are present.

I don't think it is even necessary if we designate it as a field with
its own separator that follows it.  Suppose we want to say that
structured context goes after the TAG field (if we want to leave the
latter as is). Then, all we have to say is that if structured content is
not there, the field is blank, but it still has to be followed with a
separator (presumably a space).  So, if TAG is followed by 2 spaces, you
know structured content is missing and message starts at the next
character.  So it could be...

Oct 11 22:14:15 myhost2 su: [msgno=1] Message line

Or...

Oct 11 22:14:15 myhost2 su:  Message line

But I think an even better solution is to combine structured data and
the TAG into one field. The migration strategy for old client could use
a special structured content parameter to represent the old tag value.
This will also address my concern about TAG being nothing else than
another structured content field.

So, we could have:

Oct 11 22:14:15 myhost2 [tag=su][msgno=1] Message line

Or

Oct 11 22:14:15 myhost2  Message line

Two spaces after HOST mean there is structured content. I think this
approach is much cleaner.  And is actually closer to the old format.
Here the old TAG field is still separated with a space, except spaces
are also allowed inside the square brackets as well.

Thanks!

Anton.

------------------------------

Date: Fri, 30 Jan 2004 11:43:57 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

> Rainer:
>
> I hope I have annoyed you too much with my non-stop comments,
> but I think we are making good progress!

I meant "I hope I have NOT annoyed you", obviously. :)

Sorry.

Anton.

------------------------------

Date: Fri, 30 Jan 2004 12:43:51 -0500
From: "Pape, Bruno" <[email-masked]>
Subject: RE: SyslogMIB Issue-#6

From the client side there could be value in SNMP configuration,
either for configuring the appropriate syslog servers for 10's,
100's, or 1000's of Unix desktops or embedded systems, or for
controlling the severity of the messages forwarded by specific
applications on those devices.

In the networking equipment space being able to manage the level
of messages sent by particular applications, and controlling which
servers those messages are sent to could be part of a valuable tool
for debugging application specific issues in certain environments.

Greetings,

Bruno

- -----Original Message-----
From: Rainer Gerhards [mailto:[email-masked]]
Sent: Friday, January 30, 2004 9:20 AM
To: Glenn Mansfield Keeni; [email-masked]
Subject: RE: SyslogMIB Issue-#6

Sorry for the late follow up...

I am not sure if we really desire this. As of my knowledge, most
existing syslogds have far more sophisticated rules than the stock
syslogd implementtions with just filtering on facility and severity. And
they have, because there is a customer demand.

You now can argue that by providing the generic configuration way, an
administrator can at least configure all devices (in a vendor-ignorant
way) to some desired common basic configuration. I just doubt that this
"basic configuration" would just be too limited to be actually useful
for the administrator. So I am not sure if it is really worth putting
this effort in....

I have no strong opinion on this, though. But I am a bit on the "we
should stick with read-only" side...

Rainer

> -----Original Message-----
> From: Glenn Mansfield Keeni [mailto:[email-masked]]
> Sent: Friday, January 16, 2004 10:49 AM
> To: [email-masked]
> Subject: SyslogMIB Issue-#6
>
> Issue #6. Configuration Stuff - should it be there ?
>
> If the MIB is made read-only we cannot use it for configuration
> of syslog processes. That is a disadvantage if we wanted to
> configure our syslog processes in the first place. Otherwise,
> the consequences are
>       - implementations would be simpler
>       - security implications are lesser
>       - passage through the IESG-review process may be smoother (?)
>
> So, do we want to be able to configure Syslog processes? Community
> input is required here.
>
> Status: Waiting on community input

------------------------------

Date: Fri, 30 Jan 2004 18:56:52 +0100
From: Rainer Gerhards <[email-masked]>
Subject: RE: SyslogMIB Issue-#4  // Issue-#2

Let me make a comment here that applies to issue #2 (BSD Centric)

> -----Original Message-----
> From: Glenn Mansfield Keeni [mailto:[email-masked]]
> Sent: Sunday, January 25, 2004 3:05 PM
> To: Harrington, David
> Cc: [email-masked]
> Subject: Re: SyslogMIB Issue-#4
>
> Hi Dave,
>     Thanks for the comments. I agree with you. Traps are useful.
> They provide a uniform mechanism/console for the network manager
> to manage the network, related devices and processes.
>     There are two levels at which we may want SNMP traps in the
> Syslog Messaging context.
>     a. To provide another option of how a message will be
>        processed. Presently we have
>           - log (syslogCtlLogActionTable)
>           - display on users console (syslogCtlUserActionTable)
>           - relay/forward (syslogCtlFwdActionTable)
>           - pipe (syslogCtlPipeActionTable)

In our products, we have many more options. I know for sure that Kiwi
syslogd also has many more options. I think many other vendors do have
other options, too. Upcoming versions of our *nix based syslogd will
definitely have more options. -sign may eventually require new actions
(verify signature? log diagnostic warning?) [this is speculated, just
for the records;)].

What I am trying to convey is that this is a very limited set and
different syslogd's may have totally different implementations. We can
stick with the minimal set of generic things. I am not deep enough in
SNMP to know if we (vendor) can define a mib to take care of the others
(I am sure we can ;)). But the question is how this would relate to the
"standard actions".

As a side note, the filterig/rule processing system is also totally
different in different implementations, this complicates things pretty
much. So it is probably best to stick with the minimal possibilities
from stock syslogd.

Rainer

------------------------------

Date: Fri, 30 Jan 2004 14:22:35 -0500
From: Anton Okmianski <[email-masked]>
Subject: RE: syslog-protocol-01 posted & comments

Rainer:

I made a comment earlier about how we should specify character set in
terms of Unicode and UTF-8.  You may want to reference RFC 2277, which
provides guidance on "IETF Policy on Character Sets and Languages".
Basically it says three things:

 - must support UTF-8 character set/encoding
 - must provide capability for identifying the language
 - must have section "Internationalization considerations"

I think internationalization should be part of -protocol because support
of UTF-8 already implies it. So, we need to add the last 2 things.

Identification of a language is tricky. I think we should not require
it.  For example, we need to be able to fire an authentication message
in English which contains a username in Japanese. So, how do we identify
the language such that one knows which language applies to which part?
I'd suggest we don't.

Maybe we should just define an optional multi-valued structured content
parameter like "lang" which would optionally list all languages used by
the message. We should support RFC 1766 language tags.

Thanks,
Anton.

------------------------------

Date: Sat, 31 Jan 2004 12:09:09 +1300
From: Andrew Ross <[email-masked]>
Subject: RE: SyslogMIB Issue-#6

I concur on this point. I think read only would be fine. The rules,
filters and settings in the new syslog products are too different and
complex to map into a generic SNMP system at this stage.

The only possibility I can think of would be the ability to dump the
rule set and configuration in the form of a text file. In this way, you
would rely on the syslog daemon app to process the text and extract
rules and settings from the file.

Sort of a GetConfigFile and SetConfigFile system. The size of the file
could be an issue.

Since most syslogs out there (WinSyslog, Kiwi Syslog, SDSC, Syslog-ng
etc) use a config file, or can be made to read from a config file, the
get/set system might work. You would have to place a meaningful tag at
the beginning of the file to identify the format etc.

Would that work for you Rainer?

Regards

Andrew

- -----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Rainer Gerhards
Sent: Saturday, 31 January 2004 3:20 a.m.
To: Glenn Mansfield Keeni; [email-masked]
Subject: RE: SyslogMIB Issue-#6

Sorry for the late follow up...

I am not sure if we really desire this. As of my knowledge, most
existing syslogds have far more sophisticated rules than the stock
syslogd implementtions with just filtering on facility and severity. And
they have, because there is a customer demand.

You now can argue that by providing the generic configuration way, an
administrator can at least configure all devices (in a vendor-ignorant
way) to some desired common basic configuration. I just doubt that this
"basic configuration" would just be too limited to be actually useful
for the administrator. So I am not sure if it is really worth putting
this effort in....

I have no strong opinion on this, though. But I am a bit on the "we
should stick with read-only" side...

Rainer

> -----Original Message-----
> From: Glenn Mansfield Keeni [mailto:[email-masked]]
> Sent: Friday, January 16, 2004 10:49 AM
> To: [email-masked]
> Subject: SyslogMIB Issue-#6
>
> Issue #6. Configuration Stuff - should it be there ?
>
> If the MIB is made read-only we cannot use it for configuration
> of syslog processes. That is a disadvantage if we wanted to
> configure our syslog processes in the first place. Otherwise,
> the consequences are
>       - implementations would be simpler
>       - security implications are lesser
>       - passage through the IESG-review process may be smoother (?)
>
> So, do we want to be able to configure Syslog processes? Community
> input is required here.
>
> Status: Waiting on community input

------------------------------