From internet-drafts@ietf.org Mon Feb 6 09:36:32 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 86B6321F8701; Mon, 6 Feb 2012 09:36:32 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.559 X-Spam-Level: X-Spam-Status: No, score=-102.559 tagged_above=-999 required=5 tests=[AWL=0.040, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id plQVJDOS77pg; Mon, 6 Feb 2012 09:36:31 -0800 (PST) Received: from ietfa.amsl.com (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6F3B621F8633; Mon, 6 Feb 2012 09:36:31 -0800 (PST) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: internet-drafts@ietf.org To: i-d-announce@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 3.64p1 Message-ID: <20120206173631.20235.75111.idtracker@ietfa.amsl.com> Date: Mon, 06 Feb 2012 09:36:31 -0800 Cc: sip-clf@ietf.org Subject: [sip-clf] I-D Action: draft-ietf-sipclf-problem-statement-10.txt X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:36:32 -0000 A New Internet-Draft is available from the on-line Internet-Drafts director= ies. This draft is a work item of the SIP Common Log Format Working Group o= f the IETF. Title : The Common Log Format (CLF) for the Session Initiation P= rotocol (SIP): Framework and Data Model Author(s) : Vijay K. Gurbani Eric W. Burger Tricha Anjali Humberto Abdelnur Olivier Festor Filename : draft-ietf-sipclf-problem-statement-10.txt Pages : 36 Date : 2012-02-06 Well-known web servers such as Apache and web proxies like Squid support event logging using a common log format. The logs produced using these de-facto standard formats are invaluable to system administrators for trouble-shooting a server and tool writers to craft tools that mine the log files and produce reports and trends. Furthermore, these log files can also be used to train anomaly detection systems and feed events into a security event management system. The Session Initiation Protocol (SIP) does not have a common log format, and as a result, each server supports a distinct log format that makes it unnecessarily complex to produce tools to do trend analysis and security detection. We propose a common log file format for SIP servers that can be used uniformly by user agents, proxies, registrars, redirect servers as well as back-to-back user agents. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-sipclf-problem-statement-10.= txt Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ This Internet-Draft can be retrieved at: ftp://ftp.ietf.org/internet-drafts/draft-ietf-sipclf-problem-statement-10.t= xt From vkg@bell-labs.com Mon Feb 6 09:45:47 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BFAD821F86E8 for ; Mon, 6 Feb 2012 09:45:47 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -108.802 X-Spam-Level: X-Spam-Status: No, score=-108.802 tagged_above=-999 required=5 tests=[AWL=1.797, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uCyqH-jujWiS for ; Mon, 6 Feb 2012 09:45:47 -0800 (PST) Received: from ihemail1.lucent.com (ihemail1.lucent.com [135.245.0.33]) by ietfa.amsl.com (Postfix) with ESMTP id 0752F21F86EC for ; Mon, 6 Feb 2012 09:45:46 -0800 (PST) Received: from usnavsmail4.ndc.alcatel-lucent.com (usnavsmail4.ndc.alcatel-lucent.com [135.3.39.12]) by ihemail1.lucent.com (8.13.8/IER-o) with ESMTP id q16HjiXl025490 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Mon, 6 Feb 2012 11:45:45 -0600 (CST) Received: from umail.lucent.com (umail-ce2.ndc.lucent.com [135.3.40.63]) by usnavsmail4.ndc.alcatel-lucent.com (8.14.3/8.14.3/GMO) with ESMTP id q16HjhoP009142 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Mon, 6 Feb 2012 11:45:44 -0600 Received: from shoonya.ih.lucent.com (shoonya.ih.lucent.com [135.185.238.235]) by umail.lucent.com (8.13.8/TPES) with ESMTP id q16Hjhau006704 for ; Mon, 6 Feb 2012 11:45:43 -0600 (CST) Message-ID: <4F3012C6.9030904@bell-labs.com> Date: Mon, 06 Feb 2012 11:49:58 -0600 From: "Vijay K. Gurbani" Organization: Bell Laboratories, Alcatel-Lucent User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20111222 Thunderbird/9.0 MIME-Version: 1.0 To: "sip-clf@ietf.org" Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.57 on 135.245.2.33 X-Scanned-By: MIMEDefang 2.64 on 135.3.39.12 Subject: [sip-clf] draft-ietf-sipclf-problem-statement-10: changes requested from IESG review X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:45:47 -0000 All: During the IESG review of draft-ietf-sipclf-problem-statement, there was considerable feedback on why the draft was designated as "Informational". The IESG felt that the draft went beyond being "Informational" since it actually defines both a problem statement (Section 3) and the data model (Section 8). Furthermore, the data model defined in this document was to be actually implemented by one or more representation drafts, and as such, they felt that making this document a standards-track was warranted. Clearly, as the work moved ahead, the problem-statement draft morphed from being only a problem statement to containing more concrete issues pertaining to the data model that the representation formats would adhere to. In that respect, I think the IESG is accurate. Therefore, the designation of the draft has been changed to reflect this; the new designation is that of a standards-track document. There were other modifications as well: - New paragraph added at the end of Section 2 to make the intent of this document clear, i.e., it serves as a problem statement and also contains a data model. - Section 5.2 adds some new text in relation to Wireshark. - Section 10 (Security Considerations) has been expanded in minor ways to address additional issues related to the privacy of logs at rest and logs in motion. - Normative language added to the considerations in Section 10. - Minor word changes as requested during IESG review and fixing typos. Version -10 is at http://tools.ietf.org/html/draft-ietf-sipclf-problem-statement-10 and a diff from -09 is at http://tools.ietf.org/rfcdiff?url2=draft-ietf-sipclf-problem-statement-10.txt Please review the changes so we can move the work ahead. Thanks, - vijay -- Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent 1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA) Email: vkg@{bell-labs.com,acm.org} / vijay.gurbani@alcatel-lucent.com Web: http://ect.bell-labs.com/who/vkg/ From gsalguei@cisco.com Wed Feb 8 16:25:43 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 89CCE11E80B3 for ; Wed, 8 Feb 2012 16:25:43 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.598 X-Spam-Level: X-Spam-Status: No, score=-10.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uEZNbSPwPdDh for ; Wed, 8 Feb 2012 16:25:41 -0800 (PST) Received: from av-tac-rtp.cisco.com (hen.cisco.com [64.102.19.198]) by ietfa.amsl.com (Postfix) with ESMTP id DCBFF11E80B1 for ; Wed, 8 Feb 2012 16:25:40 -0800 (PST) X-TACSUNS: Virus Scanned Received: from chook.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-rtp.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q190Pdfj011430 for ; Wed, 8 Feb 2012 19:25:40 -0500 (EST) Received: from dhcp-64-102-210-22.cisco.com (dhcp-64-102-210-22.cisco.com [64.102.210.22]) by chook.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q190PaFb023774; Wed, 8 Feb 2012 19:25:36 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-15-400101793 From: Gonzalo Salgueiro In-Reply-To: <4F21DD3E.7000002@nostrum.com> Date: Wed, 8 Feb 2012 19:25:36 -0500 Message-Id: <31A5C897-B767-4527-9346-905A80977F35@cisco.com> References: <4F21DD3E.7000002@nostrum.com> To: Robert Sparks X-Mailer: Apple Mail (2.1084) Cc: draft-ietf-sipclf-format@tools.ietf.org, sipclf-chairs@tools.ietf.org, "sip-clf@ietf.org Mailing" Subject: Re: [sip-clf] AD review: draft-ietf-sipclf-format-05 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 00:25:43 -0000 --Apple-Mail-15-400101793 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Robert -=20 Thanks for your thorough and thoughtful review. Below are some inline = comments to your feedback.=20 I will release the -06 version of the draft once we have settled on all = of the below points you have raised. On Jan 26, 2012, at 6:09 PM, Robert Sparks wrote: > AD review: draft-ietf-sipclf-format-05 >=20 > Summary: This document has open issues to address before progressing = to IETF LC >=20 > Major (mostly in document order) >=20 > 1) The version string claims it is hexadecimal encoded. The version = for this > document is expressed by the character A. Does that mean this is = version 10 > (base 10)? Are we limited to versions 10 through 15? (we only have one = byte > and section 6 points out that index lines always start with an = alphabetical > character). This description needs to be clarified. While you're = looking at > it, consider discussing whether a given log file can have records with > different versions in it. In Section 4.1, the text says that the Version is "hexadecimal encoded." = =20 This was inherited from the original indexed-ASCII draft and is = confusing.=20 I will update it to indicate that it is ASCII encoded so we aren't = subject=20 to the limit of 6 versions.=20 For quickly skipping of index records, the first character in the log file for index records is an alphabetic character (Section 6). So, given this, we felt that characters "A" -> "Z" (i.e., 26 versions) should be enough. If the working group feels that we need more versions, one could think of having the set ["a"-"z","A"-"Z"] as versions, thus giving us 52 version numbers. Regarding the same log file having records with different record versions --- since the version number is per CLF record, it does allow for having a SIP CLF file contain different records. However, our expectation is that this would not happen too often. We can put a sentence or two saying the above. > 2) The description of Timestamp and Fractional Seconds is very = confusing. > What does it mean to be represented in big-endian fashion with a most > significant octet and to be decimal encoded? It's very far from clear = that > you intend this to be an ascii string representing an integer, and = it's > not clear that you intend that it to be prefix-padded with 0s if = necessary. > Did the group consider using ABNF to describe these fields (or the = whole > log record) to avoid this kind of ambiguity? Good point. The text needs clarifying. I will remove the portion of the = text=20 that states "Represented in big-endian fashion with most significant = octet=20 first from zero starting at the left, or high-order, position. Decimal = encoded."=20 from the description of both Timestamp and Fractional Seconds fields.=20 I will also supplement the text with the following summary text: The timestamp is represented in the log file as a UTF-8 string representing the date and time of the request or response represented as the number of seconds and milliseconds since the Unix epoch. The number of milliseconds MUST be separated by a "." from the number of seconds. It is not required to zero-pad the seconds and milliseconds, although if they are zero padded, a SIP CLF reader MUST be able to interpret them by discarding leading zeroes. > 3) The document should be clear about whether the flags field values = are > extensible, and if so, how extension is expected to be managed. How = will > you handle SIP received over DCCP, Websockets, or something we haven't > heard of yet? About the only flag that is expected to be extended is the Transport Flag (byte 4). Currently, the format draft defines 3 values for it (UDP, TCP, SCTP). If Websockets and DCCP become supported transports in SIP, then yes, this field will need to be extended. We can do one of two things for extension: 1) Have IANA open up a registry for this field 2) Use the Options part of the SIPCLF record to encode the transport. That is, we define byte 4 -- Transport Flag to contain: U =3D UDP T =3D TCP S =3D SCTP O =3D Other Then, when a new transport is supported, a log writer MUST put "O" for the Transport Flag and MUST create an optional field using Vendor-ID 00000000. Example: when DCCP becomes supported, it will be encoded as follows: 00@00000000,000F,Transport: DCCP If we don't think that it is reasonable for new transports to be = ascribed a=20 value of 'Other' in the Transport Flag field, then the IANA registry is = the=20 more elegant solution. Let me know your thoughts on a way forward here=20= since the draft currently has no IANA dependencies. > 4) Section 4.2 speaks of placing "the actual values for the mandatory > fields" into the record. It's not clear what "actual values" means. = There > are other sections of the document that talk about transforming = (escaping) > values before putting them in the log record, including dealing with = tabs > and edge conditions like field that has only a question mark in it. = The > concept of encoding and escaping needs to be much clearer throughout = the > document. I'll edit that text to clarify actual values vs. escaped values. I'll = also look=20 through the document and ensure the text describing the escaping=20 mechanism(s) is clear. > 5) The document should more explicitly discuss the decision to replace > tabs with spaces. That decision was based on there being no known use > of tabs in SIP messages that means something semantically other than > whitespace. The discussion should point that out, and note that you > will not be able to use the logged information to reconstruct a > signature over the logged fields, and that any future (or proprietary) > SIP header fields that include tabs with semantic meaning (carrying a > blob perhaps) will lose this meaning while being logged. It should = also > be made brutally explicit whether you want this substitution to happen > when you are logging the entire message, or a body (that might happen > to have tabs in it). We will add a new paragraph that goes something like this (the new paragraph is added at the end of S4.2 paragraph "This data MUST appear ... (0x20) prior to being logged."): The decision to replace tabs with spaces was based on there being no known use of tabs in SIP messages to convey any other meaning than whitespace. Two consequences of the decision to replace tab with a space character are: (a) it will become impossible to reconstruct a signature over the logged field that matches the signature over fields in the original SIP message, and (b) any future SIP header fields that include tabs with a different semantic meaning than simply signifying whitespace will loose this meaning when logged. And finally, the tabs to spaces substitution MUST occur when logging mandatory fields; it SHOULD occur when when logging either the entire message or simply a SIP body. > 6) There are a couple of places where you talk about escaping '-' and = '?' > characters. The document should note that this could produce a logged = record > that is not syntactically parsible with a SIP parser (quoted-pair is = only > used in certain places in SIP messages). There should also be some = discussion > about whether a log reader unescapes anything. It was never the intent for a SIP CLF to be read by a SIP parser as it = would be=20 entirely nonsensical to it. The belief is that the reader of the SIP CLF = would know=20 what to do with it. I'll add a statement to the draft indicating this = explicitly. > 7) Why does Tag -01 restrict message bodies to just the listed types, = especially > with the open ended "miscellaneous text content" type category? If = there's a real > restriction here, it needs to be much more clearly stated. Why are you = allowing > any LWS for the separator between the content type header field value = and the > message body itself? Why isn't this simply a space? It is not clear = what > "byte encoded" means for representing binary bodies - do you mean = expanding > it into a hex representation, a uuencode representation, or soemthing = else? The decision to restrict logging of bodies to the specific types listed = in the draft=20 was a decision made by the WG [1]. The usage of a LWS separator was = specified=20 as part of that proposal for logging bodies. Making the separator a = space=20 instead of LWS is fine, in fact, it is preferable. I'll update the draft = to reflect that=20 change. I'll also draw a line in the sand and update the text indicating that = "byte encoded"=20 in this context means that bodies with binary data MUST be base64 = encoded for=20 logging (since MIME uses base64 encoding). > 8) There's no description of Tag-02. The text following the line that = starts > the description of the tag speaks of more general things covering both = tag-01 > and tag-02. I'll expand a bit on the text that says "Log entire SIP message". > 9) When describing repeating tag-01 or tag-02 multiple times to deal = with > large entities, the document should be clear that the fragments get = logged > in order, and how to reassemble the actual entity. It should also say = whether > it's ok to interleave (for example) tag-01 fragements with other = optional fields. This is a great callout. There is text in the description of Tag=3D01 = that needs to=20 be updated to clearly state that if your body or the entire SIP message is > 4096 bytes, then tough luck. We can't chunk this as it = would open=20 up a can of worms that we don't want to deal with; like indexing the = chunks=20 and a reassembly scheme. Better not to go there. I will update the text=20= to clearly state this point. > 10) The description of the Length field in section 4.3 is ambiguous. = It says > "This length does not include the header shown in Figure 5." But = figure 5 has > nothing clearly identified as "the header". I'll update the description of the Length field to state: Length Field (4 bytes): Indicates the length of only the "Value" field of this optionally logged element (as shown in Figure 5), hexadecimal encoded. This length corresponds to the length of the "Value" field only and MUST NOT include any of the other elements shown in Figure 5. > 11) Where does the document talk about how to log fields that are line = folded > in the original message? Is it ok for the logged field to contain = CRLFs, or is > the intent that the field be unfolded before logging? I will update the text to indicate that: For mandatory fields: The mandatory fields MUST NOT contain CRLF when logged. For optional fields: If an optional field contains a CRLF as part of its normal=20 production rule, the CRLF MUST be escaped by using the=20 URI encoded equivalent value of "%0D%0A". For logging bodies: All CRLF digraphs in the SIP body MUST be escaped by using the=20 URI encoded equivalent value of "%0D%0A" For logging the entire SIP message: All instances of CRLF digraphs, whether they appear in the SIP headers or the body MUST be escaped by using the URI encoded=20 equivalent value of "%0D%0A". > 12) It's not clear if the \r\n used in the examples of logging the SDP = body > are there as text-conventions for showing the example in an RFC, or if = you > intend that CRLFs be replaced by \r\n's when logging. If it's just a = convention, > please make it clear that the actual log record will have CRLFs. If = you were > expecting the CRLFs to be escaped in the log, there needs to be text = that > says that normatively somewhere, and you need to consider escaping the = escape > character. As per my comments for point #11 above, the \r\n in that example needs = to be=20 escaped with "%0D%0A". I'll update the examples to match with this new=20= text that I will be adding. > Minor >=20 > There is a maintenance risk in having all the detail of each block = duplicated > in Figure 1. Please consider taking that figure up a level, showing = the three > blocks and how they're separated, and let the other figures show the = internal > detail. This is something that we have discussed and have not found an elegant = solution for.=20 It was thought to be very valuable to display the pertinent portion of = the complete=20 CLF within the section where that was discussed. I'll do my best to = ensure that they=20 are all copied over properly from Figure 1. > There are blocks of text in this document that are really laying out = the problem. > Can they be moved into the problem statement/information = model/framework document > or simply deleted? > For example: "Implementing a logging scheme for SIP is a considerable = challenge.", > "Whilst one may question the value of the =46rom URI [...]", etc. Ok. I'll scour the document for instances of such text and remove them. > In section 4.1, you talk of "a single, 52 character hexadecimal = encoded string" > between bytes 8 and 55. There are only 48 octets between bytes 8 and = 55 > (inclusive). Please adjust the text to note that you are also talking = about > the pointers to the optional fields or describe the shorter length. I'll edit this text to be clearer. > In section 4.2's description of the To tag field you say "Value of the > tag parameter (if present)" but don't talk about what to do if the tag > parameter isn't present. In the parallel section on the =46rom tag, = you > don't talk about the potential for messages without =46rom tags. If = it's > the intent to not be able to log messages from legacy 2543 devices, = that's > fine, but the document should call that out as a design decision and = it wouldn't > hurt to provide some guidance on what to do if such a message shows = up. I'll add some text to clarify and simplify this. Basically, if the tag = is present in To and From,=20 it is logged. If it is not present, a "-" is logged. I'll leave it at = that. The intent is not=20 to differentiate between messages from legacy rfc2543 devices versus the=20= updated rfc3261. =20 > There is a lot of descriptive text in 4.2's sections on the server and = client > transaction that is a copy of what's in the = problem-statement/information model > document. It would be better to just point to the description in the = other document. This is something that we debated about quite a bit. In the end we = thought it more=20 useful for implementers to have the description of all mandatory fields = right in the=20 format draft. >=20 > Nits >=20 > How is this a "functionally equivalent event logging mechanism"? = Suggest replacing > that sentence with the single word "analogous". I'll make this change. > In the introduction, the document claims to define a format that is no = more difficult > to generate, but it's not clear what it's being compared to. No more = difficult than > what? I'll clarify this. > Section 4.1's description of Record Length can be misinterpreted. By = listing > a subset of the fields that are included, it leaves open whether any = fields > are excluded. It would be better to say "beginning with the Version = octet > and ending with the terminating line-feed". I'll make this change. > It would help to be more explicit that you are not logging the CRLF at = the end > of a given header field value. (Or is it an option for a logger to = choose to do > so?) I discussed this in one of my earlier responses. This will be stated = more explicitly=20 that the terminating CRLF of a logged field should not be logged. The = only CRLF's=20 that should be logged are those of a multi-line body, field, or complete = message=20 and they should be escaped as will be specified in the draft. > Please consider scrubbing the message used in example 5 back to the = minimal > number of octets you need to make the example useful. In particular, = consider > removing elements that have not already been standardized. I'm = surprised to see > the literal strings "server-tx" and "client-tx" in the example log = message. Was > that what you really intended? I'll look to trim down the sample message in Section 5 to a minimal set = of fields.=20 The use of server-txn and client-txn was intentional and done to = parallel the usage in=20 the data model draft. If you think a specific value is best, then I'll = make one up. > The code snippet should be delineated as such, following the > guidance in the TLP (see section 4.b of = No problem. I'll add the the markers and = around=20 the code snippet. [1] http://www.ietf.org/mail-archive/web/sip-clf/current/msg00423.html Thanks, Gonzalo >=20 >=20 > _______________________________________________ > sip-clf mailing list > sip-clf@ietf.org > https://www.ietf.org/mailman/listinfo/sip-clf >=20 --Apple-Mail-15-400101793 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
AD = review: draft-ietf-sipclf-format-05

Summary: This document has = open issues to address before progressing to IETF LC

Major = (mostly in document order)

1) The version string claims it is = hexadecimal encoded. The version for this
document is expressed by = the character A. Does that mean this is version 10
(base 10)? Are we = limited to versions 10 through 15? (we only have one byte
and section = 6 points out that index lines always start with an = alphabetical
character). This description needs to be clarified. = While you're looking at
it, consider discussing whether a given log = file can have records with
different versions in = it.

In Section 4.1, the text = says that the Version is "hexadecimal encoded." =  
This was inherited from the original = indexed-ASCII draft and is confusing. 
I will update = it to indicate that it is ASCII encoded so we aren't = subject 
to the limit of 6 = versions. 

For quickly skipping of index = records, the first character in the log
file for index records is an = alphabetic character (Section 6).  So,
given this, we felt that = characters "A" -> "Z" (i.e., 26 versions)
should be enough. =  If the working group feels that we need more
versions, one = could think of having the set ["a"-"z","A"-"Z"] as
versions, thus = giving us 52 version numbers.

Regarding the same log file having = records with different record
versions --- since the version number = is per CLF record, it does
allow for having a SIP CLF file contain = different records.  However,
our expectation is that this would = not happen too often.  We can put
a sentence or two saying the = above.

2) The description = of Timestamp and Fractional Seconds is very confusing.
What does it = mean to be represented in big-endian fashion with a most
significant = octet and to be decimal encoded? It's very far from clear that
you = intend this to be an ascii string representing an integer, and = it's
not clear that you intend that it to be prefix-padded with 0s if = necessary.
Did the group consider using ABNF to describe these fields = (or the whole
log record) to avoid this kind of = ambiguity?

Good point. The = text needs clarifying. I will remove the portion of the = text 
that states "Represented in big-endian fashion = with most significant octet 
first from zero = starting at the left, or high-order, position.  Decimal = encoded." 
from the description of both Timestamp and = Fractional Seconds fields. 
I will also supplement the = text with the following summary text:

The timestamp = is represented in the log file as a UTF-8 string
representing the = date and time of the request or response
represented as the number of = seconds and milliseconds since the
Unix epoch.  The number of = milliseconds MUST be separated by a
"." from the number of seconds. =  It is not required to zero-pad
the seconds and milliseconds, = although if they are zero padded,
a SIP CLF reader MUST be able to = interpret them by discarding
leading = zeroes.

3) The document = should be clear about whether the flags field values are
extensible, = and if so, how extension is expected to be managed. How will
you = handle SIP received over DCCP, Websockets, or something we = haven't
heard of yet?

About the = only flag that is expected to be extended is the Transport
Flag (byte = 4).  Currently, the format draft defines 3 values for
it (UDP, = TCP, SCTP).  If Websockets and DCCP become supported = transports
in SIP, then yes, this field will need to be extended. =  We can do one
of two things for extension:

 1) Have = IANA open up a registry for this field
 2) Use the Options part = of the SIPCLF record to encode the transport.
  That is, we = define byte 4 -- Transport Flag to contain:

  U =3D = UDP
  T =3D TCP
  S =3D SCTP
  O = =3D Other

  Then, when a new transport is supported, a = log writer MUST put "O"
  for the Transport Flag and MUST = create an optional field using
  Vendor-ID 00000000. =  Example: when DCCP becomes supported, it will
  be = encoded as follows:

  00@00000000,000F,Transport: = DCCP


If we don't think that it is reasonable for = new transports to be ascribed a 
value of 'Other' in the = Transport Flag field, then the IANA registry is the 
more = elegant solution. Let me know your thoughts on a way forward = here 
since the draft currently has no IANA = dependencies.

4) Section = 4.2 speaks of placing "the actual values for the mandatory
fields" = into the record. It's not clear what "actual values" means. There
are = other sections of the document that talk about transforming = (escaping)
values before putting them in the log record, including = dealing with tabs
and edge conditions like field that has only a = question mark in it. The
concept of encoding and escaping needs to be = much clearer throughout = the
document.

I'll edit that = text to clarify actual values vs. escaped values. I'll also = look 
through the document and ensure the text describing = the escaping 
mechanism(s) is = clear.

5) The document = should more explicitly discuss the decision to replace
tabs with = spaces. That decision was based on there being no known use
of tabs = in SIP messages that means something semantically other = than
whitespace. The discussion should point that out, and note that = you
will not be able to use the logged information to reconstruct = a
signature over the logged fields, and that any future (or = proprietary)
SIP header fields that include tabs with semantic = meaning (carrying a
blob perhaps) will lose this meaning while being = logged. It should also
be made brutally explicit whether you want = this substitution to happen
when you are logging the entire message, = or a body (that might happen
to have tabs in = it).

We will add a new paragraph = that goes something
like this (the new paragraph is added at the end = of S4.2 paragraph
"This data MUST appear ... (0x20) prior to being = logged."):

 The decision to replace tabs with spaces was = based on there
 being no known use of tabs in SIP messages to = convey any other
 meaning than whitespace.  Two = consequences of the decision to
 replace tab with a space = character are: (a) it will become
 impossible to reconstruct a = signature over the logged field that
 matches the signature over = fields in the original SIP message,
 and (b) any future SIP = header fields that include tabs with a
 different semantic = meaning than simply signifying whitespace
 will loose this = meaning when logged.  And finally, the tabs to
 spaces = substitution MUST occur when logging mandatory fields;
 it = SHOULD occur when when logging either the entire message = or
 simply a SIP body.

6) There are a couple of places where you talk about = escaping '-' and '?'
characters. The document should note that this = could produce a logged record
that is not syntactically parsible with = a SIP parser (quoted-pair is only
used in certain places in SIP = messages). There should also be some discussion
about whether a log = reader unescapes anything.

It was = never the intent for a SIP CLF to be read by a SIP parser as it would = be 
entirely nonsensical to it. The belief is that the = reader of the SIP CLF would know 
what to do with it. = I'll add a statement to the draft indicating this = explicitly.

7) Why does Tag = -01 restrict message bodies to just the listed types, especially
with = the open ended "miscellaneous text content" type category? If there's a = real
restriction here, it needs to be much more clearly stated. Why = are you allowing
any LWS for the separator between the content type = header field value and the
message body itself? Why isn't this simply = a space? It is not clear what
"byte encoded" means for representing = binary bodies - do you mean expanding
it into a hex representation, a = uuencode representation, or soemthing = else?

The decision to restrict = logging of bodies to the specific types listed in the = draft 
was a decision made by the WG [1]. The usage of a = LWS separator was specified 
as part of that proposal for = logging bodies. Making the separator a space 
instead of = LWS is fine, in fact, it is preferable. I'll update the draft to = reflect that 
change.

I'll also = draw a line in the sand and update the text indicating that "byte = encoded" 
in this context means that bodies with = binary data MUST be base64 encoded for 
logging (since = MIME uses base64 encoding).

8) There's no description of Tag-02. The text = following the line that starts
the description of the tag speaks of = more general things covering both tag-01
and = tag-02.

I'll expand a bit on the = text that says "Log entire SIP = message".

9) When = describing repeating tag-01 or tag-02 multiple times to deal = with
large entities, the document should be clear that the fragments = get logged
in order, and how to reassemble the actual entity. It = should also say whether
it's ok to interleave (for example) tag-01 = fragements with other optional = fields.

This is a great = callout. There is text in the description of Tag=3D01 that needs = to 
be updated to clearly state that if your body or = the entire SIP
message is > 4096 bytes, then tough luck. We = can't chunk this as it would open 
up a can of worms that = we don't want to deal with; like indexing the chunks 
and = a reassembly scheme. Better not to go there. I will update the = text 
to clearly state this = point.

10) The description = of the Length field in section 4.3 is ambiguous. It says
"This length = does not include the header shown in Figure 5." But figure 5 = has
nothing clearly identified as "the = header".

I'll update the = description of the Length field to = state:

Length Field (4 bytes):  Indicates = the length of only the "Value"
field of this optionally logged = element (as shown in Figure 5),
hexadecimal encoded. This length = corresponds to the length of the
"Value" field only and MUST NOT = include any of the other elements
shown in Figure = 5.

11) Where does the = document talk about how to log fields that are line folded
in the = original message? Is it ok for the logged field to contain CRLFs, or = is
the intent that the field be unfolded before = logging?

I will update the = text to indicate that:

For mandatory = fields:

 The mandatory fields MUST NOT contain CRLF when = logged.

For optional fields:

If an optional = field contains a CRLF as part of its normal 
production = rule, the CRLF MUST be escaped by using the 
URI encoded = equivalent value of "%0D%0A".

For logging = bodies:

All CRLF digraphs in the SIP body MUST be escaped by = using the 
URI encoded equivalent value of = "%0D%0A"

For logging the entire SIP message:

All instances = of CRLF digraphs, whether they appear in the SIP
headers or the body = MUST be escaped by using the URI = encoded 
equivalent value of = "%0D%0A".

12) It's not clear if = the \r\n used in the examples of logging the SDP body
are there as = text-conventions for showing the example in an RFC, or if you
intend = that CRLFs be replaced by \r\n's when logging. If it's just a = convention,
please make it clear that the actual log record will have = CRLFs. If you were
expecting the CRLFs to be escaped in the log, = there needs to be text that
says that normatively somewhere, and you = need to consider escaping the = escape
character.

As per my = comments for point #11 above, the \r\n in that example needs to = be 
escaped with "%0D%0A". I'll update the examples = to match with this new 
text that I will be = adding.

Minor

There is a maintenance risk in having = all the detail of each block duplicated
in Figure 1. Please consider = taking that figure up a level, showing the three
blocks and how = they're separated, and let the other figures show the = internal
detail.

This is = something that we have discussed and have not found an elegant solution = for. 
It was thought to be very valuable to display the = pertinent portion of the complete 
CLF within the section = where that was discussed. I'll do my best to ensure that = they 
are all copied over properly from Figure = 1.

There are blocks of text = in this document that are really laying out the problem.
Can they be = moved into the problem statement/information model/framework = document
or simply deleted?
For example: "Implementing a logging = scheme for SIP is a considerable challenge.",
"Whilst one may = question the value of the =46rom URI [...]", = etc.

Ok. I'll scour the document = for instances of such text and remove them.

In section 4.1, you talk of "a single, 52 character = hexadecimal encoded string"
between bytes 8 and 55. There are only 48 = octets between bytes 8 and 55
(inclusive). Please adjust the text to = note that you are also talking about
the pointers to the optional = fields or describe the shorter = length.

I'll edit this text to be = clearer.

In section 4.2's = description of the To tag field you say "Value of the
tag parameter = (if present)" but don't talk about what to do if the tag
parameter = isn't present. In the parallel section on the =46rom tag, you
don't = talk about the potential for messages without =46rom tags. If = it's
the intent to not be able to log messages from legacy 2543 = devices, that's
fine, but the document should call that out as a = design decision and it wouldn't
hurt to provide some guidance on what = to do if such a message shows = up.

I'll add some text to clarify = and simplify this. Basically, if the tag is present in To and = From, 
it is logged. If it is not present, a "-" is = logged. I'll leave it at that. The intent is = not 
to differentiate between messages from = legacy rfc2543 devices versus the 
updated rfc3261. =  

There is a lot of = descriptive text in 4.2's sections on the server and = client
transaction that is a copy of what's in the = problem-statement/information model
document. It would be better to = just point to the description in the other = document.

This is something that we = debated about quite a bit. In the end we thought it = more 
useful for implementers to have the description of = all mandatory fields right in the 
format = draft.


Nits

How = is this a "functionally equivalent event logging mechanism"? Suggest = replacing
that sentence with the single word = "analogous".

I'll make this = change.

In the = introduction, the document claims to define a format that is no more = difficult
to generate, but it's not clear what it's being compared = to. No more difficult = than
what?

I'll clarify = this.

Section 4.1's = description of Record Length can be misinterpreted. By listing
a = subset of the fields that are included, it leaves open whether any = fields
are excluded. It would be better to say "beginning with the = Version octet
and ending with the terminating = line-feed".

I'll make this = change.

It would help to be = more explicit that you are not logging the CRLF at the end
of a given = header field value. (Or is it an option for a logger to choose to = do
so?)

I discussed this in one = of my earlier responses. This will be stated more = explicitly 
that the terminating CRLF of a logged field = should not be logged. The only CRLF's 
that should be = logged are those of a multi-line body, field, or complete = message 
and they should be escaped as will be specified = in the draft.

Please = consider scrubbing the message used in example 5 back to the = minimal
number of octets you need to make the example useful. In = particular, consider
removing elements that have not already been = standardized. I'm surprised to see
the literal strings "server-tx" = and "client-tx" in the example log message. Was
that what you really = intended?

I'll look to trim down = the sample message in Section 5 to a minimal set of = fields. 
The use of server-txn and client-txn was = intentional and done to parallel the usage in 
the data = model draft. If you think a specific value is best, then I'll make one = up.

The code snippet should = be delineated as such, following the
guidance in the TLP (see section = 4.b of <http:/= /trustee.ietf.org/docs/IETF-Trust-License-Policy.pdf>

No problem. I'll add the the markers = <CODE BEGINS> and <CODE ENDS> around 
the = code snippet.



Thanks,

Gonzalo
=



_______________________________________________=
sip-clf mailing list
sip-clf@ietf.org
https://www.ietf.= org/mailman/listinfo/sip-clf


= --Apple-Mail-15-400101793-- From gsalguei@cisco.com Wed Feb 8 16:52:31 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 979B521F84D8 for ; Wed, 8 Feb 2012 16:52:31 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -9.448 X-Spam-Level: X-Spam-Status: No, score=-9.448 tagged_above=-999 required=5 tests=[AWL=-1.150, BAYES_00=-2.599, HTML_MESSAGE=0.001, MANGLED_DEALS=2.3, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M65+au-CLrZJ for ; Wed, 8 Feb 2012 16:52:31 -0800 (PST) Received: from av-tac-rtp.cisco.com (hen.cisco.com [64.102.19.198]) by ietfa.amsl.com (Postfix) with ESMTP id 25BB921F8496 for ; Wed, 8 Feb 2012 16:52:23 -0800 (PST) X-TACSUNS: Virus Scanned Received: from chook.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-rtp.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q190qMF8013731 for ; Wed, 8 Feb 2012 19:52:22 -0500 (EST) Received: from dhcp-64-102-210-22.cisco.com (dhcp-64-102-210-22.cisco.com [64.102.210.22]) by chook.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q190qGdY012269; Wed, 8 Feb 2012 19:52:16 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-17-401701545 From: Gonzalo Salgueiro In-Reply-To: Date: Wed, 8 Feb 2012 19:52:16 -0500 Message-Id: References: To: Claudio Allocchio X-Mailer: Apple Mail (2.1084) Cc: "sip-clf@ietf.org Mailing" , apps-discuss@ietf.org, draft-ietf-sipclf-format.all@tools.ietf.org Subject: Re: [sip-clf] [apps-discuss] AppsDir review of draft-ietf-sipclf-format-05 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 00:52:31 -0000 --Apple-Mail-17-401701545 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi Claudio -=20 Thanks for the thorough review. See inline comments. On Jan 19, 2012, at 5:21 AM, Claudio Allocchio wrote: >=20 > I have been selected as the Applications Area Directorate reviewer for = this draft (for background on appsdir, please see = http://trac.tools.ietf.org/area/app/trac/wiki/ApplicationsAreaDirectorate)= . >=20 > Please resolve these comments along with any other Last Call comments = you may receive. Please wait for direction from your document shepherd = or AD before posting a new version of the draft. >=20 > Document: draft-ietf-sipclf-format-05 > Title: Format for the Session Initiation Protocol (SIP) Common Log = Format > (CLF) > Reviewer: Claudio Allocchio > Review Date: 2012-01-18 > IETF Last Call Date: unknown > IESG Telechat Date: unknown >=20 > NOTE: is there is an IESG telechat which I'm missing, just copy this = messae to iesg ML! >=20 > Summary: >=20 > This draft is ready for publication as a Proposed Standard RFC; it = just needs one consideration about a minor issue about an external = reference, and fixing some Nits. This is great news. Thanks. >=20 > Major Issues: >=20 > NONE >=20 > Minor Issues: >=20 > just one ! >=20 > * Section "3. Document Conventions" >=20 > The authors have decided "For the sake of clarity and completeness" to = quote a full section of RFC4475 into this document. However there were = quite long discussions in the Apps Dir in other reviews if this quoting = practice is appropriate or not, because it can lead to discrepacies, = when the quoted document is updated or obsoleted, resuing in confusion = for the readers who just trust the quoted text without checking the = state of the referenced document. I would suggest, as in the other = cases, NOT to quote the external text, but to use an explicit external = reference only, urging the reader to check that "for the sake of = clarity". I agree with the position but in this particular case I'm of the mind = that removing that text and referencing the RFC diminishes the = effectiveness of that section. It is my preference to leave the quoted = text (which is very short) embedded in this document as it is critical = to understanding the notation used throughout the document. >=20 > Nits: >=20 > * Section "Copyright Notice" >=20 > change the year to 2012. Will do this. >=20 > * Section "3. Document Conventions" >=20 > remember to change "formatting rules for Internet-Drafts" into = "formatting rules for RFCs" when published. Great point. I'll proactively beat the RFC Editor to this. >=20 > * Section "4. Format" >=20 > I do understand that it probably does not fit into the I-D formatting = line lenghts, but... maybe an attempt to add aside the SIP CLF record = depicted in Figure 1 the grouping described as >=20 > > > >=20 > in the figure 1 itself (on the left?) may help in reading the = distinction, and maybe may even not require to repeat portions of the = figure itself later in the I-D, like in figure 3, figure 4 and figure 5. = It is really a nit and just a suggestion. Not really important. >=20 > What I am suggesting: >=20 > +----------------------------------+ > /| | > / | | > P | | > o | | > I i | | > n n | | > d t | | > e e | | > x r | | > s | | > \ | | > \| | > +----------------------------------+ > /| | > / | | > M | | > a F | | > n i | | > d e | | > a l | | > t d | | > o s | | > r | | > y \ | | > \| | > +----------------------------------+ > /| | > / | | > O | | > p F | | > t i | | > i e | | > o l | | > n d | | > a s | | > l | | > \ | | > \| | > +----------------------------------+ >=20 This is something that we have discussed and have not found an elegant = solution for.=20 While there are maintenance risks it was thought to be very valuable to = display the=20 pertinent portion of the complete CLF within the section where that was = discussed.=20 I'll do my best to ensure that they are all copied over properly from = Figure 1. > * Section "5. Example SIP CLF Record" >=20 > Remember to change "Due to internet-draft conventions" into "Due to = RFC conventions" when published (and also "to format it for an RFC" = later on...). Indeed. >=20 > * =46rom the DataTracker ID Nits check tool (References Nits): >=20 > =3D=3D Unused Reference: 'RFC5612' is defined on line 984, but no = explicit > reference was found in the text >=20 > ** Downref: Normative reference to an Informational draft: > draft-ietf-sipclf-problem-statement (ref. > 'I-D.ietf-sipclf-problem-statement') >=20 > Comment: the document shepard is aware and has a solution for these = nits. Just apply it. Absolutely. Thanks, Gonzalo >=20 > = --------------------------------------------------------------------------= ---- > Claudio Allocchio G A R R = Claudio.Allocchio@garr.it > Senior Technical Officer > tel: +39 040 3758523 Italian Academic and G=3DClaudio; = S=3DAllocchio; > fax: +39 040 3758565 Research Network P=3Dgarr; A=3Dgarr;= C=3Dit; >=20 > PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca > _______________________________________________ > apps-discuss mailing list > apps-discuss@ietf.org > https://www.ietf.org/mailman/listinfo/apps-discuss >=20 --Apple-Mail-17-401701545 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Hi = Claudio - 

Thanks for the thorough = review.

See inline = comments.

On Jan 19, 2012, at 5:21 AM, Claudio = Allocchio wrote:


I have been selected as the Applications Area = Directorate reviewer for this draft (for background on appsdir, please = see http://trac.tools.ietf.org/area/app/trac/wiki/ApplicationsAreaDir= ectorate).

Please resolve these comments along with any other = Last Call comments you may receive. Please wait for direction from your = document shepherd or AD before posting a new version of the = draft.

Document: draft-ietf-sipclf-format-05
Title: Format for = the Session Initiation Protocol (SIP) Common Log Format
=       (CLF)
Reviewer: Claudio = Allocchio
Review Date: 2012-01-18
IETF Last Call Date: = unknown
IESG Telechat Date: unknown

NOTE: is there is an IESG = telechat which I'm missing, just copy this messae to iesg = ML!

Summary:

This draft is ready for publication as a = Proposed Standard RFC; it just needs one consideration about a minor = issue about an external reference, and fixing some = Nits.

This is great news. = Thanks.

Major = Issues:

NONE

Minor Issues:

just one !

* = Section "3. Document Conventions"

The authors have decided "For = the sake of clarity and completeness" to quote a full section of RFC4475 = into this document. However there were quite long discussions in the = Apps Dir in other reviews if this quoting practice is appropriate or = not, because it can lead to discrepacies, when the quoted document is = updated or obsoleted, resuing in confusion for the readers who just = trust the quoted text without checking the state of the referenced = document. I would suggest, as in the other cases, NOT to quote the = external text, but to use an explicit external reference only, urging = the reader to check that "for the sake of = clarity".

I agree with the position = but in this particular case I'm of the mind that removing that text and = referencing the RFC diminishes the effectiveness of that section. It is = my preference to leave the quoted text (which is very short) embedded in = this document as it is critical to understanding the notation used = throughout the document.

Nits:

* Section "Copyright = Notice"

change the year to = 2012.

Will do this.

* Section "3. Document = Conventions"

remember to change "formatting rules for = Internet-Drafts" into "formatting rules for RFCs" when = published.

Great point. I'll = proactively beat the RFC Editor to this.

* Section "4. Format"

I do understand that = it probably does not fit into the I-D formatting line lenghts, but... = maybe an attempt to add aside the SIP CLF record depicted in Figure 1 = the grouping described as

=             &n= bsp;           &nbs= p;       <IndexPointers>
=             &n= bsp;           &nbs= p;       <MandatoryFields>
=             &n= bsp;           &nbs= p;       <OptionalFields>

= in the figure 1 itself (on the left?) may help in reading the = distinction, and maybe may even not require to repeat portions of the = figure itself later in the I-D, like in figure 3, figure 4 and figure 5. = It is really a nit and just a suggestion. Not really = important.

What I am suggesting:

=       +----------------------------------+      /| =             &n= bsp;           &nbs= p;        |
=     / | =             &n= bsp;           &nbs= p;        |
  P =   | =             &n= bsp;           &nbs= p;        |
  o =   | =             &n= bsp;           &nbs= p;        |
I i =   | =             &n= bsp;           &nbs= p;        |
n n =   | =             &n= bsp;           &nbs= p;        |
d t =   | =             &n= bsp;           &nbs= p;        |
e e =   | =             &n= bsp;           &nbs= p;        |
x r =   | =             &n= bsp;           &nbs= p;        |
  s =   | =             &n= bsp;           &nbs= p;        |
=    \  | =             &n= bsp;           &nbs= p;        |
=      \| =             &n= bsp;           &nbs= p;        |
=       +----------------------------------+      /| =             &n= bsp;           &nbs= p;        |
=     / | =             &n= bsp;           &nbs= p;        |
M =     | =             &n= bsp;           &nbs= p;        |
a F =   | =             &n= bsp;           &nbs= p;        |
n i =   | =             &n= bsp;           &nbs= p;        |
d e =   | =             &n= bsp;           &nbs= p;        |
a l =   | =             &n= bsp;           &nbs= p;        |
t d =   | =             &n= bsp;           &nbs= p;        |
o s =   | =             &n= bsp;           &nbs= p;        |
r =     | =             &n= bsp;           &nbs= p;        |
y  \ =  | =             &n= bsp;           &nbs= p;        |
=      \| =             &n= bsp;           &nbs= p;        |
=       +----------------------------------+      /| =             &n= bsp;           &nbs= p;        |
=     / | =             &n= bsp;           &nbs= p;        |
O =     | =             &n= bsp;           &nbs= p;        |
p F =   | =             &n= bsp;           &nbs= p;        |
t i =   | =             &n= bsp;           &nbs= p;        |
i e =   | =             &n= bsp;           &nbs= p;        |
o l =   | =             &n= bsp;           &nbs= p;        |
n d =   | =             &n= bsp;           &nbs= p;        |
a s =   | =             &n= bsp;           &nbs= p;        |
l =     | =             &n= bsp;           &nbs= p;        |
=    \  | =             &n= bsp;           &nbs= p;        |
=      \| =             &n= bsp;           &nbs= p;        |
=       +----------------------------------+

This is something that we have = discussed and have not found an elegant solution = for. 
While there are maintenance risks it was thought to = be very valuable to display the 
pertinent portion of the = complete CLF within the section where that was = discussed. 
I'll do my best to ensure that they are = all copied over properly from Figure = 1.


* = Section "5. Example SIP CLF Record"

Remember to change "Due to = internet-draft conventions" into "Due to RFC conventions" when published = (and also "to format it for an RFC" later = on...).

Indeed.

* =46rom the DataTracker ID Nits check tool = (References Nits):

 =3D=3D Unused Reference: 'RFC5612' is = defined on line 984, but no explicit
=     reference was found in the text

 ** = Downref: Normative reference to an Informational draft:
=     draft-ietf-sipclf-problem-statement (ref.
=     'I-D.ietf-sipclf-problem-statement')

Commen= t: the document shepard is aware and has a solution for these nits. Just = apply = it.

Absolutely.

<= div>Thanks,

Gonzalo


=

---------------------------------------------------= ---------------------------
Claudio Allocchio =             G =   A   R   R =          Claudio.Allocchio@garr.it =             &n= bsp;          Senior = Technical Officer
tel: +39 040 3758523 =      Italian Academic and =       G=3DClaudio; S=3DAllocchio;
fax: = +39 040 3758565        Research = Network         P=3Dgarr; = A=3Dgarr; C=3Dit;

=           PGP Key: http://www.cert.garr.it/= PGP/keys.php3#ca
_______________________________________________apps-discuss mailing list
apps-discuss@ietf.org
https:/= /www.ietf.org/mailman/listinfo/apps-discuss


= --Apple-Mail-17-401701545-- From gsalguei@cisco.com Thu Feb 9 06:44:16 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 947DB21F857F for ; Thu, 9 Feb 2012 06:44:16 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.368 X-Spam-Level: X-Spam-Status: No, score=-10.368 tagged_above=-999 required=5 tests=[AWL=0.230, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CjqweTin+T3T for ; Thu, 9 Feb 2012 06:44:15 -0800 (PST) Received: from av-tac-rtp.cisco.com (hen.cisco.com [64.102.19.198]) by ietfa.amsl.com (Postfix) with ESMTP id 87F2C21F8567 for ; Thu, 9 Feb 2012 06:44:15 -0800 (PST) X-TACSUNS: Virus Scanned Received: from chook.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-rtp.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q19EiDew029016 for ; Thu, 9 Feb 2012 09:44:14 -0500 (EST) Received: from rtp-gsalguei-8714.cisco.com (rtp-gsalguei-8714.cisco.com [10.116.61.53]) by chook.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q19EiCec025302; Thu, 9 Feb 2012 09:44:12 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-20-451617382 From: Gonzalo Salgueiro In-Reply-To: Date: Thu, 9 Feb 2012 09:44:11 -0500 Message-Id: <4FB65F28-DC68-4240-99F9-E99C457968A8@cisco.com> References: To: Claudio Allocchio X-Mailer: Apple Mail (2.1084) Cc: "sip-clf@ietf.org Mailing" , apps-discuss@ietf.org, draft-ietf-sipclf-format.all@tools.ietf.org Subject: Re: [sip-clf] [apps-discuss] AppsDir review of draft-ietf-sipclf-format-05 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 14:44:16 -0000 --Apple-Mail-20-451617382 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi Claudio -=20 On Feb 9, 2012, at 2:47 AM, Claudio Allocchio wrote: > Hello all! >=20 >>> * Section "3. Document Conventions" >>>=20 >>> The authors have decided "For the sake of clarity and completeness" = to quote a full section of RFC4475 into this document. However there = were quite long discussions in the Apps Dir in other reviews if this = quoting practice is appropriate or not, because it can lead to = discrepacies, when the quoted document is updated or obsoleted, resuing = in confusion for the readers who just trust the quoted text without = checking the state of the referenced document. I would suggest, as in = the other cases, NOT to quote the external text, but to use an explicit = external reference only, urging the reader to check that "for the sake = of clarity". >>=20 >> I agree with the position but in this particular case I'm of the mind = that removing that text and referencing the RFC diminishes the = effectiveness of that section. It is my preference to leave the quoted = text (which is very short) embedded in this document as it is critical = to understanding the notation used throughout the document. >=20 > maybe a possible compromise is to quote the text, as is now, but to = add a small note which reminds the reader to check the update/obsoletes = status of RFC4475? ;-) I could add that but bear in mind that I only care about the quoted text = dealing with formatting rules from RFC4475 and nothing else. So whether = it gets updated or obsoleted is irrelevant in this case. In fact, if it = gets updated and it affects the formatting guidelines I quoted then I = don't want to reference it at all since it is no longer useful or = significant to this draft. >=20 >>> * Section "4. Format" >>>=20 >> This is something that we have discussed and have not found an = elegant solution for. While there are maintenance risks it was thought = to be very valuable to display the pertinent portion of the complete CLF = within the section where that was discussed. I'll do my best to ensure = that they are all copied over properly from Figure 1. >=20 > as I said, it is a Nit... thanks for taking some time in thinking = about it. Thanks to you for your careful review. Warm Regards, Gonzalo >=20 > have a nice day! >=20 >=20 > = --------------------------------------------------------------------------= ---- > Claudio Allocchio G A R R = Claudio.Allocchio@garr.it > Senior Technical Officer > tel: +39 040 3758523 Italian Academic and G=3DClaudio; = S=3DAllocchio; > fax: +39 040 3758565 Research Network P=3Dgarr; A=3Dgarr;= C=3Dit; >=20 > PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca >=20 --Apple-Mail-20-451617382 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Hi = Claudio - 

On Feb 9, 2012, at 2:47 AM, Claudio = Allocchio wrote:

Hello = all!

* = Section "3. Document = Conventions"

The authors have decided "For = the sake of clarity and completeness" to quote a full section of RFC4475 = into this document. However there were quite long discussions in the = Apps Dir in other reviews if this quoting practice is appropriate or = not, because it can lead to discrepacies, when the quoted document is = updated or obsoleted, resuing in confusion for the readers who just = trust the quoted text without checking the state of the referenced = document. I would suggest, as in the other cases, NOT to quote the = external text, but to use an explicit external reference only, urging = the reader to check that "for the sake of = clarity".

I agree with = the position but in this particular case I'm of the mind that removing = that text and referencing the RFC diminishes the effectiveness of that = section. It is my preference to leave the quoted text (which is very = short) embedded in this document as it is critical to understanding the = notation used throughout the document.

maybe a = possible compromise is to quote the text, as is now, but to add a small = note which reminds the reader to check the update/obsoletes status of = RFC4475? ;-)

I could add that but = bear in mind that I only care about the quoted text dealing with = formatting rules from RFC4475 and nothing else. So whether it gets = updated or obsoleted is irrelevant in this case. In fact, if it gets = updated and it affects the formatting guidelines I quoted then I don't = want to reference it at all since it is no longer useful or significant = to this draft.

* Section "4. = Format"

This is something that we have discussed and have not = found an elegant solution for. While there are maintenance risks it was = thought to be very valuable to display the pertinent portion of the = complete CLF within the section where that was discussed. I'll do my = best to ensure that they are all copied over properly from Figure = 1.

as I said, it is a Nit... thanks for taking some = time in thinking about it.

Thanks = to you for your careful review.

Warm = Regards,

Gonzalo


have a nice = day!


----------------------------------------------------------= --------------------
Claudio Allocchio =             G =   A   R   R =          Claudio.Allocchio@garr.it =             &n= bsp;          Senior = Technical Officer
tel: +39 040 3758523 =      Italian Academic and =       G=3DClaudio; S=3DAllocchio;
fax: = +39 040 3758565        Research = Network         P=3Dgarr; = A=3Dgarr; C=3Dit;

=           PGP Key: http://www.cert.garr.it/= PGP/keys.php3#ca


= --Apple-Mail-20-451617382-- From rjsparks@nostrum.com Thu Feb 9 07:18:48 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 81D9221F8567 for ; Thu, 9 Feb 2012 07:18:48 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fDZiVf7qqoml for ; Thu, 9 Feb 2012 07:18:46 -0800 (PST) Received: from nostrum.com (nostrum-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:267::2]) by ietfa.amsl.com (Postfix) with ESMTP id 64DBB21F84E6 for ; Thu, 9 Feb 2012 07:18:45 -0800 (PST) Received: from unexplicable.local (pool-71-170-125-181.dllstx.fios.verizon.net [71.170.125.181]) (authenticated bits=0) by nostrum.com (8.14.3/8.14.3) with ESMTP id q19FIe9E014179 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 9 Feb 2012 09:18:41 -0600 (CST) (envelope-from rjsparks@nostrum.com) Message-ID: <4F33E3D0.7000605@nostrum.com> Date: Thu, 09 Feb 2012 09:18:40 -0600 From: Robert Sparks User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20111222 Thunderbird/9.0.1 MIME-Version: 1.0 To: Gonzalo Salgueiro References: <4F21DD3E.7000002@nostrum.com> <31A5C897-B767-4527-9346-905A80977F35@cisco.com> In-Reply-To: <31A5C897-B767-4527-9346-905A80977F35@cisco.com> Content-Type: multipart/alternative; boundary="------------030102090202080107060901" Received-SPF: pass (nostrum.com: 71.170.125.181 is authenticated by a trusted mechanism) Cc: draft-ietf-sipclf-format@tools.ietf.org, sipclf-chairs@tools.ietf.org, "sip-clf@ietf.org Mailing" Subject: Re: [sip-clf] AD review: draft-ietf-sipclf-format-05 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 15:18:48 -0000 This is a multi-part message in MIME format. --------------030102090202080107060901 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Inline, trimming to points with responses On 2/8/12 6:25 PM, Gonzalo Salgueiro wrote: > Robert - > > Thanks for your thorough and thoughtful review. Below are some inline > comments to your feedback. > > I will release the -06 version of the draft once we have settled on > all of the below points you have raised. > > > On Jan 26, 2012, at 6:09 PM, Robert Sparks wrote: > >> AD review: draft-ietf-sipclf-format-05 >> >> Summary: This document has open issues to address before progressing >> to IETF LC >> >> Major (mostly in document order) >> > >> 2) The description of Timestamp and Fractional Seconds is very confusing. >> What does it mean to be represented in big-endian fashion with a most >> significant octet and to be decimal encoded? It's very far from clear >> that >> you intend this to be an ascii string representing an integer, and it's >> not clear that you intend that it to be prefix-padded with 0s if >> necessary. >> Did the group consider using ABNF to describe these fields (or the whole >> log record) to avoid this kind of ambiguity? > > Good point. The text needs clarifying. I will remove the portion of > the text > that states "Represented in big-endian fashion with most significant > octet > first from zero starting at the left, or high-order, position. > Decimal encoded." > from the description of both Timestamp and Fractional Seconds fields. > I will also supplement the text with the following summary text: > > The timestamp is represented in the log file as a UTF-8 string > representing the date and time of the request or response > represented as the number of seconds and milliseconds since the > Unix epoch. The number of milliseconds MUST be separated by a > "." from the number of seconds. It is not required to zero-pad > the seconds and milliseconds, although if they are zero padded, > a SIP CLF reader MUST be able to interpret them by discarding > leading zeroes. Actually, I was under the impression that the zero padding was expected. The examples have them consistently - why did that happen? Either way, group members should say something about what the intent was. > >> 3) The document should be clear about whether the flags field values are >> extensible, and if so, how extension is expected to be managed. How will >> you handle SIP received over DCCP, Websockets, or something we haven't >> heard of yet? > > About the only flag that is expected to be extended is the Transport > Flag (byte 4). Currently, the format draft defines 3 values for > it (UDP, TCP, SCTP). If Websockets and DCCP become supported transports > in SIP, then yes, this field will need to be extended. We can do one > of two things for extension: > > 1) Have IANA open up a registry for this field > 2) Use the Options part of the SIPCLF record to encode the transport. > That is, we define byte 4 -- Transport Flag to contain: > > U = UDP > T = TCP > S = SCTP > O = Other > > Then, when a new transport is supported, a log writer MUST put "O" > for the Transport Flag and MUST create an optional field using > Vendor-ID 00000000. Example: when DCCP becomes supported, it will > be encoded as follows: > > 00@00000000,000F,Transport: DCCP > > > If we don't think that it is reasonable for new transports to be > ascribed a > value of 'Other' in the Transport Flag field, then the IANA registry > is the > more elegant solution. Let me know your thoughts on a way forward here > since the draft currently has no IANA dependencies. That's a choice the group should make - the document just needs to be explicit about what's extensible (for all the fields) and if a field is extensible, what the mechanism looks like. For the transport field, as an individual contributor, I prefer the IANA registry approach. > >> 5) The document should more explicitly discuss the decision to replace >> tabs with spaces. That decision was based on there being no known use >> of tabs in SIP messages that means something semantically other than >> whitespace. The discussion should point that out, and note that you >> will not be able to use the logged information to reconstruct a >> signature over the logged fields, and that any future (or proprietary) >> SIP header fields that include tabs with semantic meaning (carrying a >> blob perhaps) will lose this meaning while being logged. It should also >> be made brutally explicit whether you want this substitution to happen >> when you are logging the entire message, or a body (that might happen >> to have tabs in it). > > We will add a new paragraph that goes something > like this (the new paragraph is added at the end of S4.2 paragraph > "This data MUST appear ... (0x20) prior to being logged."): > > The decision to replace tabs with spaces was based on there > being no known use of tabs in SIP messages to convey any other > meaning than whitespace. Two consequences of the decision to > replace tab with a space character are: (a) it will become > impossible to reconstruct a signature over the logged field that > matches the signature over fields in the original SIP message, > and (b) any future SIP header fields that include tabs with a > different semantic meaning than simply signifying whitespace > will loose this meaning when logged. And finally, the tabs to > spaces substitution MUST occur when logging mandatory fields; > it SHOULD occur when when logging either the entire message or > simply a SIP body. This leaves out optional fields. It's also not clear why you use a SHOULD for the requirement over bodies. You should also make it clear (as part of escaping above) whether any escaping is done before encoding binary bodies. >> 6) There are a couple of places where you talk about escaping '-' and '?' >> characters. The document should note that this could produce a logged >> record >> that is not syntactically parsible with a SIP parser (quoted-pair is only >> used in certain places in SIP messages). There should also be some >> discussion >> about whether a log reader unescapes anything. > > It was never the intent for a SIP CLF to be read by a SIP parser as it > would be > entirely nonsensical to it. I think it _was_ the intent to be able to reuse parser code. Several early implementers were handing individual fields to the part of their parser that would handle a framed header field. > The belief is that the reader of the SIP CLF would know > what to do with it. I'll add a statement to the draft indicating this > explicitly. It will surprise me if a simple statement will fully address this point - be sure to capture whether the reader unescapes anyththing. > >> 7) Why does Tag -01 restrict message bodies to just the listed types, >> especially >> with the open ended "miscellaneous text content" type category? If >> there's a real >> restriction here, it needs to be much more clearly stated. Why are >> you allowing >> any LWS for the separator between the content type header field value >> and the >> message body itself? Why isn't this simply a space? It is not clear what >> "byte encoded" means for representing binary bodies - do you mean >> expanding >> it into a hex representation, a uuencode representation, or soemthing >> else? > > The decision to restrict logging of bodies to the specific types > listed in the draft > was a decision made by the WG [1]. The document needs to explain _why_ this particular subset was chosen. What basis did the group use for that choice? My read of the thread was that Vijay threw an initial list together and that just ended up in the document. It's not clear that the list was intended to be exclusive, just that whatever subsequent decisions that were made allowed at least those types. WHY are you not allowing images? Why are you not allowing multipart/mime? (If you ever did actually see an S/MIME protected body, you would not be able to log it the way the document stands now.) > The usage of a LWS separator was specified > as part of that proposal for logging bodies. Making the separator a space > instead of LWS is fine, in fact, it is preferable. I'll update the > draft to reflect that > change. > > I'll also draw a line in the sand and update the text indicating that > "byte encoded" > in this context means that bodies with binary data MUST be base64 > encoded for > logging (since MIME uses base64 encoding). Please be careful to be specific in the instructions on when to encode (for example, would you re-encode a binary format that was already base64 encoded?) > > >> 9) When describing repeating tag-01 or tag-02 multiple times to deal with >> large entities, the document should be clear that the fragments get >> logged >> in order, and how to reassemble the actual entity. It should also say >> whether >> it's ok to interleave (for example) tag-01 fragements with other >> optional fields. > > This is a great callout. There is text in the description of Tag=01 > that needs to > be updated to clearly state that if your body or the entire SIP > message is > 4096 bytes, then tough luck. We can't chunk this as it > would open > up a can of worms that we don't want to deal with; like indexing the > chunks > and a reassembly scheme. Better not to go there. I will update the text > to clearly state this point. Now I'm confused. Why do you have the ability to repeat these tags? > >> 11) Where does the document talk about how to log fields that are >> line folded >> in the original message? Is it ok for the logged field to contain >> CRLFs, or is >> the intent that the field be unfolded before logging? > > I will update the text to indicate that: > > For mandatory fields: > > The mandatory fields MUST NOT contain CRLF when logged. > > For optional fields: > > If an optional field contains a CRLF as part of its normal > production rule, the CRLF MUST be escaped by using the > URI encoded equivalent value of "%0D%0A". Why are you treating optional fields and mandatory fields differently here? If there's a good reason, fine, but be sure to include the reason in the document. > > For logging bodies: > > All CRLF digraphs in the SIP body MUST be escaped by using the > URI encoded equivalent value of "%0D%0A" > > For logging the entire SIP message: > > All instances of CRLF digraphs, whether they appear in the SIP > headers or the body MUST be escaped by using the URI encoded > equivalent value of "%0D%0A". Even when the body is binary (see previous discussion about escaping)? The group should weigh in on whether this is what they were expecting for the usual application/sdp case. >> 12) It's not clear if the \r\n used in the examples of logging the >> SDP body >> are there as text-conventions for showing the example in an RFC, or >> if you >> intend that CRLFs be replaced by \r\n's when logging. If it's just a >> convention, >> please make it clear that the actual log record will have CRLFs. If >> you were >> expecting the CRLFs to be escaped in the log, there needs to be text that >> says that normatively somewhere, and you need to consider escaping >> the escape >> character. > > As per my comments for point #11 above, the \r\n in that example needs > to be > escaped with "%0D%0A". I'll update the examples to match with this new > text that I will be adding. > >> Minor >> >> There is a maintenance risk in having all the detail of each block >> duplicated >> in Figure 1. Please consider taking that figure up a level, showing >> the three >> blocks and how they're separated, and let the other figures show the >> internal >> detail. > > This is something that we have discussed and have not found an elegant > solution for. > It was thought to be very valuable to display the pertinent portion of > the complete > CLF within the section where that was discussed. I'll do my best to > ensure that they > are all copied over properly from Figure 1. This has been a common theme in the IESG reviews of many recent documents. The issue is not the care you take while editing this document. It's the burden you put on whoever ends up with the pen on an update. > > >> There is a lot of descriptive text in 4.2's sections on the server >> and client >> transaction that is a copy of what's in the >> problem-statement/information model >> document. It would be better to just point to the description in the >> other document. > > This is something that we debated about quite a bit. In the end we > thought it more > useful for implementers to have the description of all mandatory > fields right in the > format draft. Again, this is something that IESG review has put pressure against in many recent documents. Expect to defend this during that review. At the very least, make it clear which document is authoritative should they conflict. > > >> Please consider scrubbing the message used in example 5 back to the >> minimal >> number of octets you need to make the example useful. In particular, >> consider >> removing elements that have not already been standardized. I'm >> surprised to see >> the literal strings "server-tx" and "client-tx" in the example log >> message. Was >> that what you really intended? > > I'll look to trim down the sample message in Section 5 to a minimal > set of fields. > The use of server-txn and client-txn was intentional and done to > parallel the usage in > the data model draft. If you think a specific value is best, then I'll > make one up. It's confusing as is - the example is supposed to be an actual log message, not a pseudo-code description of one isn't it? > >> > RjS --------------030102090202080107060901 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Inline, trimming to points with responses

On 2/8/12 6:25 PM, Gonzalo Salgueiro wrote:
Robert - 

Thanks for your thorough and thoughtful review. Below are some inline comments to your feedback. 

I will release the -06 version of the draft once we have settled on all of the below points you have raised.


On Jan 26, 2012, at 6:09 PM, Robert Sparks wrote:

AD review: draft-ietf-sipclf-format-05

Summary: This document has open issues to address before progressing to IETF LC

Major (mostly in document order)

<snip/>

2) The description of Timestamp and Fractional Seconds is very confusing.
What does it mean to be represented in big-endian fashion with a most
significant octet and to be decimal encoded? It's very far from clear that
you intend this to be an ascii string representing an integer, and it's
not clear that you intend that it to be prefix-padded with 0s if necessary.
Did the group consider using ABNF to describe these fields (or the whole
log record) to avoid this kind of ambiguity?

Good point. The text needs clarifying. I will remove the portion of the text 
that states "Represented in big-endian fashion with most significant octet 
first from zero starting at the left, or high-order, position.  Decimal encoded." 
from the description of both Timestamp and Fractional Seconds fields. 
I will also supplement the text with the following summary text:

The timestamp is represented in the log file as a UTF-8 string
representing the date and time of the request or response
represented as the number of seconds and milliseconds since the
Unix epoch.  The number of milliseconds MUST be separated by a
"." from the number of seconds.  It is not required to zero-pad
the seconds and milliseconds, although if they are zero padded,
a SIP CLF reader MUST be able to interpret them by discarding
leading zeroes.
Actually, I was under the impression that the zero padding was
expected. The examples have them consistently - why did that happen?
Either way, group members should say something about what the intent was.


3) The document should be clear about whether the flags field values are
extensible, and if so, how extension is expected to be managed. How will
you handle SIP received over DCCP, Websockets, or something we haven't
heard of yet?

About the only flag that is expected to be extended is the Transport
Flag (byte 4).  Currently, the format draft defines 3 values for
it (UDP, TCP, SCTP).  If Websockets and DCCP become supported transports
in SIP, then yes, this field will need to be extended.  We can do one
of two things for extension:

 1) Have IANA open up a registry for this field
 2) Use the Options part of the SIPCLF record to encode the transport.
  That is, we define byte 4 -- Transport Flag to contain:

  U = UDP
  T = TCP
  S = SCTP
  O = Other

  Then, when a new transport is supported, a log writer MUST put "O"
  for the Transport Flag and MUST create an optional field using
  Vendor-ID 00000000.  Example: when DCCP becomes supported, it will
  be encoded as follows:

  00@00000000,000F,Transport: DCCP


If we don't think that it is reasonable for new transports to be ascribed a 
value of 'Other' in the Transport Flag field, then the IANA registry is the 
more elegant solution. Let me know your thoughts on a way forward here 
since the draft currently has no IANA dependencies.
That's a choice the group should make - the document just needs to be explicit
about what's extensible (for all the fields) and if a field is extensible, what the mechanism
looks like. For the transport field, as an individual contributor, I prefer the IANA registry approach.

<snip/>

5) The document should more explicitly discuss the decision to replace
tabs with spaces. That decision was based on there being no known use
of tabs in SIP messages that means something semantically other than
whitespace. The discussion should point that out, and note that you
will not be able to use the logged information to reconstruct a
signature over the logged fields, and that any future (or proprietary)
SIP header fields that include tabs with semantic meaning (carrying a
blob perhaps) will lose this meaning while being logged. It should also
be made brutally explicit whether you want this substitution to happen
when you are logging the entire message, or a body (that might happen
to have tabs in it).

We will add a new paragraph that goes something
like this (the new paragraph is added at the end of S4.2 paragraph
"This data MUST appear ... (0x20) prior to being logged."):

 The decision to replace tabs with spaces was based on there
 being no known use of tabs in SIP messages to convey any other
 meaning than whitespace.  Two consequences of the decision to
 replace tab with a space character are: (a) it will become
 impossible to reconstruct a signature over the logged field that
 matches the signature over fields in the original SIP message,
 and (b) any future SIP header fields that include tabs with a
 different semantic meaning than simply signifying whitespace
 will loose this meaning when logged.  And finally, the tabs to
 spaces substitution MUST occur when logging mandatory fields;
 it SHOULD occur when when logging either the entire message or
 simply a SIP body.
This leaves out optional fields. It's also not clear why you use a SHOULD
for the requirement over bodies.

You should also make it clear (as part of escaping above) whether any
escaping is done before encoding binary bodies.

6) There are a couple of places where you talk about escaping '-' and '?'
characters. The document should note that this could produce a logged record
that is not syntactically parsible with a SIP parser (quoted-pair is only
used in certain places in SIP messages). There should also be some discussion
about whether a log reader unescapes anything.

It was never the intent for a SIP CLF to be read by a SIP parser as it would be 
entirely nonsensical to it.
I think it _was_ the intent to be able to reuse parser code.
Several early implementers were handing individual fields to the part of their parser that
would handle a framed header field.

The belief is that the reader of the SIP CLF would know 
what to do with it. I'll add a statement to the draft indicating this explicitly.
It will surprise me if a simple statement will fully address this point - be sure to
capture whether the reader unescapes anyththing.

7) Why does Tag -01 restrict message bodies to just the listed types, especially
with the open ended "miscellaneous text content" type category? If there's a real
restriction here, it needs to be much more clearly stated. Why are you allowing
any LWS for the separator between the content type header field value and the
message body itself? Why isn't this simply a space? It is not clear what
"byte encoded" means for representing binary bodies - do you mean expanding
it into a hex representation, a uuencode representation, or soemthing else?

The decision to restrict logging of bodies to the specific types listed in the draft 
was a decision made by the WG [1].
The document needs to explain _why_ this particular subset was chosen.
What basis did the group use for that choice? My read of the thread was that Vijay
threw an initial list together and that just ended up in the document. It's not clear
that the list was intended to be exclusive, just that whatever subsequent decisions
that were made allowed at least those types.

WHY are you not allowing images? Why are you not allowing multipart/mime?
(If you ever did actually see an S/MIME protected body, you would not be able to log it
the way the document stands now.)
The usage of a LWS separator was specified 
as part of that proposal for logging bodies. Making the separator a space 
instead of LWS is fine, in fact, it is preferable. I'll update the draft to reflect that 
change.

I'll also draw a line in the sand and update the text indicating that "byte encoded" 
in this context means that bodies with binary data MUST be base64 encoded for 
logging (since MIME uses base64 encoding).
Please be careful to be specific in the instructions on when to encode
(for example, would you re-encode a binary format that was already base64 encoded?)

<snip/>
9) When describing repeating tag-01 or tag-02 multiple times to deal with
large entities, the document should be clear that the fragments get logged
in order, and how to reassemble the actual entity. It should also say whether
it's ok to interleave (for example) tag-01 fragements with other optional fields.

This is a great callout. There is text in the description of Tag=01 that needs to 
be updated to clearly state that if your body or the entire SIP
message is > 4096 bytes, then tough luck. We can't chunk this as it would open 
up a can of worms that we don't want to deal with; like indexing the chunks 
and a reassembly scheme. Better not to go there. I will update the text 
to clearly state this point.

Now I'm confused. Why do you have the ability to repeat these tags?

<snip/>

11) Where does the document talk about how to log fields that are line folded
in the original message? Is it ok for the logged field to contain CRLFs, or is
the intent that the field be unfolded before logging?

I will update the text to indicate that:

For mandatory fields:

 The mandatory fields MUST NOT contain CRLF when logged.

For optional fields:

If an optional field contains a CRLF as part of its normal 
production rule, the CRLF MUST be escaped by using the 
URI encoded equivalent value of "%0D%0A".
Why are you treating optional fields and mandatory fields differently here?
If there's a good reason, fine, but be sure to include the reason in the document.

For logging bodies:

All CRLF digraphs in the SIP body MUST be escaped by using the 
URI encoded equivalent value of "%0D%0A"

For logging the entire SIP message:

All instances of CRLF digraphs, whether they appear in the SIP
headers or the body MUST be escaped by using the URI encoded 
equivalent value of "%0D%0A".
Even when the body is binary (see previous discussion about escaping)?
The group should weigh in on whether this is what they were expecting for
the usual application/sdp case.

12) It's not clear if the \r\n used in the examples of logging the SDP body
are there as text-conventions for showing the example in an RFC, or if you
intend that CRLFs be replaced by \r\n's when logging. If it's just a convention,
please make it clear that the actual log record will have CRLFs. If you were
expecting the CRLFs to be escaped in the log, there needs to be text that
says that normatively somewhere, and you need to consider escaping the escape
character.

As per my comments for point #11 above, the \r\n in that example needs to be 
escaped with "%0D%0A". I'll update the examples to match with this new 
text that I will be adding.

Minor

There is a maintenance risk in having all the detail of each block duplicated
in Figure 1. Please consider taking that figure up a level, showing the three
blocks and how they're separated, and let the other figures show the internal
detail.

This is something that we have discussed and have not found an elegant solution for. 
It was thought to be very valuable to display the pertinent portion of the complete 
CLF within the section where that was discussed. I'll do my best to ensure that they 
are all copied over properly from Figure 1.
This has been a common theme in the IESG reviews of many recent documents. The issue
is not the care you take while editing this document. It's the burden you put on whoever
ends up with the pen on an update.


There is a lot of descriptive text in 4.2's sections on the server and client
transaction that is a copy of what's in the problem-statement/information model
document. It would be better to just point to the description in the other document.

This is something that we debated about quite a bit. In the end we thought it more 
useful for implementers to have the description of all mandatory fields right in the 
format draft.
Again, this is something that IESG review has put pressure against in many recent
documents. Expect to defend this during that review. At the very least, make it clear
which document is authoritative should they conflict.


Please consider scrubbing the message used in example 5 back to the minimal
number of octets you need to make the example useful. In particular, consider
removing elements that have not already been standardized. I'm surprised to see
the literal strings "server-tx" and "client-tx" in the example log message. Was
that what you really intended?

I'll look to trim down the sample message in Section 5 to a minimal set of fields. 
The use of server-txn and client-txn was intentional and done to parallel the usage in 
the data model draft. If you think a specific value is best, then I'll make one up.
It's confusing as is - the example is supposed to be an actual log message, not a
pseudo-code description of one isn't it?




RjS
--------------030102090202080107060901-- From vkg@bell-labs.com Thu Feb 9 09:58:14 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 966D121E8014 for ; Thu, 9 Feb 2012 09:58:14 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.877 X-Spam-Level: X-Spam-Status: No, score=-106.877 tagged_above=-999 required=5 tests=[AWL=-0.278, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id r6xlJ08ygBLj for ; Thu, 9 Feb 2012 09:58:13 -0800 (PST) Received: from ihemail3.lucent.com (ihemail3.lucent.com [135.245.0.37]) by ietfa.amsl.com (Postfix) with ESMTP id A56A321F85A4 for ; Thu, 9 Feb 2012 09:58:13 -0800 (PST) Received: from usnavsmail3.ndc.alcatel-lucent.com (usnavsmail3.ndc.alcatel-lucent.com [135.3.39.11]) by ihemail3.lucent.com (8.13.8/IER-o) with ESMTP id q19HwCQ4003517 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 9 Feb 2012 11:58:12 -0600 (CST) Received: from umail.lucent.com (umail-ce2.ndc.lucent.com [135.3.40.63]) by usnavsmail3.ndc.alcatel-lucent.com (8.14.3/8.14.3/GMO) with ESMTP id q19HwBhL015429 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 9 Feb 2012 11:58:11 -0600 Received: from shoonya.ih.lucent.com (shoonya-135185238235.ih.lucent.com [135.185.238.235]) by umail.lucent.com (8.13.8/TPES) with ESMTP id q19HwAZ2008559; Thu, 9 Feb 2012 11:58:11 -0600 (CST) Message-ID: <4F340A33.7030606@bell-labs.com> Date: Thu, 09 Feb 2012 12:02:27 -0600 From: "Vijay K. Gurbani" Organization: Bell Laboratories, Alcatel-Lucent User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20111222 Thunderbird/9.0 MIME-Version: 1.0 To: Robert Sparks References: <4F21DD3E.7000002@nostrum.com> <31A5C897-B767-4527-9346-905A80977F35@cisco.com> <4F33E3D0.7000605@nostrum.com> In-Reply-To: <4F33E3D0.7000605@nostrum.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.57 on 135.245.2.37 X-Scanned-By: MIMEDefang 2.64 on 135.3.39.11 Cc: "sip-clf@ietf.org" Subject: Re: [sip-clf] AD review: draft-ietf-sipclf-format-05 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 17:58:14 -0000 On 02/09/2012 09:18 AM, Robert Sparks wrote: > Inline, trimming to points with responses Robert: Some feedback on your responses inline. With respect to extending the Transport flag, you wrote that: > That's a choice the group should make - the document just needs to be > explicit about what's extensible (for all the fields) and if a field > is extensible, what the mechanism looks like. For the transport > field, as an individual contributor, I prefer the IANA registry > approach. I agree that IANA is cleaner. It will be nice if we hear more views from the WG so we can proceed accordingly. With respect to replacing tabs with spaces, you wrote that: > This leaves out optional fields. It's also not clear why you use a SHOULD > for the requirement over bodies. I believe that the SIP CLF record will be used as a medium to do a quick spot check as well as a medium for detailed analysis. When used as a medium for a quick spot check, I believe that the SIP CLF record should be amenable to being manipulated by normal Unix text processing tools. Thus, those who use the standard Unix command line text tools to look for something in the SIP CLF log file will be mostly interested in the mandatory header fields (i.e., grep "2 INVITE" logfile or grep "S2388-188" logfile to find all records with server transaction ID "S2388-188"). The result could be piped to tr(1), awk(1), cut(1) or perl for further processing. Quick example: $ grep "2 INVITE" logfile | cut -f13 will print the server transaction ID (field 13) of all records where the CSeq field is "2 INVITE". That is why I think that tabs in the mandatory fields MUST be changed to spaces. Exploring the body and optional fields is part of detailed analysis of the SIP CLF record. To do this, one would need an actual SIP CLF reader, which will be presumably more intelligent and does not need a LWS delimiter to figure out boundaries (it already has the length of the body or the optional field). In fact, the more I think of it the more it seems that the SIP CLF record should really consist of three distinct stanzas, each separated by a LF: Index-pointers 0x0A Mandatory-fields 0x0A Optional-fields 0x0A Right now, we separate the SIP CLF record in two stanzas: Index-pointers 0x0A Mandatory-Fields-and-Optional-fields 0x0A The problem with separating the SIP CLF record in two stanzas is that a "grep S2388-188 logfile" will return the mandatory field line PLUS any optional fields logged. These optional fields are therefore forced to change tab to spaces. If we have 3 stanzas, then the optional fields can retain tabs. Regarding the SIP CLF record being syntactically amenable to a parse by a normal SIP parser, you wrote that: > I think it _was_ the intent to be able to reuse parser code. Several > early implementers were handing individual fields to the part of > their parser that would handle a framed header field. The distinction being made here is whether individual *fields* in the SIP CLF record can be extracted and given to a SIP parser that would normally parse those sequence of strings versus an existing SIP parser that will successfully parse a SIP CLF record in its *entirety*. Clearly, the latter is not possible. The former is a possibility. To be safe, the document should indicate that escaping '-' and '?' may produce a field that is not syntactically parsed by a SIP parser that would normally be able to parse that field if it appeared in a SIP header (as opposed to a log file). Thanks, - vijay -- Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent 1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60563 (USA) Email: vkg@{bell-labs.com,acm.org} / vijay.gurbani@alcatel-lucent.com Web: http://ect.bell-labs.com/who/vkg/ From gsalguei@cisco.com Thu Feb 9 13:01:35 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 70CF821E804A for ; Thu, 9 Feb 2012 13:01:35 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.406 X-Spam-Level: X-Spam-Status: No, score=-10.406 tagged_above=-999 required=5 tests=[AWL=0.192, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z0jSE0ftvA+0 for ; Thu, 9 Feb 2012 13:01:34 -0800 (PST) Received: from av-tac-rtp.cisco.com (hen.cisco.com [64.102.19.198]) by ietfa.amsl.com (Postfix) with ESMTP id 9D94421E8019 for ; Thu, 9 Feb 2012 13:01:34 -0800 (PST) X-TACSUNS: Virus Scanned Received: from chook.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-rtp.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q19L1XFa006846 for ; Thu, 9 Feb 2012 16:01:33 -0500 (EST) Received: from dhcp-64-102-210-22.cisco.com (dhcp-64-102-210-22.cisco.com [64.102.210.22]) by chook.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q19L1LNS026610; Thu, 9 Feb 2012 16:01:21 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-33-474246821 From: Gonzalo Salgueiro In-Reply-To: <4F340A33.7030606@bell-labs.com> Date: Thu, 9 Feb 2012 16:01:21 -0500 Message-Id: <446A716C-A606-4812-AD1E-5CB8739A9F2D@cisco.com> References: <4F21DD3E.7000002@nostrum.com> <31A5C897-B767-4527-9346-905A80977F35@cisco.com> <4F33E3D0.7000605@nostrum.com> <4F340A33.7030606@bell-labs.com> To: "Vijay K. Gurbani" X-Mailer: Apple Mail (2.1084) Cc: "sip-clf@ietf.org" Subject: Re: [sip-clf] AD review: draft-ietf-sipclf-format-05 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 21:01:35 -0000 --Apple-Mail-33-474246821 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Feb 9, 2012, at 1:02 PM, Vijay K. Gurbani wrote: > On 02/09/2012 09:18 AM, Robert Sparks wrote: >> Inline, trimming to points with responses >=20 > Robert: Some feedback on your responses inline. >=20 > With respect to extending the Transport flag, you wrote that: >=20 >> That's a choice the group should make - the document just needs to be >> explicit about what's extensible (for all the fields) and if a field >> is extensible, what the mechanism looks like. For the transport >> field, as an individual contributor, I prefer the IANA registry >> approach. >=20 > I agree that IANA is cleaner. It will be nice if we hear more views > from the WG so we can proceed accordingly. I'll weigh in that I think the IANA registration route is more elegant = and practical from an extensibility point of view. Regards, Gonzalo --Apple-Mail-33-474246821 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
On 02/09/2012 09:18 AM, Robert Sparks = wrote:
Inline, trimming to points with = responses

Robert: Some feedback on your responses = inline.

With respect to extending the Transport flag, you wrote = that:

That's a choice the group should = make - the document just needs to be
explicit about what's extensible (for all the fields) and = if a field
is extensible, what = the mechanism looks like. For the transport
field, as an individual contributor, I prefer the IANA = registry
approach.

I agree that IANA is = cleaner.  It will be nice if we hear more views
from the WG so = we can proceed accordingly.

I'll = weigh in that I think the IANA registration route is more elegant and = practical from an extensibility point of = view.

Regards,

Gonzalo


= --Apple-Mail-33-474246821-- From gsalguei@cisco.com Thu Feb 9 14:53:49 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D28FA21E8044 for ; Thu, 9 Feb 2012 14:53:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.434 X-Spam-Level: X-Spam-Status: No, score=-10.434 tagged_above=-999 required=5 tests=[AWL=0.164, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M53NG748G3p1 for ; Thu, 9 Feb 2012 14:53:48 -0800 (PST) Received: from av-tac-rtp.cisco.com (hen.cisco.com [64.102.19.198]) by ietfa.amsl.com (Postfix) with ESMTP id 01FA811E8073 for ; Thu, 9 Feb 2012 14:53:47 -0800 (PST) X-TACSUNS: Virus Scanned Received: from chook.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-rtp.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q19Mrlkb017549 for ; Thu, 9 Feb 2012 17:53:47 -0500 (EST) Received: from dhcp-64-102-210-22.cisco.com (dhcp-64-102-210-22.cisco.com [64.102.210.22]) by chook.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q19Mrjst029004; Thu, 9 Feb 2012 17:53:45 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-40-480990469 From: Gonzalo Salgueiro In-Reply-To: <4F33E3D0.7000605@nostrum.com> Date: Thu, 9 Feb 2012 17:53:44 -0500 Message-Id: <02D40CF4-CDB3-4962-A5DE-832DA1E0C457@cisco.com> References: <4F21DD3E.7000002@nostrum.com> <31A5C897-B767-4527-9346-905A80977F35@cisco.com> <4F33E3D0.7000605@nostrum.com> To: Robert Sparks X-Mailer: Apple Mail (2.1084) Cc: draft-ietf-sipclf-format@tools.ietf.org, sipclf-chairs@tools.ietf.org, "sip-clf@ietf.org Mailing" Subject: Re: [sip-clf] AD review: draft-ietf-sipclf-format-05 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 22:53:49 -0000 --Apple-Mail-40-480990469 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi Robert -=20 Thanks for your responsiveness on this. I'll close the loop on just = those points=20 that Vijay didn't already touch on in his response. >>> 2) The description of Timestamp and Fractional Seconds is very = confusing. >>> What does it mean to be represented in big-endian fashion with a = most >>> significant octet and to be decimal encoded? It's very far from = clear that >>> you intend this to be an ascii string representing an integer, and = it's >>> not clear that you intend that it to be prefix-padded with 0s if = necessary. >>> Did the group consider using ABNF to describe these fields (or the = whole >>> log record) to avoid this kind of ambiguity? >>=20 >> Good point. The text needs clarifying. I will remove the portion of = the text=20 >> that states "Represented in big-endian fashion with most significant = octet=20 >> first from zero starting at the left, or high-order, position. = Decimal encoded."=20 >> from the description of both Timestamp and Fractional Seconds fields.=20= >> I will also supplement the text with the following summary text: >>=20 >> The timestamp is represented in the log file as a UTF-8 string >> representing the date and time of the request or response >> represented as the number of seconds and milliseconds since the >> Unix epoch. The number of milliseconds MUST be separated by a >> "." from the number of seconds. It is not required to zero-pad >> the seconds and milliseconds, although if they are zero padded, >> a SIP CLF reader MUST be able to interpret them by discarding >> leading zeroes. > Actually, I was under the impression that the zero padding was > expected. The examples have them consistently - why did that happen? > Either way, group members should say something about what the intent = was. The examples will be updated to reflect the updated text regarding the=20= zero padding. =20 >>> 7) Why does Tag -01 restrict message bodies to just the listed = types, especially >>> with the open ended "miscellaneous text content" type category? If = there's a real >>> restriction here, it needs to be much more clearly stated. Why are = you allowing >>> any LWS for the separator between the content type header field = value and the >>> message body itself? Why isn't this simply a space? It is not clear = what >>> "byte encoded" means for representing binary bodies - do you mean = expanding >>> it into a hex representation, a uuencode representation, or = soemthing else? >>=20 >> The decision to restrict logging of bodies to the specific types = listed in the draft=20 >> was a decision made by the WG [1]. > The document needs to explain _why_ this particular subset was chosen. > What basis did the group use for that choice? My read of the thread = was that Vijay > threw an initial list together and that just ended up in the document. = It's not clear > that the list was intended to be exclusive, just that whatever = subsequent decisions > that were made allowed at least those types. >=20 > WHY are you not allowing images? Why are you not allowing = multipart/mime? > (If you ever did actually see an S/MIME protected body, you would not = be able to log it > the way the document stands now.) We floated some text on the list that contained a list of those body = types the WG thought=20 MUST be logged. Nobody responded with an alternative, neither did = anyone respond=20 against the proposal. So we went with it. I'm happy to reopen this = discussion again on the=20 list, but if no one else answers than we are no better off than before. I share your concern that we are being needlessly restrictive so I'll = put forth a proposal=20 to the WG in the next version of the draft that removes the constraint = of logging bodies=20 of those specified types. The logging mechanism will remain unchanged = but without=20 body type restrictions. >> The usage of a LWS separator was specified=20 >> as part of that proposal for logging bodies. Making the separator a = space=20 >> instead of LWS is fine, in fact, it is preferable. I'll update the = draft to reflect that=20 >> change. >>=20 >> I'll also draw a line in the sand and update the text indicating that = "byte encoded"=20 >> in this context means that bodies with binary data MUST be base64 = encoded for=20 >> logging (since MIME uses base64 encoding). > Please be careful to be specific in the instructions on when to encode > (for example, would you re-encode a binary format that was already = base64 encoded?) Sounds good. I'll add text to address this. >>> 9) When describing repeating tag-01 or tag-02 multiple times to deal = with >>> large entities, the document should be clear that the fragments get = logged >>> in order, and how to reassemble the actual entity. It should also = say whether >>> it's ok to interleave (for example) tag-01 fragements with other = optional fields. >>=20 >> This is a great callout. There is text in the description of Tag=3D01 = that needs to=20 >> be updated to clearly state that if your body or the entire SIP >> message is > 4096 bytes, then tough luck. We can't chunk this as it = would open=20 >> up a can of worms that we don't want to deal with; like indexing the = chunks=20 >> and a reassembly scheme. Better not to go there. I will update the = text=20 >> to clearly state this point. >=20 > Now I'm confused. Why do you have the ability to repeat these tags? It *is* currently confusing. The updated text will remove this ability = to have multiple=20 Tag -01 and -02 entries. The only repeated Tag that will be allowed is = -00 ONLY when=20 logging an optional field that occurs more than once in a SIP message = (e.g. Contact,=20 Route, Record-Route, etc.). >>> 11) Where does the document talk about how to log fields that are = line folded >>> in the original message? Is it ok for the logged field to contain = CRLFs, or is >>> the intent that the field be unfolded before logging? >>=20 >> I will update the text to indicate that: >>=20 >> For mandatory fields: >>=20 >> The mandatory fields MUST NOT contain CRLF when logged. >>=20 >> For optional fields: >>=20 >> If an optional field contains a CRLF as part of its normal=20 >> production rule, the CRLF MUST be escaped by using the=20 >> URI encoded equivalent value of "%0D%0A". > Why are you treating optional fields and mandatory fields differently = here? > If there's a good reason, fine, but be sure to include the reason in = the document. I'm simply laying out the escape rules for the various CLF log elements. >> For logging bodies: >>=20 >> All CRLF digraphs in the SIP body MUST be escaped by using the=20 >> URI encoded equivalent value of "%0D%0A" >>=20 >> For logging the entire SIP message: >>=20 >> All instances of CRLF digraphs, whether they appear in the SIP >> headers or the body MUST be escaped by using the URI encoded=20 >> equivalent value of "%0D%0A". > Even when the body is binary (see previous discussion about escaping)? > The group should weigh in on whether this is what they were expecting = for > the usual application/sdp case. I think _ANY_ time a CRLF is present in a body it is escaped. If it = isn't there in a=20 binary body, then there is no need to escape. >>> Minor >>>=20 >>> There is a maintenance risk in having all the detail of each block = duplicated >>> in Figure 1. Please consider taking that figure up a level, showing = the three >>> blocks and how they're separated, and let the other figures show the = internal >>> detail. >>=20 >> This is something that we have discussed and have not found an = elegant solution for.=20 >> It was thought to be very valuable to display the pertinent portion = of the complete=20 >> CLF within the section where that was discussed. I'll do my best to = ensure that they=20 >> are all copied over properly from Figure 1. > This has been a common theme in the IESG reviews of many recent = documents. The issue > is not the care you take while editing this document. It's the burden = you put on whoever > ends up with the pen on an update. I certainly understand. I just don't have a better solution at this = time. >>> There is a lot of descriptive text in 4.2's sections on the server = and client >>> transaction that is a copy of what's in the = problem-statement/information model >>> document. It would be better to just point to the description in the = other document. >>=20 >> This is something that we debated about quite a bit. In the end we = thought it more=20 >> useful for implementers to have the description of all mandatory = fields right in the=20 >> format draft. > Again, this is something that IESG review has put pressure against in = many recent > documents. Expect to defend this during that review. At the very = least, make it clear > which document is authoritative should they conflict. I'll do my best to defend this because I think it is what is best for = this document and its=20 'one stop shop' readability for implementers. If this turns out to be a = battle not worth=20 waging with the IESG then I will concede and add a reference. >>> Please consider scrubbing the message used in example 5 back to the = minimal >>> number of octets you need to make the example useful. In particular, = consider >>> removing elements that have not already been standardized. I'm = surprised to see >>> the literal strings "server-tx" and "client-tx" in the example log = message. Was >>> that what you really intended? >>=20 >> I'll look to trim down the sample message in Section 5 to a minimal = set of fields.=20 >> The use of server-txn and client-txn was intentional and done to = parallel the usage in=20 >> the data model draft. If you think a specific value is best, then = I'll make one up. > It's confusing as is - the example is supposed to be an actual log = message, not a > pseudo-code description of one isn't it? Yes, it should be an actual log message. I'll update as requested. Thanks, Gonzalo >>=20 >>>=20 >>=20 >=20 > RjS --Apple-Mail-40-480990469 Content-Transfer-Encoding: 7bit Content-Type: text/html; charset=us-ascii Hi Robert - 

Thanks for your responsiveness on this. I'll close the loop on just those points 
that Vijay didn't already touch on in his response.

<snip>

2) The description of Timestamp and Fractional Seconds is very confusing.
What does it mean to be represented in big-endian fashion with a most
significant octet and to be decimal encoded? It's very far from clear that
you intend this to be an ascii string representing an integer, and it's
not clear that you intend that it to be prefix-padded with 0s if necessary.
Did the group consider using ABNF to describe these fields (or the whole
log record) to avoid this kind of ambiguity?

Good point. The text needs clarifying. I will remove the portion of the text 
that states "Represented in big-endian fashion with most significant octet 
first from zero starting at the left, or high-order, position.  Decimal encoded." 
from the description of both Timestamp and Fractional Seconds fields. 
I will also supplement the text with the following summary text:

The timestamp is represented in the log file as a UTF-8 string
representing the date and time of the request or response
represented as the number of seconds and milliseconds since the
Unix epoch.  The number of milliseconds MUST be separated by a
"." from the number of seconds.  It is not required to zero-pad
the seconds and milliseconds, although if they are zero padded,
a SIP CLF reader MUST be able to interpret them by discarding
leading zeroes.
Actually, I was under the impression that the zero padding was
expected. The examples have them consistently - why did that happen?
Either way, group members should say something about what the intent was.

The examples will be updated to reflect the updated text regarding the 
zero padding.

<snip/> 
<snip>

7) Why does Tag -01 restrict message bodies to just the listed types, especially
with the open ended "miscellaneous text content" type category? If there's a real
restriction here, it needs to be much more clearly stated. Why are you allowing
any LWS for the separator between the content type header field value and the
message body itself? Why isn't this simply a space? It is not clear what
"byte encoded" means for representing binary bodies - do you mean expanding
it into a hex representation, a uuencode representation, or soemthing else?

The decision to restrict logging of bodies to the specific types listed in the draft 
was a decision made by the WG [1].
The document needs to explain _why_ this particular subset was chosen.
What basis did the group use for that choice? My read of the thread was that Vijay
threw an initial list together and that just ended up in the document. It's not clear
that the list was intended to be exclusive, just that whatever subsequent decisions
that were made allowed at least those types.

WHY are you not allowing images? Why are you not allowing multipart/mime?
(If you ever did actually see an S/MIME protected body, you would not be able to log it
the way the document stands now.)

We floated some text on the list that contained a list of those body types the WG thought 
MUST be logged.  Nobody responded with an alternative, neither did anyone respond 
against the proposal.  So we went with it.  I'm happy to reopen this discussion again on the 
list, but if no one else answers than we are no better off than before.

I share your concern that we are being needlessly restrictive so I'll put forth a proposal 
to the WG in the next version of the draft that removes the constraint of logging bodies 
of those specified types. The logging mechanism will remain unchanged but without 
body type restrictions.

The usage of a LWS separator was specified 
as part of that proposal for logging bodies. Making the separator a space 
instead of LWS is fine, in fact, it is preferable. I'll update the draft to reflect that 
change.

I'll also draw a line in the sand and update the text indicating that "byte encoded" 
in this context means that bodies with binary data MUST be base64 encoded for 
logging (since MIME uses base64 encoding).
Please be careful to be specific in the instructions on when to encode
(for example, would you re-encode a binary format that was already base64 encoded?)

Sounds good. I'll add text to address this.

9) When describing repeating tag-01 or tag-02 multiple times to deal with
large entities, the document should be clear that the fragments get logged
in order, and how to reassemble the actual entity. It should also say whether
it's ok to interleave (for example) tag-01 fragements with other optional fields.

This is a great callout. There is text in the description of Tag=01 that needs to 
be updated to clearly state that if your body or the entire SIP
message is > 4096 bytes, then tough luck. We can't chunk this as it would open 
up a can of worms that we don't want to deal with; like indexing the chunks 
and a reassembly scheme. Better not to go there. I will update the text 
to clearly state this point.

Now I'm confused. Why do you have the ability to repeat these tags?

It *is* currently confusing. The updated text will remove this ability to have multiple 
Tag -01 and -02 entries. The only repeated Tag that will be allowed is -00 ONLY when 
logging an optional field that occurs more than once in a SIP message (e.g. Contact, 
Route, Record-Route, etc.).

11) Where does the document talk about how to log fields that are line folded
in the original message? Is it ok for the logged field to contain CRLFs, or is
the intent that the field be unfolded before logging?

I will update the text to indicate that:

For mandatory fields:

 The mandatory fields MUST NOT contain CRLF when logged.

For optional fields:

If an optional field contains a CRLF as part of its normal 
production rule, the CRLF MUST be escaped by using the 
URI encoded equivalent value of "%0D%0A".
Why are you treating optional fields and mandatory fields differently here?
If there's a good reason, fine, but be sure to include the reason in the document.

I'm simply laying out the escape rules for the various CLF log elements.

For logging bodies:

All CRLF digraphs in the SIP body MUST be escaped by using the 
URI encoded equivalent value of "%0D%0A"

For logging the entire SIP message:

All instances of CRLF digraphs, whether they appear in the SIP
headers or the body MUST be escaped by using the URI encoded 
equivalent value of "%0D%0A".
Even when the body is binary (see previous discussion about escaping)?
The group should weigh in on whether this is what they were expecting for
the usual application/sdp case.

I think _ANY_ time a CRLF is present in a body it is escaped. If it isn't there in a 
binary body, then there is no need to escape.

Minor

There is a maintenance risk in having all the detail of each block duplicated
in Figure 1. Please consider taking that figure up a level, showing the three
blocks and how they're separated, and let the other figures show the internal
detail.

This is something that we have discussed and have not found an elegant solution for. 
It was thought to be very valuable to display the pertinent portion of the complete 
CLF within the section where that was discussed. I'll do my best to ensure that they 
are all copied over properly from Figure 1.
This has been a common theme in the IESG reviews of many recent documents. The issue
is not the care you take while editing this document. It's the burden you put on whoever
ends up with the pen on an update.

I certainly understand. I just don't have a better solution at this time.

There is a lot of descriptive text in 4.2's sections on the server and client
transaction that is a copy of what's in the problem-statement/information model
document. It would be better to just point to the description in the other document.

This is something that we debated about quite a bit. In the end we thought it more 
useful for implementers to have the description of all mandatory fields right in the 
format draft.
Again, this is something that IESG review has put pressure against in many recent
documents. Expect to defend this during that review. At the very least, make it clear
which document is authoritative should they conflict.

I'll do my best to defend this because I think it is what is best for this document and its 
'one stop shop' readability for implementers. If this turns out to be a battle not worth 
waging with the IESG then I will concede and add a reference.

Please consider scrubbing the message used in example 5 back to the minimal
number of octets you need to make the example useful. In particular, consider
removing elements that have not already been standardized. I'm surprised to see
the literal strings "server-tx" and "client-tx" in the example log message. Was
that what you really intended?

I'll look to trim down the sample message in Section 5 to a minimal set of fields. 
The use of server-txn and client-txn was intentional and done to parallel the usage in 
the data model draft. If you think a specific value is best, then I'll make one up.
It's confusing as is - the example is supposed to be an actual log message, not a
pseudo-code description of one isn't it?

Yes, it should be an actual log message. I'll update as requested.

Thanks,

Gonzalo





RjS

--Apple-Mail-40-480990469-- From musgravepj@gmail.com Thu Feb 16 02:44:21 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D962621F8772 for ; Thu, 16 Feb 2012 02:44:21 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.369 X-Spam-Level: X-Spam-Status: No, score=-3.369 tagged_above=-999 required=5 tests=[AWL=0.230, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zyUvj83YJTH2 for ; Thu, 16 Feb 2012 02:44:21 -0800 (PST) Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by ietfa.amsl.com (Postfix) with ESMTP id E042521F8770 for ; Thu, 16 Feb 2012 02:44:20 -0800 (PST) Received: by iagf6 with SMTP id f6so3289161iag.31 for ; Thu, 16 Feb 2012 02:44:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:content-type:content-transfer-encoding:subject:date:message-id :to:mime-version:x-mailer; bh=1/71yTe/+qM7Xa2QOkswUfhUhuBTr5gSm7L4xrZFAGw=; b=ZtWOXy8UQ2XgF2xLJXvpx6A2PM+L3I8Zccnp+YOhSkVQ1/B5b5/wP1lna3fFaHJVP0 j5u0eykIpzDZr605wa4oaIcewhZYBo4Cg8hUw6Q1MRh7TIkZ7YgUVpN6MN2FgLVCy5EB lYgf+x4IjGgvQB6qq0mfF51U/BP+QhEcyyv/E= Received: by 10.50.95.166 with SMTP id dl6mr2214113igb.27.1329389060643; Thu, 16 Feb 2012 02:44:20 -0800 (PST) Received: from [192.168.1.104] ([204.237.33.112]) by mx.google.com with ESMTPS id ub10sm1402728igb.7.2012.02.16.02.44.18 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 16 Feb 2012 02:44:19 -0800 (PST) From: Peter Musgrave Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Thu, 16 Feb 2012 05:44:19 -0500 Message-Id: <040C2C1A-2943-4BF8-83B6-835EF6A6231C@gmail.com> To: "sip-clf@ietf.org Mailing" Mime-Version: 1.0 (Apple Message framework v1084) X-Mailer: Apple Mail (2.1084) Subject: [sip-clf] No meeting at IETF83 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Feb 2012 10:44:22 -0000 Hi all,=20 To be clear we are not meeting at IETF83.=20 Over the next few weeks I hope we can look at Vijay and Gonzalo's recent = work and get these docs DONE.=20 Thanks,=20 Peter= From musgravepj@gmail.com Thu Feb 16 02:48:39 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9E24121F86C6 for ; Thu, 16 Feb 2012 02:48:39 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.407 X-Spam-Level: X-Spam-Status: No, score=-3.407 tagged_above=-999 required=5 tests=[AWL=0.192, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n6ffp78Cy+2j for ; Thu, 16 Feb 2012 02:48:35 -0800 (PST) Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by ietfa.amsl.com (Postfix) with ESMTP id 8250921F869E for ; Thu, 16 Feb 2012 02:48:35 -0800 (PST) Received: by iagf6 with SMTP id f6so3294505iag.31 for ; Thu, 16 Feb 2012 02:48:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=phx24B50FgZ4R3YHxJtHVQftY2uPj3BIoQAgSiD3CUc=; b=i6ItgoGqJdBj157pLyg6sOvaTf5o88x5KjlaIMsq63e5ogUcwB78EeRJjdq8UOZnxe uTbK/k3fjlZ+rTaLOlZ2H3t5qeK+Ie7UaJoTQUg4dYA42AoJ5Jbd14lhvCBZOFBFTaiR qK3AVa/77Dyae+8F0fRCoS960VPGSvunXvMqk= Received: by 10.43.44.69 with SMTP id uf5mr1667137icb.41.1329389314969; Thu, 16 Feb 2012 02:48:34 -0800 (PST) Received: from [192.168.1.104] ([204.237.33.112]) by mx.google.com with ESMTPS id f8sm10529500ibl.6.2012.02.16.02.48.33 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 16 Feb 2012 02:48:34 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Peter Musgrave In-Reply-To: <446A716C-A606-4812-AD1E-5CB8739A9F2D@cisco.com> Date: Thu, 16 Feb 2012 05:48:31 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <50CF086B-7495-43CF-8C6A-12AA67E2BF54@gmail.com> References: <4F21DD3E.7000002@nostrum.com> <31A5C897-B767-4527-9346-905A80977F35@cisco.com> <4F33E3D0.7000605@nostrum.com> <4F340A33.7030606@bell-labs.com> <446A716C-A606-4812-AD1E-5CB8739A9F2D@cisco.com> To: Gonzalo Salgueiro X-Mailer: Apple Mail (2.1084) Cc: "sip-clf@ietf.org" Subject: Re: [sip-clf] AD review: draft-ietf-sipclf-format-05 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Feb 2012 10:48:39 -0000 HI all,=20 (as individual) I think a registry is more future-proof.=20 Peter On 2012-02-09, at 4:01 PM, Gonzalo Salgueiro wrote: >=20 > On Feb 9, 2012, at 1:02 PM, Vijay K. Gurbani wrote: >=20 >> On 02/09/2012 09:18 AM, Robert Sparks wrote: >>> Inline, trimming to points with responses >>=20 >> Robert: Some feedback on your responses inline. >>=20 >> With respect to extending the Transport flag, you wrote that: >>=20 >>> That's a choice the group should make - the document just needs to = be >>> explicit about what's extensible (for all the fields) and if a field >>> is extensible, what the mechanism looks like. For the transport >>> field, as an individual contributor, I prefer the IANA registry >>> approach. >>=20 >> I agree that IANA is cleaner. It will be nice if we hear more views >> from the WG so we can proceed accordingly. >=20 > I'll weigh in that I think the IANA registration route is more elegant = and practical from an extensibility point of view. >=20 > Regards, >=20 > Gonzalo >=20 >=20 > _______________________________________________ > sip-clf mailing list > sip-clf@ietf.org > https://www.ietf.org/mailman/listinfo/sip-clf