From gsalguei@cisco.com Thu Jul 12 14:40:19 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5FF5A11E80E3 for ; Thu, 12 Jul 2012 14:40:19 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.742 X-Spam-Level: X-Spam-Status: No, score=-2.742 tagged_above=-999 required=5 tests=[AWL=-0.143, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QkJSE+0Dwisy for ; Thu, 12 Jul 2012 14:40:18 -0700 (PDT) Received: from av-tac-rtp.cisco.com (av-tac-rtp.cisco.com [64.102.19.209]) by ietfa.amsl.com (Postfix) with ESMTP id 2878311E80D2 for ; Thu, 12 Jul 2012 14:40:16 -0700 (PDT) X-TACSUNS: Virus Scanned Received: from chook.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-rtp.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q6CLen5I014028 for ; Thu, 12 Jul 2012 17:40:49 -0400 (EDT) Received: from dhcp-10-150-53-182.cisco.com (dhcp-10-150-53-182.cisco.com [10.150.53.182]) by chook.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q6CLenAi007028; Thu, 12 Jul 2012 17:40:49 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=iso-8859-1 From: Gonzalo Salgueiro In-Reply-To: <4FDA2807.6080802@nostrum.com> Date: Thu, 12 Jul 2012 17:40:49 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <066F0992-9BFD-4CC2-81FF-0697529E179F@cisco.com> References: <4F9EE0A4.2000905@nostrum.com> <4FDA2807.6080802@nostrum.com> To: Robert Sparks X-Mailer: Apple Mail (2.1278) Cc: "sip-clf@ietf.org Mailing" Subject: Re: [sip-clf] AD review: draft-ietf-sipclf-format-06 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jul 2012 21:40:19 -0000 Robert -=20 Inline responses to your comments/questions that required follow up. On Jun 14, 2012, at 2:05 PM, Robert Sparks wrote: >=20 >=20 > On 6/8/12 2:01 PM, Gonzalo Salgueiro wrote: >> Robert - >>=20 >> I apologize for my delayed response. Thanks for the continued = detailed review and feedback. I'll respond to each of your points = inline... >>=20 >> On Apr 30, 2012, at 2:57 PM, Robert Sparks wrote: >>=20 >>> Summary: there are still a few remaining clarifications to capture = before IETFLC on this document. >>>=20 >>> This is an update to the AD review on -05. Thanks for addressing the = majority of the concerns in that review. There are a few that remain, = and the changes introduced some new ones. >>>=20 >>> 1) Thanks for adding the additional description motivating replacing = tabs with spaces when creating log entries. The new text says that this = replacement SHOULD occur when logging bodies or entire messages. Why is = this a SHOULD? When would you not do this replacement? >> I thought through this and can't recall any good reason why it isn't = mandated for messages and bodies as well. We can change it to a MUST. >>=20 >>> The new text says the decision to do this substitution was based on = there being "no known use of tabs in SIP messages". What possible = unknown uses are there? It would be better to say "No standardized use = of tabs". I also think you mean to scope that claim to SIP header = fields. We allow logging arbitrary bodies, and I don't think we have = done the research to claim that tabs are meaningless (other than as = whitespace) in all possible body types. >> I'm fine with your suggested text. I think we can even amplify it a = bit to address your comments around scope. I propose something like: >>=20 >> The decision to replace tabs with spaces was based on there being >> no standardized use of tabs in SIP headers to convey any other >> meaning than whitespace. Tabs may appear in message bodies, and >> in the event that the bodies are logged, the conversion to space >> may cause problems when reconstructing the body from the = corresponding >> log entry. The two consequences of the decision to replace tabs ... >>=20 >> Does that sound reasonable? > Yes >>=20 >>> 2) The description of escaping and encoding in Tag=3D01 is still = ambiguous. You say you must base64 encode any binary body. You also say = you must escape CRLFs. I suspect you intend for those to be mutually = exclusive? What are you expecting the implementer to use to decide if = the body is binary or not? We should be making much more precise use of = the terms defined in the media type specifications to make this clear = (to avoid things like encoding a body that's already encoded). >> Our intent is to be clear that CRLFs are to be escaped for ANY body = type. Is your question about order of operations in regards to escaping = CRLFs and base64 encoding a binary body (something like MIME types of = application/ISUP and application/QSIG)? > application/jpg. Are you going to escape bits of a compressed picture = that just happen to contain the CRLF sequence or not? What part of the = text makes that clear? I think I get what you are hinting at but I need to play it back to you = for verification. The current text states: =3D=3D=3D=3D=3D ...Note that binary bodies MUST be base64encoded to render them in the = SIP CLF log file. If an optionally logged SIP message body contains any CRLFs they MUST be = escaped by using the URI encoded equivalent value of "%0D%0A". This = escaping mechanism applies to all body types. =3D=3D=3D=3D=3D So we don't make any distinction in treatment between the various = possible body types. I don't believe that we should. I think what the = document may be missing to make this escaping mechanism clear is the = order of operation. I believe I need to explicitly state that the = translation to base64 must occur before the escaping. This would = eliminate any ambiguity about the possibility of ever having the escaped = CRLF sequence of %0D%0A.=20 To your specific point, if a binary body (like an image) is present then = it would have to be base64 encoded first and that base64 character = stream could never include the CRLF escape sequence of %0D%0A because = '%' is not a valid base64 character. Would this clarification in the = text around order of operation address the ambiguity in escaping base64 = encoded binary bodies? > Look at 3.1.1.11 in RFC4475 - how would that body be logged? Looking at the message in S3.1.1.11 of RFC4475, the binary body = (represented in hex) is: = 3082015206092A864886F70D010702A08201433082013F0201013109300706052B0E03021A= 300B06092A864886F70D010701318201203082011C020101307C3070310B30090603550406= 13025553311330110603550408130A43616C69666F726E69613111300F0603550407130853= 616E204A6F7365310E300C060355040A1305736970697431293027060355040B1320536970= 6974205465737420436572746966696361746520417574686F726974790208019500710233= 0113300706052B0E03021A300D06092A864886F70D01010105000481808EF466F948F0522D= D2E5978E9D95AAE9F2FE15A06659716292E8DA2AA8D8350A68CEFFAE3CBD2BFF1675DDD564= 8E593DD64728F26220F7E941749E330D9A15EDABDB93D10C42102E7B7289D29CC0C9AE2EFB= C7C0CFF9172F3B027E4FC027E1546DE4B6AA3ABB3E66CCCB5DD6C64B8383149CB8E6FF182D= 944FE57B65BC99D005 If I base64 encode this it produces six lines. When escaped according to = this draft they become: = MIIBUgYJKoZIhvcNAQcCoIIBQzCCAT8CAQExCTAHBgUrDgMCGjALBgkqhkiG9w0BBwExggEgMI= IB%0D%0AHAIBATB8MHAxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMREwDwYDVQ= QHEwhTYW4g%0D%0ASm9zZTEOMAwGA1UEChMFc2lwaXQxKTAnBgNVBAsTIFNpcGl0IFRlc3QgQ2= VydGlmaWNhdGUgQXV0%0D%0AaG9yaXR5AggBlQBxAjMBEzAHBgUrDgMCGjANBgkqhkiG9w0BAQ= EFAASBgI70ZvlI8FIt0uWXjp2V%0D%0Aquny/hWgZllxYpLo2iqo2DUKaM7/rjy9K/8Wdd3VZI= 5ZPdZHKPJiIPfpQXSeMw2aFe2r25PRDEIQ%0D%0ALntyidKcwMmuLvvHwM/5Fy87An5PwCfhVG= 3ktqo6uz5mzMtd1sZLg4MUnLjm/xgtlE/le2W8mdAF%0D%0A > How would it be different if the 0D0A appeared in the binary body = part? If I insert the two bytes '0D0A' in this binary stream I get: = 3082015206092A864886F70D010702A08201433082013F0201013109300706052B0E03021A= 300B06092A864886F70D010701318201203082011C020101307C3070310B30090603550406= 13025553311330110603550408130A43616C69666F726E69613111300F0603550407130853= 616E204A6F7365310E300C060355040A1305736970697431293027060355040B1320536970= 6974205465737420436572746966696361746520417574686F726974790208019500710233= 0113300706052B0E03021A300D06092A864886F70D01010105000481808EF466F948F0522D= D2E5978E9D95AAE9F2FE15A06659716292E8DA2AA8D8350A68CEFFAE3CBD2BFF1675DDD564= 8E593DD64728F26220F7E941749E330D9A15EDABDB93D10C42102E7B7289D29CC0C9AE2EFB= C7C0CFF9172F3B027E4FC027E1546DE4B6AA3ABB3E66CCC0D0AB5DD6C64B8383149CB8E6FF= 182D944FE57B65BC99D005 Which when base64 encoded and the CRLF's escaped you would get: = MIIBUgYJKoZIhvcNAQcCoIIBQzCCAT8CAQExCTAHBgUrDgMCGjALBgkqhkiG9w0BBwExggEgMI= IB%0D%0AHAIBATB8MHAxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMREwDwYDVQ= QHEwhTYW4g%0D%0ASm9zZTEOMAwGA1UEChMFc2lwaXQxKTAnBgNVBAsTIFNpcGl0IFRlc3QgQ2= VydGlmaWNhdGUgQXV0%0D%0AaG9yaXR5AggBlQBxAjMBEzAHBgUrDgMCGjANBgkqhkiG9w0BAQ= EFAASBgI70ZvlI8FIt0uWXjp2V%0D%0Aquny/hWgZllxYpLo2iqo2DUKaM7/rjy9K/8Wdd3VZI= 5ZPdZHKPJiIPfpQXSeMw2aFe2r25PRDEIQ%0D%0ALntyidKcwMmuLvvHwM/5Fy87An5PwCfhVG= 3ktqo6uz5mzMDQq13WxkuDgxScuOb/GC2UT+V7ZbyZ%0D%0A0AU=3D%0D%0A So the '0D0A' appearing in a binary body would pose no issues so long as = we base64 encode first and subsequently escape the CRLF's. =20 >> I think we may need a larger discussion to see how the MIME = framework specified binary bodies. I am not a MIME expert so I'd = appreciate your guidance here on what you think would be the best = approach to address your concern. Do we need to discuss with MIME folks? > Start with the questions above. Creating a few worked examples would = help find the places we need to be clear. The CRLFs in each multipart body that delimit the entity from the header = will be escaped appropriately. I think this bit of clarification of the = text on order of operation settles the issue but I would need your = confirmation. >>> I don't remember a response to my question about whether log readers = should unescape anything (apologies if I'm just not finding a response = as I'm rereading the threads). >>> I think from the context of the conversations, the intent was that = such a reader would never unescape (since you want to read these with = grep and the like). Someone reading a logged body isn't going to be able = to tell that the logging system re-encoded a body into base64 unless = you're leaving evidence of having done something somewhere else. Are = you? >> The log reader does not unescape anything. > The document should say that. >> Sure, some information may be lost along the lines of something like = converting a binary body to base64 to log it. The reader may not know = that the body was originally binary; however, the human user behind the = log reader can infer more semantics if the need be. The intent of = logging has never been to faithfully recreate the bit-exact SIP message = that was logged in the first place. > The document should say that something using the log can't determine = the original encoding. OK, I'll add something for this. >>=20 >>> 3) This sentence needs adjusting: " It should be noted that as a = result of the escaping mechanisms used in this document ('-' and '?') a = field that would normally be able to parse if it appeared in a SIP = header (as opposed to a log file) may not be syntactically parsable by a = SIP parser." I suggest this replacement: "It should be noted that any = field value that is modified by the escaping mechanisms defined in this = document before logging ('-','?', and CRLF) is likely no longer = well-formed SIP and will fail when given to such a parser". >> I'm perfectly fine with the suggested update. This issue and (2) = above have me thinking that maybe we need to explicitly state in the = draft that the intent of logging has never been to >> faithfully recreate the bit-exact SIP message that was logged in the = first place. Let me know your thoughts. > Yep. That has ramifications on what these logs can get used for. Ditto. >>=20 >>> 4) Section 4.2's description of To and =46rom tags still doesn't say = what to do when the tags aren't present (or am I missing that = discussion)? >> Later in that same section 4.2 we state: >>=20 >> An element will not always have an appropriate value to provide for >> one of these fields, even when the field is required to appear in the >> SIP CLF record. In such circumstances, when a given mandatory field >> is not present then that empty field MUST be encoded as a single >> horizontal dash ("-"). >>=20 >> It was my impression that would be enough. Do you think we also need = to add something in the description of the the To and =46rom Tags? > The turnaround time was long enough that I had to go back to the -05 = review for context. >=20 > I still trip over the text in section 4.2 that you quote - the second = sentence is really fuzzy. > Is the field empty or not present? Even in context, the reader has to = think about whether > you mean "mandatory to log" or "mandatory in SIP" while reading = "mandatory field" in that > sentence. It is confusing. I'll add some clarifying text indicating that "mandatory field" = references the mandatory fields described in Section 4.2 of this = document. > This is compounded by the text in the To tag field discussion going = out of its way to say > "(if present)", but the text in the =46rom tag field being silent = about it. (That's the context > that was missing from _this_ email message). That leads to people = wondering what you > meant by mandatory. Possible misinterpretations include "logging the = To tag isn't mandatory > if it's not present". >=20 > Please consider rewriting what you quote above, and treat To and =46rom = the same when you talk > about their tags. Good point. To and =46rom should be synced as it could lead to = misinterpretation. I'll do that. >>=20 >>=20 >>> 5) (Nit: the change of text from 'Internet-Draft' to 'RFC' = highlighted a problem for me that was already in -05: >>> I suggest this change:) >>> OLD: >>> A Base64 encoded version of this log entry (without the changes >>> required to format it for an RFC) is shown below. >>> NEW: >>> A Base64 encoded version of this log entry (modified as >>> required to format it for an RFC) is shown below. >> I'd typically not have a problem with changing a nit like this. The = new text seems to indicate that the Base64 encoded version is what has = been modified to meet RFC formatting standards when that is just the = opposite of what we are stating. We are stating that the base64 version = is what is unencumbered by the formatting restrictions of an RFC unlike = the non-Base64 version of the example. Do you see my point? Or am I = being daft? Perhaps the following is better: >>=20 >> OLD: >> A Base64 encoded version of this log entry (without the changes >> required to format it for an RFC) is shown below. >> NEW: >> A Base64 encoded version of this log entry (unaffected by the = formatting restrictions imposed by an RFC) is shown below. > I see the disconnect now. > How about this instead: >=20 > A bit-exact version of the actual log entry is provided here, Base64 = encoded. I think this is perfectly fine and will add it. Thanks, Gonzalo >>=20 >> Does that meet the need? >>=20 >> Cheers, >>=20 >> Gonzalo >>=20 >>=20 >>=20 >>=20 >=20 From rjsparks@nostrum.com Thu Jul 12 15:35:53 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7DCD411E80A5 for ; Thu, 12 Jul 2012 15:35:53 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.527 X-Spam-Level: X-Spam-Status: No, score=-102.527 tagged_above=-999 required=5 tests=[AWL=0.073, BAYES_00=-2.599, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oh0g62cqdJ42 for ; Thu, 12 Jul 2012 15:35:52 -0700 (PDT) Received: from nostrum.com (nostrum-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:267::2]) by ietfa.amsl.com (Postfix) with ESMTP id C075711E8098 for ; Thu, 12 Jul 2012 15:35:52 -0700 (PDT) Received: from unexplicable.local (pool-173-71-47-121.dllstx.fios.verizon.net [173.71.47.121]) (authenticated bits=0) by nostrum.com (8.14.3/8.14.3) with ESMTP id q6CMaPqs088754 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=OK); Thu, 12 Jul 2012 17:36:26 -0500 (CDT) (envelope-from rjsparks@nostrum.com) Message-ID: <4FFF5169.6040108@nostrum.com> Date: Thu, 12 Jul 2012 17:36:25 -0500 From: Robert Sparks User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: Gonzalo Salgueiro References: <4F9EE0A4.2000905@nostrum.com> <4FDA2807.6080802@nostrum.com> <066F0992-9BFD-4CC2-81FF-0697529E179F@cisco.com> In-Reply-To: <066F0992-9BFD-4CC2-81FF-0697529E179F@cisco.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Received-SPF: pass (nostrum.com: 173.71.47.121 is authenticated by a trusted mechanism) Cc: "sip-clf@ietf.org Mailing" Subject: Re: [sip-clf] AD review: draft-ietf-sipclf-format-06 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jul 2012 22:35:53 -0000 Trimming to one point and replying inline. (Your proposals for everything else are fine). On 7/12/12 4:40 PM, Gonzalo Salgueiro wrote: > >>>> 2) The description of escaping and encoding in Tag=01 is still ambiguous. You say you must base64 encode any binary body. You also say you must escape CRLFs. I suspect you intend for those to be mutually exclusive? What are you expecting the implementer to use to decide if the body is binary or not? We should be making much more precise use of the terms defined in the media type specifications to make this clear (to avoid things like encoding a body that's already encoded). >>> Our intent is to be clear that CRLFs are to be escaped for ANY body type. Is your question about order of operations in regards to escaping CRLFs and base64 encoding a binary body (something like MIME types of application/ISUP and application/QSIG)? >> application/jpg. Are you going to escape bits of a compressed picture that just happen to contain the CRLF sequence or not? What part of the text makes that clear? > I think I get what you are hinting at but I need to play it back to you for verification. The current text states: > > ===== > ...Note that binary bodies MUST be base64encoded to render them in the SIP CLF log file. > > If an optionally logged SIP message body contains any CRLFs they MUST be escaped by using the URI encoded equivalent value of "%0D%0A". This escaping mechanism applies to all body types. > ===== > > So we don't make any distinction in treatment between the various possible body types. I don't believe that we should. You want to base64 application/jpg, but (generally) not application/sdp (It's possible to have non-ASCII range UTF-8 characters in SDP - would you encode such a body?) > I think what the document may be missing to make this escaping mechanism clear is the order of operation. I believe I need to explicitly state that the translation to base64 must occur before the escaping. This would eliminate any ambiguity about the possibility of ever having the escaped CRLF sequence of %0D%0A. > > To your specific point, if a binary body (like an image) is present then it would have to be base64 encoded first and that base64 character stream could never include the CRLF escape sequence of %0D%0A because '%' is not a valid base64 character. Would this clarification in the text around order of operation address the ambiguity in escaping base64 encoded binary bodies? > > Making it clear that these are performed in that order is good. Including an example in the draft that shows the result of applying that ordering will help. But I don't think we've discussed the part of my original question that had to do with how an implementation decides whether the type it's looking at is "binary" or not. What is the thing to look at that tells you to treat application/jpg different from application/sdp, and that different from an sdp body that contains some thing out of the ascii range? Just looking at the media type isn't going to be enough. Is what you're really trying to say is "base64 encode any body that contains an octet outside "? If so, you need something in the message that tells you that you've done this encoding. RjS From vkg@bell-labs.com Thu Jul 12 15:51:46 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 626DC21F8532 for ; Thu, 12 Jul 2012 15:51:46 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -109.488 X-Spam-Level: X-Spam-Status: No, score=-109.488 tagged_above=-999 required=5 tests=[AWL=1.111, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5n28ZsWMr1gq for ; Thu, 12 Jul 2012 15:51:45 -0700 (PDT) Received: from ihemail1.lucent.com (ihemail1.lucent.com [135.245.0.33]) by ietfa.amsl.com (Postfix) with ESMTP id A3E2721F852B for ; Thu, 12 Jul 2012 15:51:45 -0700 (PDT) Received: from usnavsmail4.ndc.alcatel-lucent.com (usnavsmail4.ndc.alcatel-lucent.com [135.3.39.12]) by ihemail1.lucent.com (8.13.8/IER-o) with ESMTP id q6CMqJ8U003537 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Thu, 12 Jul 2012 17:52:19 -0500 (CDT) Received: from umail.lucent.com (umail-ce2.ndc.lucent.com [135.3.40.63]) by usnavsmail4.ndc.alcatel-lucent.com (8.14.3/8.14.3/GMO) with ESMTP id q6CMqIVl030340 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Thu, 12 Jul 2012 17:52:19 -0500 Received: from shoonya.ih.lucent.com (shoonya.ih.lucent.com [135.185.237.229]) by umail.lucent.com (8.13.8/TPES) with ESMTP id q6CMqIsi003249 for ; Thu, 12 Jul 2012 17:52:18 -0500 (CDT) Message-ID: <4FFF568D.9000808@bell-labs.com> Date: Thu, 12 Jul 2012 17:58:21 -0500 From: "Vijay K. Gurbani" Organization: Bell Laboratories, Alcatel-Lucent User-Agent: Mozilla/5.0 (X11; Linux i686; rv:13.0) Gecko/20120605 Thunderbird/13.0 MIME-Version: 1.0 To: sip-clf@ietf.org References: <4F9EE0A4.2000905@nostrum.com> <4FDA2807.6080802@nostrum.com> <066F0992-9BFD-4CC2-81FF-0697529E179F@cisco.com> <4FFF5169.6040108@nostrum.com> In-Reply-To: <4FFF5169.6040108@nostrum.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.57 on 135.245.2.33 X-Scanned-By: MIMEDefang 2.64 on 135.3.39.12 Subject: Re: [sip-clf] AD review: draft-ietf-sipclf-format-06 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jul 2012 22:51:46 -0000 On 07/12/2012 05:36 PM, Robert Sparks wrote: [...] > But I don't think we've discussed the part of my original question that > had to do with how an implementation decides whether the type it's > looking at is "binary" or not. > What is the thing to look at that tells you to treat application/jpg > different from application/sdp, and that different from an sdp body that > contains some thing out of the ascii range? Just looking at the media > type isn't going to be enough. > > Is what you're really trying to say is "base64 encode any body that > contains an octet outside "? If so, you need > something in the message that tells you that you've done this encoding. Robert: Neither Gonzalo or I are MIME experts, so we are open to suggestions. Are you thinking of gleaning this through the Content-Transfer-Encoding header? But that may not be present in SIP unless the entity consist of multipart subtypes ...? It will be nice to wrap this up and if it is expedient to hammer this out with someone who is better versed in MIME than us, we can do so. Thanks, - vijay -- Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent 1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60563 (USA) Email: vkg@{bell-labs.com,acm.org} / vijay.gurbani@alcatel-lucent.com Web: http://ect.bell-labs.com/who/vkg/ From gsalguei@cisco.com Thu Jul 12 17:09:05 2012 Return-Path: X-Original-To: sip-clf@ietfa.amsl.com Delivered-To: sip-clf@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B374411E80BD for ; Thu, 12 Jul 2012 17:09:05 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.724 X-Spam-Level: X-Spam-Status: No, score=-2.724 tagged_above=-999 required=5 tests=[AWL=-0.125, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rVkUDxsKnhOU for ; Thu, 12 Jul 2012 17:09:04 -0700 (PDT) Received: from av-tac-rtp.cisco.com (av-tac-rtp.cisco.com [64.102.19.209]) by ietfa.amsl.com (Postfix) with ESMTP id AE56A11E80A2 for ; Thu, 12 Jul 2012 17:09:04 -0700 (PDT) X-TACSUNS: Virus Scanned Received: from chook.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-rtp.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q6D09c3t027322 for ; Thu, 12 Jul 2012 20:09:38 -0400 (EDT) Received: from dhcp-10-150-53-182.cisco.com (dhcp-10-150-53-182.cisco.com [10.150.53.182]) by chook.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id q6D09ZmJ001643; Thu, 12 Jul 2012 20:09:38 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=iso-8859-1 From: Gonzalo Salgueiro In-Reply-To: <4FFF5169.6040108@nostrum.com> Date: Thu, 12 Jul 2012 20:09:32 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <61DC09AD-2C4A-4A8A-B6C7-7314C42C1E52@cisco.com> References: <4F9EE0A4.2000905@nostrum.com> <4FDA2807.6080802@nostrum.com> <066F0992-9BFD-4CC2-81FF-0697529E179F@cisco.com> <4FFF5169.6040108@nostrum.com> To: Robert Sparks X-Mailer: Apple Mail (2.1278) Cc: "sip-clf@ietf.org Mailing" Subject: Re: [sip-clf] AD review: draft-ietf-sipclf-format-06 X-BeenThere: sip-clf@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: SIP Common Log File format discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jul 2012 00:09:05 -0000 On Jul 12, 2012, at 6:36 PM, Robert Sparks wrote: > Trimming to one point and replying inline. (Your proposals for = everything else are fine). >=20 > On 7/12/12 4:40 PM, Gonzalo Salgueiro wrote: >>=20 >>>>> 2) The description of escaping and encoding in Tag=3D01 is still = ambiguous. You say you must base64 encode any binary body. You also say = you must escape CRLFs. I suspect you intend for those to be mutually = exclusive? What are you expecting the implementer to use to decide if = the body is binary or not? We should be making much more precise use of = the terms defined in the media type specifications to make this clear = (to avoid things like encoding a body that's already encoded). >>>> Our intent is to be clear that CRLFs are to be escaped for ANY body = type. Is your question about order of operations in regards to escaping = CRLFs and base64 encoding a binary body (something like MIME types of = application/ISUP and application/QSIG)? >>> application/jpg. Are you going to escape bits of a compressed = picture that just happen to contain the CRLF sequence or not? What part = of the text makes that clear? >> I think I get what you are hinting at but I need to play it back to = you for verification. The current text states: >>=20 >> =3D=3D=3D=3D=3D >> ...Note that binary bodies MUST be base64encoded to render them in = the SIP CLF log file. >>=20 >> If an optionally logged SIP message body contains any CRLFs they MUST = be escaped by using the URI encoded equivalent value of "%0D%0A". This = escaping mechanism applies to all body types. >> =3D=3D=3D=3D=3D >>=20 >> So we don't make any distinction in treatment between the various = possible body types. I don't believe that we should. > You want to base64 application/jpg, but (generally) not = application/sdp (It's possible to have non-ASCII range UTF-8 characters = in SDP - would you encode such a body?) No. I would only encode bodies that are decidedly "binary". >=20 >> I think what the document may be missing to make this escaping = mechanism clear is the order of operation. I believe I need to = explicitly state that the translation to base64 must occur before the = escaping. This would eliminate any ambiguity about the possibility of = ever having the escaped CRLF sequence of %0D%0A. >>=20 >> To your specific point, if a binary body (like an image) is present = then it would have to be base64 encoded first and that base64 character = stream could never include the CRLF escape sequence of %0D%0A because = '%' is not a valid base64 character. Would this clarification in the = text around order of operation address the ambiguity in escaping base64 = encoded binary bodies? >>=20 >>=20 > Making it clear that these are performed in that order is good. > Including an example in the draft that shows the result of applying = that ordering will help. OK. I'll try and come up with a basic example that gets the concept = across. Maybe I'll run through the exact one I did in this email thread = and reference the message in S3.1.1.11 in RFC4474. >=20 > But I don't think we've discussed the part of my original question = that had to do with how an implementation decides whether the type it's = looking at is "binary" or not. > What is the thing to look at that tells you to treat application/jpg = different from application/sdp, and that different from an sdp body that = contains some thing out of the ascii range? Just looking at the media = type isn't going to be enough. >=20 > Is what you're really trying to say is "base64 encode any body that = contains an octet outside "? If so, you need = something in the message that tells you that you've done this encoding. I agree that a complete list of "binary" Content-Types gets tricky and = trying to only base64 encode portions of a multipart message becomes = even more unwieldy. I think this is a problem left to the implementer = and not one we need to be tackling. There are a myriad of different ways = to skin this cat. For example, the quick and dirty approach used by gnu = grep, gnu diff is a quick check for the NUL byte to declare whether a = file is "binary" or not: not_text =3D (((binary_files =3D=3D BINARY_BINARY_FILES && !out_quiet) || binary_files =3D=3D WITHOUT_MATCH_BINARY_FILES) && memchr (bufbeg, eol ? '\0' : '\200', buflim - bufbeg)); Here are a variety of different checks using python (with varying levels = of efficacy):=20 = http://stackoverflow.com/questions/898669/how-can-i-detect-if-a-file-is-bi= nary-non-text-in-python/3002505#3002505 The verification of non-UTF-8 characters is something that should fall = to the SIP CLF implementation IMO. Can we take this approach? Thanks, Gonzalo >=20 > RjS >=20 >=20 >=20