Subject: iSCSI: BASE64 and numerical values

From: "Julian Satran" <[email-masked]>
Date: Sat, 18 May 2002 11:19:23 +0300

I was going to suggest this myself :-) Julo
----- Forwarded by Julian Satran/Haifa/IBM on 05/18/2002 11:07 AM -----

Bill Studenmund <[email-masked]>
Sent by: [email-masked]
05/15/2002 05:46 AM
Please respond to Bill Studenmund

        To:     <[email-masked]>
        cc: 
        Subject:        iSCSI: BASE64 and numerical values

I was looking at the latest draft (Julian's 12-90.pdf), and thinking about
this. I see some problems, and I'd like to suggest we DROP support for
using base64 to encode numberical values. Definitely keep it for binary
strings, just not numbers.

The reason I suggest this is base64 encodes arbitrary-length binary
streams. To consider that stream a number, we need to add some more
structure. "Network Byte Order" is the first thing that comes to mind.
We've got ntohs() and ntohl(), also known as ntoh16() and ntoh32() on some
systems, which will save the day.

Unfortunatlely since the base64 stream can be arbitrary length, we'd need
something like ntoh24() too. We could after all have used just three bytes
to encode that number, especially since the spec discourages leading
zero-characters (encoded as 'A's).

Also, since regular-numerical-values can go up to just shy of 2**64, in
principle the only thing saving us from needing ntoh40(), ntoh48(), and
ntoh56() is the fact that I don't think we have any numerical parameters
that have valid values that large.

So I'd like to suggest we disallow using base64 to encode numbers.
Byte-swapping an arbitrary-length byte stream seems really messy.
Alternatively, I'd like to suggest we constrain base64-encoded numbers to
be 1, 2, 4, or 8 bytes long when decoded, and to be in network byte order.

Sorry for not catching this last week when we were looking at section 4.1.

Thoughts?

Take care,

Bill

----- Forwarded by Julian Satran/Haifa/IBM on 05/18/2002 11:07 AM -----

"Michael Krueger" <[email-masked]>
Sent by: [email-masked]
05/15/2002 09:04 PM
Please respond to "Michael Krueger"

        To:     "Bill Studenmund" <[email-masked]>, <[email-masked]>
        cc: 
        Subject:        RE: iSCSI: BASE64 and numerical values

> So I'd like to suggest we disallow using base64 to encode numbers.
> Byte-swapping an arbitrary-length byte stream seems really messy.
> Alternatively, I'd like to suggest we constrain base64-encoded numbers 
to
> be 1, 2, 4, or 8 bytes long when decoded, and to be in network byte 
order.

Bill, thanks for bringing this up, because I wanted to make the same 
point.
Implementing base-64 encoding for numbers is a fair amount of work for 
very
little benefit--no, let me be blunt: for ZERO benefit.  As I understand 
it,
the whole idea of the text-based login was to make it human-readable and 
to
allow cutting and pasting of values from, say, a management console. Maybe
I lack imagination, but I cannot conceive of any real-world circumstance
under which a NUMBER would be displayed on a console or entered by a human
(except, perhaps, by a pathologically mischievous one ;-) in base-64
encoding.  A BINARY ITEM, yes, but a NUMBER, never.

Let's keep it simple: decimal and hexadecimal for numbers, hexadecimal and
base 64 for binary items.

Michael

--
Michael J. Krueger              mailto:[email-masked]
Wind River Networks                         http://www.windriver.com
500 Wind River Way                               phone: 510-749-2130
Alameda, CA  94501                                 fax: 510-749-2010

---

From: "Julian Satran" <[email-masked]>
Date: Tue, 21 May 2002 19:07:47 +0300

And ignore completely the needs of some cryptographic protocols for large 
numerical values? 
I do not see why 6 bit values are more difficult to handle than 4 or 8 bit 
ones.

Julo

Robert Snively <[email-masked]>
05/21/2002 06:13 PM
Please respond to Robert Snively

        To:     "'Bill Studenmund'" <[email-masked]>, Julian 
Satran/Haifa/IBM@IBMIL
        cc:     [email-masked]
        Subject:        RE: iSCSI: BASE64 and numerical values

I have been having continuing problems understanding why
iSCSI is considering any use of base64, and I find myself
in total agreement with Bill's clear exposition of the
situation.

Upon reviewing RFC-2045, it is clear that base64 is a method
for containing handy binary values in a EMAIL structure that does
not use TCP/IP to transmit data, but rather uses obsolete
7-bit ASCII or (slightly less obsolete) EBCDIC strings to 
carry the data.  TCP/IP transmits data neutral octets, 
so binary values can be carried in their native binary 
string format (allowing for a possible requirement to pad
strings to an octet boundary).  The most standard way to
present those binary strings on a printer or as an input
value to a program is hexadecimal encoding.

For those reasons, I agree with Bill that we should delete
base64 from the text and use hexadecimal as he suggests.

Bob

> -----Original Message-----
> From: Bill Studenmund [mailto:[email-masked]]
> Sent: Monday, May 20, 2002 2:54 PM
> To: Julian Satran
> Cc: [email-masked]
> Subject: Re: iSCSI: BASE64 and numerical values
> 
> 
> On Sat, 18 May 2002, Julian Satran wrote:
> 
> > Yes for regular numerical values it makes sense. We will 
> have to keep it
> > for large numerical values used for cryptography.  Julo
> 
> I realize I'm driving a semantic nit, but I don't think we should use
> BASE64 for any "numeric" values, only "binary" ones. I 
> believe we probably
> agree on what we want, I'm just worried that if the wording isn't 100%
> tight, the decode path can become a real mess (since we're 
> supposed to be
> liberal in what we accept :-).
> 
> I'd like to suggest we drop BASE64 from the text that 
> mentions numerical
> values. I really don't look forward to byteswapping a 256 or 512 bit
> number as part of BASE64 decode. :-)
> 
> I'd also like to suggest that for encoding large binary values using
> hexadecimal, we state that the hexadecimal representation is 
> that of the
> number in network byte order. i.e. if I have the five octets 
> 02 45 78 ab
> de, the hex constant is 0x024578abde. That makes the decode logic REAL
> easy; grab two hex characters & convert to one octet. Repeat 
> as needed.
> Please note that this is not the output you'd get with something like
> printf("0x%x", number) on a little-endian machine.
> 
> Thirdly, I'd like to suggest we drop decimal support from 
> binary objects.
> It too has the cumbersome byte-swapping issue that I started 
> this thread
> about, just in reverse. :-)
> 
> I realize that's a lot for one EMail. :-)
> 
> To try and be proactive about this, I'd like to suggest 
> revised text for
> section 4.1 (of course needs word-wrapping). For the definitions:
> 
>      hex-constant: hexadecimal constant encoded as a string start-
>        ing with "0x" or "0X" followed by 1 or more digits or the
>        letters a, b, c, d, e, f, A, B, C, D, E and F. Hex-constants
>        are used to encode numerical values or binary strings. When
>        used to encode numerical values the excessive use of leading
>        0 digits is discouraged. When used to encode binary strings
>        hexadecimal constants must have an even number of digits,
>        and have an implicit byte-length that
>        includes 4 bits for every hexadecimal digit of the constant,
>        including leading zeroes (i.e., a hex-constant of n hexadeci-
>        mal digits has a byte-length of n/2). Additionally, when used
>        to encode binary strings, hexadecimal constants shall be
>        expressed in network byte order (i.e., the octet stream 02
>        45 78 ab de would be expressed as "0x024578abde").
> 
>      decimal-constant: an unsigned decimal number - the digit 0 or a
>        string of 1 or more digits starting with a non-zero digit.
>        This encoding is not used for numerical values equal or
>        greater than 2**64.
> 
>      base64-constant: base64 constant encoded as a string starting
>        with "0b" or "0B" followed by 1 or more digits or letters or
>        plus or slash or equal. The encoding is done according to
>        [RFC2045]. base64-constants are used to encode binary strings.
>        Base64-constants have an implicit byte-length that includes 6
>        bits for every character of the constant excluding trailing
>        equals (i.e., a base64-constant of n base64 characters
>        excluding the trailing equals and m trailing equals has a
>        byte-length of ((the integer part of) ((n+3)*3/4) - m).
> 
>        regular-numerical-value :  an unsigned integer less than 2**64
>          encoded as a decimal-constant, or hex constant.
> 
>        large-numerical-value :  an unsigned integer larger than or
>          equal to 2**64 encoded as a hex constant.
> 
> [no change for numerical-value or numerical-range]
> 
>        binary-value :  a binary string encoded as a hex-con-
>          stant or base64-constant.  The length of the string is either
>          specified by the key definition or is implicit byte-length of
>          the encoded string.
> 
> [delete regualar-binary-value and large-binary-value since 
> the distinction
> is gone]
> 
> I don't see it explicitly mentioned anywhere, but I think all 
> of the CHAP
> (and SRP, but not sure) keys are large numericalal values, and the
> kerberos and SPKM keys are binary values.
> 
> Also, I think the CHAP keys need the field length features of 
> hexadecimal
> binary strings, but they are numbers, not binary strings. Not 
> sure how to
> word that.
> 
> Take care,
> 
> Bill
> 
>

---

From: [email-masked]
Date: Tue, 21 May 2002 10:46:39 -0600

Julian,
 
Don't the large cryptographic strings represent the coefficients of binary polynomials? I think those are binary values rather than numerical ones so it makes sense to restrict the base64 to binary objects as Bill suggests.
 
Regards,
Pat
 
-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Tuesday, May 21, 2002 9:08 AM
To: Robert Snively
Cc: [email-masked]; 'Bill Studenmund'
Subject: RE: iSCSI: BASE64 and numerical values

And ignore completely the needs of some cryptographic protocols for large numerical values?   
I do not see why 6 bit values are more difficult to handle than 4 or 8 bit ones. 

Julo 

05/21/2002 06:13 PM 

        To:        "'Bill Studenmund'" <[email-masked]>, Julian Satran/Haifa/IBM@IBMIL 
        cc:        [email-masked] 
        Subject:        RE: iSCSI: BASE64 and numerical values 

Bob

---

From: "Martin, Nick (Server Storage)" <[email-masked]>
Date: Tue, 21 May 2002 15:11:38 -0500

Robert,

My comments are below.

> From: Robert Snively [mailto:[email-masked]]
> Sent: Tuesday, May 21, 2002 1:44 PM
> To: Bill Studenmund; Julian Satran; [email-masked]
> I guess that my point was that we should be transmitting the
> binary value, not the encoding of the binary value.
>
In the current draft and all those I saw preceding one can not transmit "raw binary" values during login phase.  To be transmitted within a Login, Login Response, Text, or Text Response PDU, the value must be formatted as a null terminated string of the form keyword=value.  Further the set of characters which may be used is restricted as specified.
 
> Especially if you care about the length, that is the most
> compact version of transmission.
> 
Agreed this is the most compact, but not the most universal.  As many working in both big endian and little endian worlds can attest, a number conveyed in "binary" is not universally understood.

> You are never using them in the text-encoded format, so
> why transmit them in a text-encoded format.  The values are
> ALWAYS used only in the binary format.  TCP/IP does not
> have any requirement for them to ever be expressed in the
> text-encoded format.
> 
We need to encode in some text format.  The rules for doing so need to be agreed.
The distinction between binary strings and binary numbers must be clear.  Strings of any sort will be transmitted lowest addressed byte first.  Binary strings would be encoded into text strings lowest addressed byte first.  Text encoding of numbers are always transmitted most significant digit (hex, decimal, or base64).  A conversion between native binary numeric representation and text implies native endian awareness.  When dealing with binary values on the order of say 1024 bits, one must be aware of endian issues to encode, transmit, receive, encode and use the result correctly.  Is this value used as a string, or a number?  If a string, one will want to encode and transmit low addressed byte first.  If binary one will want to encode and transmit the most significant "digit" first (either highest address or lowest address depending on native arithmetic byte order).

> If you need to input them, describe them for debug purposes, or 
> print them to a screen, then put them in the
> format that is most natural for parsing to natural byte and
> word boundaries, which is hexadecimal, but would be outside the
> scope of the iSCSI document anyway.
> 
Decimal is handy for humans, but not indispensable for machine to machine.
I agree with others that hexadecimal is the simplest, and least ambiguous encoding. 
Still one needs to know whether a string or a binary value is being processed unless one is fortunate enough to only be concerned with big-endian (network byte order) hardware.

Prior use of Base64 was designed explicitly for binary files which are effectively binary strings. It will encode, transmit, and decode lowest byte address first.

If the intent is to use base64 to transmit large binary numbers, it could be done, but why bother when hexadecimal is so straight forward?  
The big deal with 4 bits, 6 bits, or 8 bits encoding is the loading, masking, shifting, and storing of the next bits from the correct location at the correct time from 8 bits per byte addressable memory systems.  If the whole value fits in one arithmetic register at one time, no big deal.  However for larger quantities, 4 bits (hexadecimal) and 8 bits (raw binary which is prohibited) are simple, 6 bits (base64) and 3.something bits (decimal) are not.

Thanks,
Nick

> > Sent: Tuesday, May 21, 2002 9:28 AM
> > To: Robert Snively
> > Cc: Julian Satran; [email-masked]
> > On Tue, 21 May 2002, Robert Snively wrote:
> > 
> > I must not have been totally clear. I'm not suggesting we 
> > remove base64,
> > I'm suggesting we limit its use to the cases where (IMHO) it 
> > makes sense.
> > While there will be much fewer cases in the text with the changes I
> > suggest, it will still be in the spec.
> > 
> > Some of the cryptographic authentications throw around large binary
> > strings. Quite large (compared to the numbers and strings we 
> > often use)
> > binary strings. The fact that if you support these modes of 
> > authentication
> > you are supposed to be able to accept four times as much 
> text in login
> > commands reflects the size of these strings.
> > 
> > Base64 is a real win for these. Hex can encode 4 bits per 
> byte, while
> > base64 can encode 6. Thus an n byte string will be about 
> > 1.33*n bytes in
> > base 64, while it will be 2n bytes in hex. That size 
> > difference is why we
> > want base64.
> > 
> > The main point I've been trying to make is that binary 
> > strings and numbers
> > are different things, and that encoding methods for one 
> aren't really
> > appropriate for the other. Specifics: decimal & hex work great for
> > numbers, and hex and base64 work great for binary strings. 
> That's it.
> > Also, the hex for numbers isn't the same as the hex for 
> > binary strings.
> > One is the number in hex, the other is more like hexdump -x 
> > without the
> > spaces and without the offsets.
> > 
> > Take care,
> > 
> > Bill
> > 
> > 
> > 
>

---

From: Robert Snively <[email-masked]>
Date: Tue, 21 May 2002 14:08:48 -0700

Then I think we may have to correct the description in
Clause 4.1 to allow the / character.  In addition,
we probably should specify them as US-ASCII (as RFC 2045 does)
instead of UTF-08 Unicode.  

> In the current draft and all those I saw preceding one can 
> not transmit "raw binary" values during login phase.  To be 
> transmitted within a Login, Login Response, Text, or Text 
> Response PDU, the value must be formatted as a null 
> terminated string of the form keyword=value.  Further the set 
> of characters which may be used is restricted as specified.

---

From: "Julian Satran" <[email-masked]>
Date: Wed, 22 May 2002 03:05:14 +0300

Nick,

I said already that I will say something about order in numbers - and that 
holds for hex or b64
and that will be the "normal"  high-endian encoding.
Why do you consider hex and b64 to be  different?

Julo

"Martin, Nick (Server Storage)" <[email-masked]>
05/21/2002 11:11 PM
Please respond to "Martin, Nick (Server Storage)"

        To:     "Robert Snively" <[email-masked]>, "Bill Studenmund" 
<[email-masked]>, Julian Satran/Haifa/IBM@IBMIL, 
<[email-masked]>
        cc: 
        Subject:        RE: iSCSI: BASE64 and numerical values

Robert,

My comments are below.

> binary value, not the encoding of the binary value.
In the current draft and all those I saw preceding one can not transmit 
"raw binary" values during login phase.  To be transmitted within a Login, 
Login Response, Text, or Text Response PDU, the value must be formatted as 
a null terminated string of the form keyword=value.  Further the set of 
characters which may be used is restricted as specified.
 
> compact version of transmission.
Agreed this is the most compact, but not the most universal.  As many 
working in both big endian and little endian worlds can attest, a number 
conveyed in "binary" is not universally understood.

> have any requirement for them to ever be expressed in the
We need to encode in some text format.  The rules for doing so need to be 
agreed.
The distinction between binary strings and binary numbers must be clear. 
Strings of any sort will be transmitted lowest addressed byte first. 
Binary strings would be encoded into text strings lowest addressed byte 
first.  Text encoding of numbers are always transmitted most significant 
digit (hex, decimal, or base64).  A conversion between native binary 
numeric representation and text implies native endian awareness.  When 
dealing with binary values on the order of say 1024 bits, one must be 
aware of endian issues to encode, transmit, receive, encode and use the 
result correctly.  Is this value used as a string, or a number?  If a 
string, one will want to encode and transmit low addressed byte first.  If 
binary one will want to encode and transmit the most significant "digit" 
first (either highest address or lowest address depending on native 
arithmetic byte order).

Decimal is handy for humans, but not indispensable for machine to machine.
I agree with others that hexadecimal is the simplest, and least ambiguous 
encoding. 
Still one needs to know whether a string or a binary value is being 
processed unless one is fortunate enough to only be concerned with 
big-endian (network byte order) hardware.

Prior use of Base64 was designed explicitly for binary files which are 
effectively binary strings. It will encode, transmit, and decode lowest 
byte address first.

If the intent is to use base64 to transmit large binary numbers, it could 
be done, but why bother when hexadecimal is so straight forward? 
The big deal with 4 bits, 6 bits, or 8 bits encoding is the loading, 
masking, shifting, and storing of the next bits from the correct location 
at the correct time from 8 bits per byte addressable memory systems.  If 
the whole value fits in one arithmetic register at one time, no big deal. 
However for larger quantities, 4 bits (hexadecimal) and 8 bits (raw binary 
which is prohibited) are simple, 6 bits (base64) and 3.something bits 
(decimal) are not.

Thanks,
Nick