Subject: iSCSI CRC: A CRC-checking example

From: Mark Bakke <[email-masked]>
Date: Fri, 17 Aug 2001 15:13:05 -0500

Julian and CRC enthusiasts-

After several people ran the iSCSI CRC through their
hardware and software implementations, we've found that
there are a few important things to add to the CRC spec;
that the bit order must be flipped, and the CRC bytes
must be sent in network order.

We should also specify that when running the CRC check
on an iSCSI PDU with its CRC, that the remainder (before
complement and bit-reversal) is a constant 0x1c2d19ed.

How about replacing:

The generator polynomial selected is evaluated in [Castagnioli93].
When using the CRC the CRC register must be initialized to all 1s
(0xFFFFFFFF) and the CRC bits must be complemented before
transmission. Padding bytes, when present, in a segment covered by a
CRC, should be set to 0 and are included in the CRC.

With:

The generator polynomial selected is evaluated in [Castagnioli93].
When using the CRC the CRC register must be initialized to all 1s
(0xFFFFFFFF).  Input bytes are processed with bit 7 being the most
significant bit.  Before transmission, the CRC bits must be
reflected (bit 31 swapped with bit 0, bit 30 swapped with bit 1,
etc.), and complemented.  The CRC bytes must be transmitted in
network order.  Padding bytes, when present, in a segment covered
by a CRC, should be set to 0 and are included in the CRC.

Running the CRC-32C generator on an input stream that includes a
valid CRC-32C will result in the constant remainder 0x1c2d19ed,
(without being reversed and complemented).

-- 
Mark A. Bakke
Cisco Systems
[email-masked]
763.398.1054

Subject: RE: iSCSI: [Fwd: Crc-32c example in iSCSI spec]
From: "Wheat, Stephen R" <[email-masked]>
To: "'Mark Bakke'" <[email-masked]>
Date: Fri, 17 Aug 2001 11:31:10 -0700

Mark,

OK, we're cooking with fire now.  Yup, the reflected-complemented result
matches yours.

Thanks,
Stephen

-----Original Message-----
From: Mark Bakke [mailto:[email-masked]]
Sent: Friday, August 17, 2001 10:53 AM
To: Wheat, Stephen R
Subject: Re: iSCSI: [Fwd: Crc-32c example in iSCSI spec]

Stephen-

The remainder was from running an entire test case through
the CRC generator, including the CRC itself.  I then had to
complement and bit-flip the remainder to make it work out the
same as the documented Ethernet remainder.  So here's how
I get the remainder for the 32-bytes of ones example:

Run

       0:        ff ff ff ff
       4:        ff ff ff ff
       8:        ff ff ff ff
      12:        ff ff ff ff
      16:        ff ff ff ff
      20:        ff ff ff ff
      24:        ff ff ff ff
      28:        ff ff ff ff

      32:        43 ab a8 62

Through the generator to get the CRC of the whole thing,
including the CRC that was sent.

In all cases, if the CRC is correct for the data, the CRC
will be 0x48674bc7.  Since the CRC that came out of my
generator was not really the remainder, it was the bit-flipped,
complemented remainder, I have to reverse both of these to
get 0x1c2d19ed.

--
Mark

"Wheat, Stephen R" wrote:
> 
> OK, until now, I was matching everything you posted (after the
correction).
> Now, I cannot get the same result as you on the remainder, either with a
> simple engine, or a table driven.
> 
> So, what exactly are you passing through the engine for this latest
> confirmation calculation?
> 
> I have been assuming you've been passing a 4-byte stream to the
crc-engine,
> where the 4-byte stream consists of the generation polynomial, byte order
> 0x41 0x6f 0xdc 0x1e
> 
> I'm not looking for you to help debug my engine; just to confirm the input
> to it.
> 
> Thanks,
> Stephen
> 
> -----Original Message-----
> From: Mark Bakke [mailto:[email-masked]]
> Sent: Friday, August 17, 2001 10:06 AM
> To: Sanjay Goyal
> Cc: Ips (E-mail)
> Subject: Re: iSCSI: [Fwd: Crc-32c example in iSCSI spec]
> 
> Sanjay Goyal wrote:
> >
> > Hi
> >  I get the remainder for iSCSI as 0x1c2d19ed if I complement my CRC and
> then
> > pass it through the CRC engine.
> >
> >  As per CRC generation for data: the thing which is not clear to me is
> >    why do we need to bit-swap the CRC reminader as is done in all your
> > examples.
> 
> As far as I can tell, bit-swapping does nothing to help or hurt
> the effectiveness of the CRC.  When I ran my examples, I thought
> that it would be the simplest for our CRC to different from the
> Ethernet CRC only in the polynomial.  Ethernet, FDDI, FC, and SCSI
> all do this, so I figured it would cause the least confusion if
> we did the same.
> 
> > We can just complement it and append it after the DATA bytes.
> >  The other side also can just pass it through the CRC engine to check
it.
> >
> > Regards
> > Sanjay Goyal
> >
> > -----Original Message-----
> > From: Mark Bakke [mailto:[email-masked]]
> > Sent: Thursday, August 16, 2001 1:43 PM
> > To: Steve Blightman; [email-masked]
> > Subject: Re: [Fwd: Crc-32c example in iSCSI spec]
> >
> > One more thing that might be helpful.  When the Ethernet
> > polynomial is used in SCSI to generate its CRC, the T10
> > doc specifies the remainder polynomial that one should see
> > after running the data with a valid CRC through.  For
> > the Ethernet CRC, this was specified as 0xc704dd7b.  This
> > remainder polynomial is taken before the CRC is complemented
> > and bit-reflected.  For iSCSI, I came up with a remainder of
> > 0x1c2d19ed.  Can anyone verify this result?
> >
> > --
> > Mark
> >
> > Mark Bakke wrote:
> > >
> > > Steve-
> > >
> > > I just ran some Ethernet packets with known CRCs through
> > > my iSCSI/Ethernet CRC generator, and found the same thing
> > > as you did.
> > >
> > > All of my examples need to be byte-swapped, along with the
> > > fix I already posted for the all-ones example.  Here is a
> > > new set of examples, which will be in -08.  I also ran
> > > 64 bytes of zeroes and ones, which now agree with your
> > > numbers as well.
> > >
> > > Thanks again for bringing this up.
> > >
> > > --
> > > Mark
> > >
> > >      07 CRC Examples
> > >
> > >         N.B. all Values are Hexadecimal
> > >
> > >           Byte:        0  1  2  3
> > >
> > >              0:       01 a0 00 00
> > >              4:       00 00 00 00
> > >              8:       00 00 00 00
> > >             12:       00 00 00 00
> > >             16:       04 05 00 00
> > >             20:       00 01 00 00
> > >             24:       00 00 00 05
> > >             28:       00 00 00 04
> > >             32:       2a 00 00 00
> > >             36:       00 00 00 00
> > >             40:       80 00 00 00
> > >             44:       00 00 00 00
> > >
> > >            CRC:       93 70 51 db
> > >
> > >         32 bytes of zeroes:
> > >
> > >           Byte:        0  1  2  3
> > >
> > >              0:       00 00 00 00
> > >            ...
> > >             28:       00 00 00 00
> > >
> > >            CRC:       aa 36 91 8a
> > >
> > >         32 bytes of ones:
> > >
> > >           Byte:        0  1  2  3
> > >
> > >              0:       ff ff ff ff
> > >            ...
> > >             28:       ff ff ff ff
> > >
> > >            CRC:       43 ab a8 62
> > >
> > >         32 bytes of incrementing 00..1f:
> > >
> > >           Byte:        0  1  2  3
> > >
> > >              0:       00 01 02 03
> > >            ...
> > >             28:       1c 1d 1e 1f
> > >
> > >            CRC:       4e 79 dd 46
> > >
> > >
> > > Steve Blightman wrote:
> > > >
> > > > I believe the examples for the ISCSI CRC  have the wrong endianness.
> > > >
> > > > As yiou suugested over the phone I ran some Ethernet frames through
a
> > > > simulation. I have some difficulty running the exact simulations you
> > > > wanted becuase the minimum size Ethernet frame is 64 bytes.
> > > >
> > > > However using the Ethernet CRC polynomial,
> > > > Running 64 bytes of 0 onto the wire, we append "36 63 8d 75" onto
the
> > > > wire for the CRC - "36" goes out first.
> > > > Running 64 bytes of all 1, we append "ba 87 61 0f" onto the wire -
> "ba"
> > > > goes out first.
> > > >
> > > > Using the same logic for the ISCSI polynomial
> > > > Running 64 bytes of 0 I think we should append "67 eb c8 03" - "67"
> > > > going out first
> > > > and running 64 bytes of all 1 we should append "66 4e cd 2f" - "66"
> > > > going out first
> > > >
> > > > And now for 32 bytes with the ISCSI polynomial
> > > >
> > > > Running 32 bytes of 0 we should append "aa 36 91 8a" - "aa" going
out
> > > > first
> > > > Running 32 bytes of all 1 we should append "43 ab a8 62" - "43"
going
> > > > out first
> > > >
> > > > I don't want to get into an endless endian debate, but I believe it
is
> > > > important to get the order of these bytes in the right order, so
that
> we
> > > > can use the same hardware to check as well as to generate CRCs.
> > > >
> > > > Thanks for your help on this,
> > > > Steve Blightman
> > >
> > > --
> > > Mark A. Bakke
> > > Cisco Systems
> > > [email-masked]
> > > 763.398.1054
> >
> > --
> > Mark A. Bakke
> > Cisco Systems
> > [email-masked]
> > 763.398.1054
> >
> >
>
----------------------------------------------------------------------------
> ------------------------
> >
> >    Part 1.2    Type: application/ms-tnef
> >            Encoding: base64
> 
> --
> Mark A. Bakke
> Cisco Systems
> [email-masked]
> 763.398.1054

-- 
Mark A. Bakke
Cisco Systems
[email-masked]
763.398.1054

---

From: "Douglas Otis" <[email-masked]>
Date: Fri, 17 Aug 2001 15:26:51 -0700

Mark,

You appear to be incorrect about which bit is processed first.  The table is
reflected and therefore there should be no need to do any reflection as you
describe except within building the table.  Here is some example code for
generating this reflected table.

/* Example CRC32 reflected table creation */

#include <stdio.h>
#include <stdlib.h>

#define OUTPUT_FILE   "crc32_tab.h"
#define CRC32_POLY			0x1EDC6F41L
#define CRC_TYPE  "\
/* Castagnoli93                                                        */\n\
/* x^32+x^28+x^27+x^26+x^25+x^23+x^22+x^20+x^19+x^18+x^14+x^13+x^11+   */\n\
/* x^10+* x^9+x^8+x^6+x^0                                              */\n\
/* Guy Castagnoli Stefan Braeuer and Martin Herrman                    */\n\
/* \"Optimization of Cyclic Redundancy-Check Codes
*/\n\
/* with 24 and 32 Parity Bits\",
*/\n\
/* IEEE Transactions on Communications, Vol. 41, No. 6, June 1993      */\n"

FILE *tf;

unsigned long
  reflect_32 (unsigned long b)
{
  int i;
  unsigned long rw = 0;

  for (i = 0; i < 32; i++)
    {
      if (b & 1)
        rw |= 1 << (31 - i);
      b >>= 1;
    }
  return (rw);
}

unsigned long
  build_crc_table (int index)
{
  int i;
  unsigned long rb;

  rb = reflect_32 (index);

  for (i = 0; i < 8; i++)
    {
      if (rb & 0x80000000L)
	rb = (rb << 1) ^ CRC32_POLY;
      else
	rb <<= 1;
    }
  return (reflect_32 (rb));
}

/* here is a means to verify the bype order via serial equivalent */

unsigned long
  serial_gen (unsigned char buf[], int length)
{
  int i;
  int j = 0;
  unsigned char d;
  unsigned long crc = 0;

  while (j < length)
    {
      d = buf[j++];
      for (i = 0; i < 8; i++)
	  {
	  if ((crc >> 31) ^ (d & 1))
	      crc = (crc << 1) ^ CRC32_POLY;
	  else
	      crc <<= 1;
	  d >>= 1;
	  }
    }

  return (crc);
}

main ()
{
  int i;

  printf ("\nGenerating CRC32 Table file <%s>\n", OUTPUT_FILE);
  if ((tf = fopen (OUTPUT_FILE, "w")) == NULL)
    {
      printf ("Unable to open %s\n", OUTPUT_FILE);
      exit (1);
    }
  fprintf (tf, "#ifndef __crc32_table_h__\n");
  fprintf (tf, "#define __crc32_table_h__\n\n");
  fprintf (tf, "#define CRC32_POLY 0x%08lX\n", CRC32_POLY);
  fprintf (tf, "#define CRC32(c,d) (c=(c>>8)^crc32tab[(c^(d))&0xFF])\n");
  fprintf (tf, "\
/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */\n\
/* 32 Bit Reflected CRC table generation for SCTP.                     */\n\
/* To accomodate serial byte data being shifted out least              */\n\
/* significant bit first, the table's 32 bit words are reflected       */\n\
/* which flips both byte and bit MS and LS positions.  The CRC         */\n\
/* is calculated MS bits first from the perspective of the serial      */\n\
/* stream.  The x^32 term is implied and the x^0 term may also         */\n\
/* be shown as +1.  The polynomial code used is 0x%08lX.            */\n%s\
/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */\n"
	   , CRC32_POLY, CRC_TYPE);

  fprintf (tf, "\nunsigned long  crc32tab[256] =\n{\n");
  for (i = 0; i < 256; i++)
    {
      fprintf (tf, "0x%08lXL, ", build_crc_table (i));
      if ((i & 3) == 3)
        fprintf (tf, "\n");
    }

  fprintf (tf, "};\n\n#endif\n");

  if (fclose (tf) != 0)
    printf ("Unable to close <%s>." OUTPUT_FILE);
  else
    printf ("\nThe CRC32 table has been written to <%s>.\n", OUTPUT_FILE);
}

Doug

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Mark Bakke
> Sent: Friday, August 17, 2001 1:13 PM
> To: Julian Satran; IPS
> Subject: iSCSI CRC: A CRC-checking example
>
>
>