Subject: [Ips] Calculating the VA in iSER header

From: Erez Zilber <[email-masked]>
Date: Tue, 15 Apr 2008 16:40:12 +0300

We're trying to understand what should be the write VA (tagged offset)
in the iSER header for WRITE commands. If unsolicited data is to be
sent, should the VA be the original VA or should it be original VA +
FirstBurstLength?

Example:

InitialR2T=No

FirstBurstLength = 1000

Base address of the registered buffer = 0

Now, what should be the VA in the iSER header? 0 or 1000?

We read the following paragraph in the iSER spec, but didn't get an
answer from there:

* If there is solicited data to be transferred for the SCSI write or
bidirectional command, as indicated by the Expected Data Transfer
Length in the SCSI Command PDU exceeding the value of
UnsolicitedDataSize, the iSER layer at the initiator MUST do the
following:

a. It MUST allocate a Write STag for the I/O Buffer defined by
the qualifier DataDescriptorOut. The DataDescriptorOut
describes the I/O buffer starting with the immediate
unsolicited data (if any), followed by the non-immediate
unsolicited data (if any) and solicited data. This means
that the BufferOffset for the SCSI Data-out for this
command is equal to the TO. This implies that a zero TO
for this STag points to the beginning of this I/O Buffer.

Thanks,

-- 

____________________________________________________________

Erez Zilber | 972-9-971-7689

Software Engineer, Storage Solutions

Voltaire – _The Grid Backbone_

__

www.voltaire.com <http://www.voltaire.com/>

_______________________________________________
Ips mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ips

---

From: Mike Ko <[email-masked]>
Date: Tue, 15 Apr 2008 10:18:43 -0700

VA is a concept introduced in an Infiniband annex to support iSER.  It 
appears in the expanded iSER header for Infiniband use only to support the 
non-Zero Based Virtual Address (non-ZBVA) used in Infiniband vs the ZBVA 
used in IETF.

"The DataDescriptorOut describes the I/O buffer starting with the 
immediate unsolicited data (if any), followed by the non-immediate 
unsolicited data (if any) and solicited data."  If non-ZBVA mode is used, 
then VA points to the beginning of this buffer.  So in your example, the 
VA field in the expanded iSER header will be zero.  Note that for IETF, 
ZBVA is assumed and there is no provision to specify a different VA in the 
iSER header.

Tagged offset (TO) refers to the offset within a tagged buffer in RDMA 
Write and RDMA Read Request Messages.  When sending non-immediate 
unsolicited data, Send Message types are used and the TO field is not 
present.  Instead, the buffer offset is appropriately represented by the 
Buffer Offset field in the SCSI Data-Out PDU.  Note that Tagged Offset is 
not the same as write VA and it does not appear in the iSER header.

Mike

Erez Zilber <[email-masked]> 
Sent by: [email-masked]
04/15/2008 06:40 AM

To
[email-masked]
cc

Subject
[Ips] Calculating the VA in iSER header

Example:

InitialR2T=No

FirstBurstLength = 1000

Thanks,

-- 

Voltaire ? _The Grid Backbone_

__

---

From: HUJason <[email-masked]>
Date: Wed, 16 Apr 2008 09:58:32 +0800

Dears,

I am curently implement iscsi target on a resource-limted device. In my implementation, orignal to-be-rejected PDU is not available when it realised that a Reject should be sent to initiator.
 
But as RFC3720 said, Target must send back the orignal PDU as the Reject's payload. 
 
So my question is: Why we need to send back the orignal PDU ? Can we simply eliminate it?
 
Thanks!
Best Regards,
Jason 
_________________________________________________________________
新年换新颜，快来妆扮自己的MSN给心仪的TA一个惊喜！
http://im.live.cn/emoticons/?ID=18

---

From: [email-masked]
Date: Tue, 15 Apr 2008 23:04:15 -0400

The original PDU header has to be sent back so that the initiator
knows what was rejected.  iSCSI has a number of mechanisms that
can be used by the target to control resource consumption and
those should be used to ensure that adequate resources are
available to deal with the PDU that is to be rejected.
 
Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

________________________________

	From: [email-masked] [mailto:[email-masked]] On Behalf Of HUJason
	Sent: Tuesday, April 15, 2008 9:59 PM
	To: [email-masked]
	Subject: [Ips] Send Reject PDU when orignal to-be-rejected PDU is notavailable

	Dears,

		 
		 
		Thanks!

	Best Regards,

	Jason 

________________________________

	“七件武器，七种完美” 立刻体验！ <http://get.live.cn>

---

From: HUJason <[email-masked]>
Date: Wed, 16 Apr 2008 11:45:43 +0800

Hi David,
Thank you for the suggestion. A little more to consider is that it seems that whatever mechanisms we use, as a target, we must reserve all PDU header comes from Initiator until the coressponding statSN is explicitly acked. For a resource-limted iscsi target engine(all SCSI data is on host, so it is not in this question), It can be considered as a big memory consumer and becomes a bottlenack. Best Regards,
Jason

From: [email-masked]: RE: [Ips] Send Reject PDU when orignal to-be-rejected PDU is notavailableDate: Tue, 15 Apr 2008 23:04:15 -0400To: [email-masked]; [email-masked]

Thanks,
--David

----------------------------------------------------David L. Black, Distinguished EngineerEMC Corporation, 176 South St., Hopkinton, MA  01748+1 (508) 293-7953             FAX: +1 (508) [email-masked]        Mobile: +1 (978) 394-7754----------------------------------------------------

From: [email-masked] [mailto:[email-masked]] On Behalf Of HUJasonSent: Tuesday, April 15, 2008 9:59 PMTo: [email-masked]: [Ips] Send Reject PDU when orignal to-be-rejected PDU is notavailable
Dears,
I am curently implement iscsi target on a resource-limted device. In my implementation, orignal to-be-rejected PDU is not available when it realised that a Reject should be sent to initiator. But as RFC3720 said, Target must send back the orignal PDU as the Reject's payload.  So my question is: Why we need to send back the orignal PDU ? Can we simply eliminate it? Thanks!
Best Regards,
Jason 

“七件武器，七种完美” 立刻体验！ 
_________________________________________________________________
用手机MSN聊天写邮件看空间，无限沟通，分享精彩！
http://mobile.msn.com.cn/

---

From: [email-masked]
Date: Wed, 16 Apr 2008 00:30:10 -0400

Jason,
 
The CmdSN window allows the target to control the resources consumed
by commands.  I don't understand the need to save "all PDU header
comes from the Initiator until corresponding statSN is explicitly
ACKed".  What purpose is that accomplishing (e.g., why save all
the iSCSI Data-Out PDU headers for a SCSI WRITE)?
 
In any case, the original PDU header is required for a Reject in order
for the initiator to figure out what was rejected.  If the initiator
cannot figure that out, it is going to have a very difficult time
recovering from the Reject.
 
Thanks,
--David

	From: [email-masked] [mailto:[email-masked]] On Behalf Of HUJason
	Sent: Tuesday, April 15, 2008 11:46 PM
	To: Black, David; [email-masked]
	Subject: Re: [Ips] Send Reject PDU when orignal to-be-rejected PDU is notavailable

	Hi David,
	
	Thank you for the suggestion. A little more to consider is that it seems that whatever mechanisms we use, as a target, we must reserve all PDU header comes from Initiator until the coressponding statSN is explicitly acked. For a resource-limted iscsi target engine(all SCSI data is on host, so it is not in this question), It can be considered as a big memory consumer and becomes a bottlenack. 

	Best Regards,
	Jason

		From: [email-masked]
		Subject: RE: [Ips] Send Reject PDU when orignal to-be-rejected PDU is notavailable
		Date: Tue, 15 Apr 2008 23:04:15 -0400
		To: [email-masked]; [email-masked]

		Thanks,
		--David

			Dears,

				 
				 
				Thanks!

			Best Regards,

			Jason 

			“七件武器，七种完美” 立刻体验！ <http://get.live.cn/>  

	Windows Live Writer，支持离线撰写博客内容，随时随地想写就写。 立即使用！ <http://get.live.cn/product/writer.html>

---

From: HUJason <[email-masked]>
Date: Wed, 16 Apr 2008 14:12:51 +0800

David,
 
In some situations, for example, one or more DATA-OUT PDUs of a WRITE are CRC ERROR. If the implementation(just like mine) choose to use Pipeline fashion process in which TX and RX are seperated as two stages, then it seems we have no choice but to hold those CRC ERROR DATA-OUT PDU headers until the REJECTs sent out.  
 
As we do not know how many DATA-OUT PDU will become CRC ERROR in RX stage, so we choose to reserve all DATA-OUT PDU headers in the end. Best Regards,
Jason HU

From: [email-masked]: RE: [Ips] Send Reject PDU when orignal to-be-rejected PDU is notavailableDate: Wed, 16 Apr 2008 00:30:10 -0400To: [email-masked]; [email-masked]

Jason,

 
Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of HUJasonSent: Tuesday, April 15, 2008 11:46 PMTo: Black, David; [email-masked]: Re: [Ips] Send Reject PDU when orignal to-be-rejected PDU is notavailable 
Hi David,Thank you for the suggestion. A little more to consider is that it seems that whatever mechanisms we use, as a target, we must reserve all PDU header comes from Initiator until the coressponding statSN is explicitly acked. For a resource-limted iscsi target engine(all SCSI data is on host, so it is not in this question), It can be considered as a big memory consumer and becomes a bottlenack. Best Regards,Jason

 
Thanks,
--David

“七件武器，七种完美” 立刻体验！ 

Windows Live Writer，支持离线撰写博客内容，随时随地想写就写。 立即使用！ 
_________________________________________________________________
用手机MSN聊天写邮件看空间，无限沟通，分享精彩！
http://mobile.msn.com.cn/

---

From: "Eli Dorfman" <[email-masked]>
Date: Wed, 16 Apr 2008 11:22:04 +0300

According to Mike's explanation below it seems that we have a bug in
iSER initiator.
Fixing this bug will require a fix in the stgt iSER code.

The problem is that the initiator send a VA which already includes an
offset for the unsolicited data (which is wrong).
In iser_initiator.c::iser_prepare_write_cmd the code looks like this:
hdr->write_va   = cpu_to_be64(regd_buf->reg.va + unsol_sz);

we think that it should be modified to:
hdr->write_va   = cpu_to_be64(regd_buf->reg.va);

Let's discuss this and verify we interpret the spec correctly.
If agreed we will send a patch.

Eli

2008/4/15 Mike Ko <[email-masked]>:
> "The DataDescriptorOut describes the I/O buffer starting with the immediate
> unsolicited data (if any), followed by the non-immediate unsolicited data
> (if any) and solicited data."  If non-ZBVA mode is used, then VA points to
> the beginning of this buffer.  So in your example, the VA field in the
> expanded iSER header will be zero.  Note that for IETF, ZBVA is assumed and
> there is no provision to specify a different VA in the iSER header.
>
> Tagged offset (TO) refers to the offset within a tagged buffer in RDMA Write
> and RDMA Read Request Messages.  When sending non-immediate unsolicited
> data, Send Message types are used and the TO field is not present.  Instead,
> the buffer offset is appropriately represented by the Buffer Offset field in
> the SCSI Data-Out PDU.  Note that Tagged Offset is not the same as write VA
> and it does not appear in the iSER header.
>
> Mike
>
>
>
> To [email-masked]
> Subject [Ips] Calculating the VA in iSER header
>
>
>
>
>
>

---

From: [email-masked]
Date: Wed, 16 Apr 2008 10:24:49 -0400

But that only requires holding a Data-Out PDU header while the
corresponding data is in the RX pipe - once the data digest CRC
has been checked, the Data-Out PDU headers can be discarded -
they don't have to be held until the status is sent and acked.
 
Reject is primarily intended for errors that can be detected
on receipt of the PDU.
 
Thanks,
--David

	From: HUJason [mailto:[email-masked]] 
	Sent: Wednesday, April 16, 2008 2:13 AM
	To: Black, David; [email-masked]
	Subject: RE: [Ips] Send Reject PDU when orignal to-be-rejected PDU is notavailable

	David,

	As we do not know how many DATA-OUT PDU will become CRC ERROR in RX stage, so we choose to reserve all DATA-OUT PDU headers in the end. 

	Best Regards,
	Jason HU

		From: [email-masked]
		Subject: RE: [Ips] Send Reject PDU when orignal to-be-rejected PDU is notavailable
		Date: Wed, 16 Apr 2008 00:30:10 -0400
		To: [email-masked]; [email-masked]

		Jason,

		 
		Thanks,
		--David

			Hi David,

			Best Regards,
			Jason

				 
				Thanks,
				--David

					Dears,

					 
					 
					Thanks!

					Best Regards,

					Jason

---

From: Pete Wyckoff <[email-masked]>
Date: Wed, 16 Apr 2008 10:48:30 -0400

> If agreed we will send a patch.

Agree with the interpretation of the spec, and it's probably a bit
clearer that way too.  But we have working initiators and targets
that do it the "wrong" way.

The transition involved in fixing both sides will lead to problems.
How does a target detect an unfixed initiator and vice versa?  A
mismatched pair will lead to data corruption.

We could address this in a few ways:

1. Flag day: all initiators and targets change at the same time.
Will see data corruption if someone unluckily runs one or the other
using old non-fixed code.

2. Rewrite the IB Annex to codify what's done in practice, and don't
"fix" any code.

3. Start using the Hello messages and extend them to specify if the
VA marks the start of the buffer or the unsol offset.

I really don't look forward to the bug reports we'll get from a
flag da approach.  Old linux versions tend to hang around for a
very long time, and people are often reluctant to upgrade.

		-- Pete

---

From: HUJason <[email-masked]>
Date: Thu, 17 Apr 2008 09:10:36 +0800

David,
 
Oh, sorry I am wrong on that point, PDU headers don't have to be held until the status is sent and acked. I will make it correctly.
 
Thanks,Jason

From: [email-masked]: RE: [Ips] Send Reject PDU when orignal to-be-rejected PDU is notavailableDate: Wed, 16 Apr 2008 10:24:49 -0400To: [email-masked]; [email-masked]

 
Thanks,
--David

From: HUJason [mailto:[email-masked]] Sent: Wednesday, April 16, 2008 2:13 AMTo: Black, David; [email-masked]: RE: [Ips] Send Reject PDU when orignal to-be-rejected PDU is notavailable
David, In some situations, for example, one or more DATA-OUT PDUs of a WRITE are CRC ERROR. If the implementation(just like mine) choose to use Pipeline fashion process in which TX and RX are seperated as two stages, then it seems we have no choice but to hold those CRC ERROR DATA-OUT PDU headers until the REJECTs sent out.   As we do not know how many DATA-OUT PDU will become CRC ERROR in RX stage, so we choose to reserve all DATA-OUT PDU headers in the end. Best Regards,Jason HU

Jason,

 
Thanks,
--David

 
Thanks,
--David

“七件武器，七种完美” 立刻体验！ 

MSN 中文网，最新时尚生活资讯，白领聚集门户。
http://cn.msn.com

---

From: HUJason <[email-masked]>
Date: Thu, 17 Apr 2008 16:22:08 +0800

Dears,
 
If target works on ERL == 0 and meets a PROTOCOL ERROR condition that it can not figure out PDU length any more(no FIM...), what should the target do for session recover? Can it simply drop the TCP connection?

Many Many Thanks!
Jason

---

From: "Eli Dorfman" <[email-masked]>
Date: Thu, 17 Apr 2008 14:13:01 +0300

On Wed, Apr 16, 2008 at 6:46 PM, Roland Dreier <[email-masked]> wrote:
>  Yes... I guess the key question is whether there are any initiators that
>  do things the "right" way.
>
>
>  Seems unacceptable to me... it doesn't make sense at all to break every
>  setup in the world just to be "right" according to the spec.

This will break only when both initiator and target will use
InitialR2T=No, which means allow unsolicited data.
As far as I know, STGT is not very common (and its version in RHEL5.1
is considered experimental). Its default is also InitialR2T=Yes.
Voltaire's iSCSI over iSER target also uses default InitialR2T=Yes.
So it seems that nothing will break.

>  If existing practice is universally to do things "wrong" then this seems
>  to me by far the best way to proceed.

Assuming there aren't many iSER installation that currently work with
unsolicited data, then it is the right time to do it right.
Future implementation will rely on the spec and unless you modify the
spec this will lead to greater confusion.
_______________________________________________
Ips mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ips

---

From: Roland Dreier <[email-masked]>
Date: Wed, 16 Apr 2008 08:46:25 -0700

> VA marks the start of the buffer or the unsol offset.

this seems like a pain for not much benefit... every initiator and
target needs new code to handle the negotiation, and you don't get
anything except the satisfaction of following the letter of the spec.
_______________________________________________
Ips mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ips

---

From: [email-masked]
Date: Thu, 17 Apr 2008 13:42:15 -0400

Yes, as any data received subsequently on that connection will
have to be discarded, but see Section 6.4 of RFC 3720 for
applicable timeouts.
 
Thanks,
--David

	From: [email-masked] [mailto:[email-masked]] On Behalf Of HUJason
	Sent: Thursday, April 17, 2008 4:22 AM
	To: [email-masked]
	Subject: [Ips] How to deal with PROTOCOL ERROR on iscsi target side

	Dears,

	Many Many Thanks!
	Jason

---

From: Ken Sandars <[email-masked]>
Date: Wed, 30 Apr 2008 17:43:19 +1000

Hi all,

It appears the current Linux iSER initiator does not send the HELLO message when the connection transits to full feature phase. The stgt target also ignores this message (if it were to appear). Both of these implementations use a non-conformant iSER header (they add write_va and read_va fields, which incidentally do not appear to be used). Are these changes documented anywhere in the IB domain, or are these variations needed for another reason?

If these deviations from the RFC are not needed and were to be fixed (along with the offset fix), then these implementations can detect the current mode of operation by examining the size of the iSER header received. The choice to proceed in the broken way, or to terminate the connection (with big loud error messages) is the implementor's choice. Either way, the issue is detected and corruption avoided.

Thoughts?

Cheers
Ken

> Date: Tue, 29 Apr 2008 13:05:16 -0400
> From: [email-masked]
> To: [email-masked]
> CC: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [Stgt-devel] [ofa-general] Re: [Ips] Calculating the VA in iSER	header
> 
> [email-masked] wrote on Thu, 17 Apr 2008 14:13 +0300:
> I finally got a chance to look at this just now.  I think you mean
> default is InitialR2T=No above, which means no unsolicited data.
> That is the default case, and true, the two different meanings
> of the initiator-supplied VA coincide.
> 
> But you missed the impact of immediate data.  We run with the
> defaults (I think) that say the first write request packet should be
> filled with a bit of the coming data stream.  From iscsid.conf:
> 
>     # To enable immediate data (i.e., the initiator sends unsolicited data
>     # with the iSCSI command packet), uncomment the following line:
>     #
>     # The default is Yes
>     node.session.iscsi.ImmediateData = Yes
> 
> Looking at the offset printed out by your patch, it is indeed
> non-zero for the first RDMA read.  Please correct me if I am
> mistaken about this---you must have tested all four variations of
> with and without the patches on initiator and target side, but I did
> not.
> 
> Hence I am still a bit unhappy about having to deal with the
> fallout, with no way to detect it.  For our local use, I'll keep an
> older version of stgt in use until we switch to a new kernel, then
> merge up the target side change.  It is a bother, but I can deal
> with it.  For other institutions, this lockstep upgrade requirement
> will not be obvious until they debug the resulting data corruption.
> 
> Still, I do understand why it would be nice to conform to the spec,
> and it is maybe a bit cleaner that way too.  Maybe you can help with
> the bug reports on stgt-devel during the transition, and maintain
> and publish a patch to let it work with old kernels.
> 
> 		-- Pete
> Stgt-devel mailing list
> [email-masked]
> https://lists.berlios.de/mailman/listinfo/stgt-devel

_________________________________________________________________
Find the job of your dreams before someone else does
http://mycareer.com.au/?s_cid=596064

---

From: "Erez Zilber" <[email-masked]>
Date: Wed, 30 Apr 2008 14:54:14 +0300

> If these deviations from the RFC are not needed and were to be fixed (along with the offset fix), then these implementations can detect the current mode of operation by examining the size > of the iSER header received. The choice to proceed in the broken way, or to terminate the connection (with big loud error messages) is the implementor's choice. Either way, the issue is detected and corruption avoided.
> 
> Thoughts?

Take a look at the iSER for IB annex:
 
http://www.infinibandta.org/members/spec/Annex_iSER.PDF

Erez

---

From: Ken Sandars <[email-masked]>
Date: Thu, 1 May 2008 18:24:15 +1000

>> [Ken] It appears the current Linux iSER initiator does not send the HELLO message when
>> the connection transits to full feature phase. The stgt target also ignores this
>> message (if it were to appear). 
[Ken] The IBTA document does not mention the HELLO/HELLOREPLY messages.
Implementing this message exchange gives a distinction between the current implementations
and those that will correctly calculate the write_va (as per Pete Wyckoff's option 3).

>> [Ken] Both of these implementations use a non-conformant iSER header (they add
>> write_va and read_va fields, which incidentally do not appear to be used). Are
>> these changes documented anywhere in the IB domain, or are these variations
>> needed for another reason?
>
> [Erez] Take a look at the iSER for IB annex:

[Ken] Ouch. That link requires a username/password. Looks like it is only available to members
of the Infiniband Trade Association. Fortunately I gained access to it with username "open" and
password "standard". ;-)

[Ken] Neither of these implementations send or examine the iSER CM REQ/REP message private
data. The document doesn't define what action to take when this message is absent. Interestingly,
when the target reports that "ZBVA shall be used for this connection" and "the target shall issue
Send with Invalidate as needed" then it appears the iSER header specified in RFC5046 should be
used for control-type PDUs. Is there any plan to conform with the list of requirements for IBTA
compliance?

Cheers,
Ken

_________________________________________________________________
Never miss another e-mail with Hotmail on your mobile.
http://www.livelife.ninemsn.com.au/article.aspx?id=343869