25.19. Modifying Link Loss Behavior
This section describes how to modify the link loss behavior of devices that use either Fibre Channel or iSCSI protocols.
25.19.1. Fibre Channel
If a driver implements the Transport
dev_loss_tmo callback, access attempts to a device through a link will be blocked when a transport problem is detected. To verify if a device is blocked, run the following command:
$ cat /sys/block/device/device/state
This command will return
blocked if the device is blocked. If the device is operating normally, this command will return running.
Procedure 25.15. Determining the State of a Remote Port
- To determine the state of a remote port, run the following command:
$ cat /sys/class/fc_remote_port/rport-H:B:R/port_state
- This command will return
Blockedwhen the remote port (along with devices accessed through it) are blocked. If the remote port is operating normally, the command will returnOnline. - If the problem is not resolved within
dev_loss_tmoseconds, the rport and devices will be unblocked and all I/O running on that device (along with any new I/O sent to that device) will be failed.
Procedure 25.16. Changing dev_loss_tmo
- To change the
dev_loss_tmovalue,echoin the desired value to the file. For example, to setdev_loss_tmoto 30 seconds, run:$ echo 30 > /sys/class/fc_remote_port/rport-H:B:R/dev_loss_tmo
For more information about
dev_loss_tmo, refer to Section 25.4.1, “Fibre Channel API”.
When a link loss exceeds
dev_loss_tmo, the scsi_device and sdN devices are removed. Typically, the Fibre Channel class will leave the device as is; i.e. /dev/sdx will remain /dev/sdx. This is because the target binding is saved by the Fibre Channel driver so when the target port returns, the SCSI addresses are recreated faithfully. However, this cannot be guaranteed; the sdx will be restored only if no additional change on in-storage box configuration of LUNs is made.
25.19.2. iSCSI Settings with dm-multipath
If
dm-multipath is implemented, it is advisable to set iSCSI timers to immediately defer commands to the multipath layer. To configure this, nest the following line under device { in /etc/multipath.conf:
features "1 queue_if_no_path"
This ensures that I/O errors are retried and queued if all paths are failed in the
dm-multipath layer.
You may need to adjust iSCSI timers further to better monitor your SAN for problems. Available iSCSI timers you can configure are NOP-Out Interval/Timeouts and
replacement_timeout, which are discussed in the following sections.
25.19.2.1. NOP-Out Interval/Timeout
To help monitor problems the SAN, the iSCSI layer sends a NOP-Out request to each target. If a NOP-Out request times out, the iSCSI layer responds by failing any running commands and instructing the SCSI layer to requeue those commands when possible.
When
dm-multipath is being used, the SCSI layer will fail those running commands and defer them to the multipath layer. The multipath layer then retries those commands on another path. If dm-multipath is not being used, those commands are retried five times before failing altogether.
Intervals between NOP-Out requests are
5 seconds by default. To adjust this, open /etc/iscsi/iscsid.conf and edit the following line:
node.conn[0].timeo.noop_out_interval = [interval value]
Once set, the iSCSI layer will send a NOP-Out request to each target every [interval value] seconds.
By default, NOP-Out requests time out in
5 seconds[9]. To adjust this, open /etc/iscsi/iscsid.conf and edit the following line:
node.conn[0].timeo.noop_out_timeout = [timeout value]
This sets the iSCSI layer to timeout a NOP-Out request after [timeout value] seconds.
SCSI Error Handler
If the SCSI Error Handler is running, running commands on a path will not be failed immediately when a NOP-Out request times out on that path. Instead, those commands will be failed after
replacement_timeout seconds. For more information about replacement_timeout, refer to Section 25.19.2.2, “replacement_timeout”.
To verify if the SCSI Error Handler is running, run:
# iscsiadm -m session -P 3
25.19.2.2. replacement_timeout
replacement_timeout controls how long the iSCSI layer should wait for a timed-out path/session to reestablish itself before failing any commands on it. The default replacement_timeout value is 120 seconds.
To adjust
replacement_timeout, open /etc/iscsi/iscsid.conf and edit the following line:
node.session.timeo.replacement_timeout = [replacement_timeout]
The
1 queue_if_no_path option in /etc/multipath.conf sets iSCSI timers to immediately defer commands to the multipath layer (refer to Section 25.19.2, “iSCSI Settings with dm-multipath”). This setting prevents I/O errors from propagating to the application; because of this, you can set replacement_timeout to 15-20 seconds.
By configuring a lower
replacement_timeout, I/O is quickly sent to a new path and executed (in the event of a NOP-Out timeout) while the iSCSI layer attempts to re-establish the failed path/session. If all paths time out, then the multipath and device mapper layer will internally queue I/O based on the settings in /etc/multipath.conf instead of /etc/iscsi/iscsid.conf.
Important
Whether your considerations are failover speed or security, the recommended value for
replacement_timeout will depend on other factors. These factors include the network, target, and system workload. As such, it is recommended that you thoroughly test any new configurations to replacements_timeout before applying it to a mission-critical system.
iSCSI and DM Multipath overrides
The
recovery_tmo sysfs option controls the timeout for a particular iSCSI device. The following options globally override recovery_tmo values:
- The
replacement_timeoutconfiguration option globally overrides therecovery_tmovalue for all iSCSI devices. - For all iSCSI devices that are managed by DM Multipath, the
fast_io_fail_tmooption in DM Multipath globally overrides therecovery_tmovalue. Thefast_io_fail_tmooption in DM Multipath also overrides thefast_io_fail_tmooption in Fibre Channel devices.The DM Multipathfast_io_fail_tmooption takes precedence overreplacement_timeout. Red Hat does not recommend usingreplacement_timeoutto overriderecovery_tmoin devices managed by DM Multipath because DM Multipath always resetsrecovery_tmowhen themultipathdservice reloads.
25.19.3. iSCSI Root
When accessing the root partition directly through an iSCSI disk, the iSCSI timers should be set so that iSCSI layer has several chances to try to reestablish a path/session. In addition, commands should not be quickly re-queued to the SCSI layer. This is the opposite of what should be done when
dm-multipath is implemented.
To start with, NOP-Outs should be disabled. You can do this by setting both NOP-Out interval and timeout to zero. To set this, open
/etc/iscsi/iscsid.conf and edit as follows:
node.conn[0].timeo.noop_out_interval = 0 node.conn[0].timeo.noop_out_timeout = 0
In line with this,
replacement_timeout should be set to a high number. This will instruct the system to wait a long time for a path/session to reestablish itself. To adjust replacement_timeout, open /etc/iscsi/iscsid.conf and edit the following line:
node.session.timeo.replacement_timeout = replacement_timeout
After configuring
/etc/iscsi/iscsid.conf, you must perform a re-discovery of the affected storage. This will allow the system to load and use any new values in /etc/iscsi/iscsid.conf. For more information on how to discover iSCSI devices, refer to Section 25.15, “Scanning iSCSI Interconnects”.
Configuring Timeouts for a Specific Session
You can also configure timeouts for a specific session and make them non-persistent (instead of using
/etc/iscsi/iscsid.conf). To do so, run the following command (replace the variables accordingly):
# iscsiadm -m node -T target_name -p target_IP:port -o update -n node.session.timeo.replacement_timeout -v $timeout_value
Important
The configuration described here is recommended for iSCSI sessions involving root partition access. For iSCSI sessions involving access to other types of storage (namely, in systems that use
dm-multipath), refer to Section 25.19.2, “iSCSI Settings with dm-multipath”.
