6.1. interface_id

This must be set to 'S' (capital ess). If not, the ENOSYS error message is placed in errno. The idea is to allow interface variants in the future that identify themselves with a different value. [The parallel port generic driver (pg) uses the letter 'P' to identify itself.] The type of interface_id is int.

6.2. dxfer_direction

The type of dxfer_direction is int. This is required to be one of the following:

• SG_DXFER_NONE /* e.g. a SCSI Test Unit Ready command */

• SG_DXFER_TO_DEV /* e.g. a SCSI WRITE command */

• SG_DXFER_FROM_DEV /* e.g. a SCSI READ command */

• SG_DXFER_TO_FROM_DEV

• SG_DXFER_UNKNOWN

6.3. cmd_len

This is the length in bytes of the SCSI command that 'cmdp' points to. As a SCSI command is expected an EMSGSIZE error number is produced if the value is less than 6 or greater than 16. Further, if the SCSI mid level has a further limit then EMSGSIZE is produced in this case as well. [4] The type of cmd_len is unsigned char.

6.4. mx_sb_len

This is the maximum size that can be written back to the 'sbp' pointer when a sense_buffer is output which is usually in an error situation. The actual number written out is given by 'sb_len_wr'. In all cases 'sb_len_wr' <= 'mx_sb_len' . The type of mx_sb_len is unsigned char.

6.5. iovec_count

This is the number of scatter gather elements in an array pointed to by 'dxferp'. If the value is zero then scatter gather (in the user space) is _not_ being used and 'dxferp' points to the data transfer buffer. If the value is greater than zero then each element of the array is assumed to be of the form:

typedef struct sg_iovec { void * iov_base; /* starting address */ size_t iov_len; /* length in bytes */ } sg_iovec_t;

Note that this structure has been named and defined in such a way to parallel "struct iovec" used by the readv() and writev() system calls in Linux. See "man 2 readv".

Note that the scatter gather capability offered by 'iovec_count' is unrelated to the scatter gather capability (often associated with DMA) offered by most modern SCSI adapters. Furthermore iovec_count's variety of scatter gather (into the user space) is only available when normal (or "indirect") IO is being used. Hence when the SG_FLAG_DIRECT_IO or SG_FLAG_MMAP_IO are set in 'flags' then 'iovec_count' should be zero.

The type of iovec_count is unsigned short.

6.6. dxfer_len

This is the number of bytes to be moved in the data transfer associated with the command. The direction of the transfer is indicated by 'dxfer_direction'. If 'dxfer_len' is zero then no data transfer takes place. [5]

If iovec_count is non-zero then 'dxfer_len' should be equal to the sum of iov_len lengths. If not, the minimum of the two is the transfer length. The type of dxfer_len is unsigned int.

6.7. dxferp

If 'iovec_count' is zero then this value is a pointer to user memory of at least 'dxfer_len' bytes in length. If there is a data transfer associated with the command then the data will be transferred to or from this user memory. If 'iovec_count' is greater than zero then this value points to a scatter-gather array in user memory. Each element of this array should be an object of type sg_iovec_t. Note that data is sometimes written to user memory (e.g. from a failed SCSI READ) even when an error has occurred.

If mmap-ed IO is selected then the value in 'dxferp' is ignored and any data transfers will be to and from the address returned by the prior mmap() call.

The type of dxferp is void * .

6.8. cmdp

This value points to the SCSI command to be executed. The command is assumed to be 'cmd_len' bytes long. If cmdp is NULL then the system call yields an EMSGSIZE error number. The user memory pointed to is only read (not written to). The type of cmdp is unsigned char * .

6.9. sbp

This value points to user memory of at least 'mx_sb_len' bytes length where the SCSI sense buffer will be output. Most successful commands do not output a sense buffer and this will be indicated by 'sb_len_wr' being zero. Note that there are error conditions that don't result in a sense buffer be generated. The sense buffer results from the "auto-sense" mechanism in the SCSI mid-level driver. This mechanism detects a CHECK_CONDITION status and issues a REQUEST SENSE command and conveys its response back as the "sense buffer". The type of sbp is unsigned char * .

6.10. timeout

This value is used to timeout the given command. The units of this value are milliseconds. The time being measured is from when a command is sent until when sg is informed the request has been completed. A following read() can take as long as the user likes. Timeouts are best avoided, especially if SCSI bus resets will adversely effect other devices on that SCSI bus. When the timeout expires, the SCSI mid level attempts error recovery. Error recovery completes when the first action in the following list is successful. Note that a more extreme measure is being taken at each step.

• the SCSI command that has timed out is aborted [6]

• a SCSI device reset is attempted

• a SCSI bus reset is attempted. Note this may have an adverse effect on other devices sharing that SCSI bus.

• a SCSI host (bus adapter) reset is attempted. This is an attempt to re-initialize the adapter card associated with the SCSI device that has the timed out command.

The two error statuses containing the word "TIME(_)OUT" are typically _not_ related to a command timing out. DID_TIME_OUT in the 'host_status' usually means an (unexpected) device selection timeout. DRIVER_TIMEOUT in the 'driver_status' byte means the SCSI adapter is unable to control the devices on its SCSI bus (and has given up).

The type of timeout is unsigned int (and it represents milliseconds).

6.11. flags

These are single or multi-bit values that can be "or-ed" together:

• SG_FLAG_DIRECT_IO This is a request for direct IO on the data transfer. If it cannot be performed then the driver automatically performs indirect IO instead. If it is important to find out which type of IO was performed then check the values from the SG_INFO_DIRECT_IO_MASK in 'info' when the request packet is completed (i.e. after read() or ioctl(,SG_IO,) ). The default action is to do indirect IO.

• SG_FLAG_LUN_INHIBIT The default action of the sg driver to overwrite internally the top 3 bits of the second SCSI command byte with the LUN associated with the file descriptor's device. To inhibit this action set this flag. For SCSI 3 (or later) devices, this internal LUN overwrite does not occur.

• SG_FLAG_MMAP_IO When set the driver will attempt to procure the reserved buffer. If the reserved buffer is occupied (EBUSY) or too small (ENOMEM) then the operation (write() or ioctl(SG_IO)) fails. No data transfers occur between the dxferp pointer and the reserved buffer (dxferp is ignored). In order for a user application to access mmap-ed IO, it must have successfully executed an appropriate mmap() system call on this sg file descriptor. This precondition is not checked by write() or ioctl(SG_IO) when this flag is set. Setting this flag and SG_FLAG_DIRECT_IO results in a EINVAL error.

• SG_FLAG_NO_DXFER When set user space data transfers to or from the kernel buffers do not take place. This only has effect during indirect IO. This flag is for testing bus speed (e.g. the "sg_rbuf" utility uses it).

6.12. pack_id

This value is not normally acted upon by the sg driver. It is provided so the user can identify the request. This is useful when command queuing is being used. The "abnormal" case is when SG_SET_FORCE_PACK_ID is set and a 'pack_id' other than -1 is given to read(). In this case the read() will wait to fetch a request that matches this 'pack_id'. If this mode is used be careful to set 'dxfer_direction' to a valid value (actually any of the SG_DXFER_* values will do) on input to the read(), together with the wanted pack_id. The type of pack_id is int.

6.13. usr_ptr

This value is not acted upon by the sg driver. It is meant to allow the user to associate some object with this request (e.g. to maintain state information). The type of usr_ptr is void * .

6.14. status

This is the SCSI status byte as defined by the SCSI standard. Note that it can have vendor information set in bits 0, 6 and 7 (although this is uncommon). Further note that this 'status' data does _not_ match the definitions in <scsi/scsi.h> (e.g. CHECK_CONDITION). The following 'masked_status' does match those definitions. [7] The type of status is unsigned char .

6.15. masked_status

Logically: masked_status == ((status & 0x3e) >> 1) . So 'masked_status' strips the vendor information bits off 'status' and then shifts it right one position. This makes it easier to do things like "if (CHECK_CONDITION == masked_status) ..." using the definitions in <scsi/scsi.h>. The defined values in this file are:

• GOOD [0x00]

• CHECK_CONDITION [0x01]

• CONDITION_GOOD [0x02]

• BUSY 0x04

• INTERMEDIATE_GOOD 0x08

• INTERMEDIATE_C_GOOD 0x0a

• RESERVATION_CONFLICT 0x0c

• COMMAND_TERMINATED 0x11

• QUEUE_FULL 0x14

Note that SCSI 3 defines some additional status codes. [8] The type of masked_status is unsigned char .

6.16. msg_status

The messaging level in SCSI is under the command level and knowledge of what is happening at the messaging level is very rarely needed. Furthermore most modern chip-sets used in SCSI adapters completely hide this value. Nearly all adapters will return zero in 'msg_status' all the time. The type of msg_status is unsigned char .

6.17. sb_len_wr

This is the actual number of bytes written to the user memory pointed to by 'sbp'. 'sb_len_wr' is always <= 'mx_sb_len'. Linux 2.2 series kernels (and earlier) truncate this value to a maximum of 16 bytes. The actual number of bytes written will not exceed the length indicated by "Additional Sense Length" field (byte 7) of the Request Sense response. The type of sb_len_wr is unsigned char .

6.18. host_status

These codes potentially come from the firmware on a host adapter or from one of several hosts that an adapter driver controls. The 'host_status' field has the following values whose #defines mimic those which are only visible within the kernel (with the "SG_ERR_" removed from the front of each define). A copy of these defines can be found in sg_err.h (see Appendix A):

• SG_ERR_DID_OK [0x00] NO error

• SG_ERR_DID_NO_CONNECT [0x01] Couldn't connect before timeout period

• SG_ERR_DID_BUS_BUSY [0x02] BUS stayed busy through time out period

• SG_ERR_DID_TIME_OUT [0x03] TIMED OUT for other reason (often this an unexpected device selection timeout)

• SG_ERR_DID_BAD_TARGET [0x04] BAD target, device not responding?

• SG_ERR_DID_ABORT [0x05] Told to abort for some other reason. From lk 2.4.15 the SCSI subsystem supports 16 byte commands however few adapter drivers do. Those HBA drivers that don't support 16 byte commands will yield this error code if a 16 byte command is passed to a SCSI device they control.

• SG_ERR_DID_PARITY [0x06] Parity error. Older SCSI parallel buses have a parity bit for error detection. This probably indicates a cable or termination problem.

• SG_ERR_DID_ERROR [0x07] Internal error detected in the host adapter. This may not be fatal (and the command may have succeeded). The aic7xxx and sym53c8xx adapter drivers sometimes report this for data underruns or overruns. [9]

• SG_ERR_DID_RESET [0x08] The SCSI bus (or this device) has been reset. Any SCSI device on a SCSI bus is capable of instigating a reset.

• SG_ERR_DID_BAD_INTR [0x09] Got an interrupt we weren't expecting

• SG_ERR_DID_PASSTHROUGH [0x0a] Force command past mid-layer

• SG_ERR_DID_SOFT_ERROR [0x0b] The low level driver wants a retry

6.19. driver_status

One driver can potentially control several host adapters. For example Advansys provide one Linux adapter driver that controls all adapters made by that company - if 2 of more Advansys adapters are in 1 machine, then 1 driver controls both. When ('driver_status' & SG_ERR_DRIVER_SENSE) is true the 'sense_buffer' is also output. The 'driver_status' field has the following values whose #defines mimic those which are only visible within the kernel (with the "SG_ERR_" removed from the front of each define). A copy of these defines can be found in sg_err.h (see the utilities section):

• SG_ERR_DRIVER_OK [0x00] Typically no suggestion

• SG_ERR_DRIVER_BUSY [0x01]

• SG_ERR_DRIVER_SOFT [0x02]

• SG_ERR_DRIVER_MEDIA [0x03]

• SG_ERR_DRIVER_ERROR [0x04]

• SG_ERR_DRIVER_INVALID [0x05]

• SG_ERR_DRIVER_TIMEOUT [0x06] Adapter driver is unable to control the SCSI bus to its is setting its devices offline (and giving up)

• SG_ERR_DRIVER_HARD [0x07]

• SG_ERR_DRIVER_SENSE [0x08] Implies sense_buffer output

• above status 'or'ed with one of the following suggestions

• SG_ERR_SUGGEST_RETRY [0x10]

• SG_ERR_SUGGEST_ABORT [0x20]

• SG_ERR_SUGGEST_REMAP [0x30]

• SG_ERR_SUGGEST_DIE [0x40]

• SG_ERR_SUGGEST_SENSE [0x80]

6.20. resid

This is the residual count from the data transfer. It is 'dxfer_len' less the number of bytes actually transferred. In practice it only reports underruns (i.e. positive number) as data overruns should never happen. This value will be zero if there was no underrun or the SCSI adapter doesn't support this feature. [10] The type of resid is int .

6.21. duration

This value will be the number of milliseconds from when a SCSI command was sent until sg is informed that it is complete. For i386 machines the granularity is 10ms while on alpha machines it is 1ms. This value is rounded toward zero. The type of duration is unsigned int .

6.22. info

This value is designed to convey useful information back to the user about the associated request. This information does not necessarily indicate an error. Several single bit and multi-bit fields are "or-ed" together to make this value.

A single bit component contained in SG_INFO_OK_MASK indicates whether some error or status field is non-zero. If either 'masked_status', 'host_status' or 'driver_status' are non-zero then SG_INFO_CHECK is set. The associated values are:

• SG_INFO_OK_MASK [0x1]

• SG_INFO_OK [0x0] no sense, host nor driver "noise"

• SG_INFO_CHECK [0x1] something abnormal happened. In most but not all cases, the sense buffer will be written. If the sense buffer has not been written than 'sb_len_wr' will be zero. This flag indicates either 'masked_status', 'host_status' or 'driver_status' is non-zero.

A multi bit component contained in SG_INFO_DIRECT_IO_MASK indicates what type of data transfer has just taken place. If indirect IO (or no data transfer) has taken place then SG_INFO_INDIRECT_IO is matched. Note that even if direct IO was requested in 'flags' the driver may choose to do indirect IO instead. If direct IO was requested and performed then SG_INFO_DIRECT_IO will be matched. Currently SG_INFO_MIXED_IO is never set. The associated values are:

• SG_INFO_DIRECT_IO_MASK [0x6]

• SG_INFO_INDIRECT_IO [0x0] data xfer via kernel buffers (or no xfer)

• SG_INFO_DIRECT_IO [0x2]

• SG_INFO_MIXED_IO [0x4] part direct, part indirect IO

7.1. open()

Note that multiple file descriptors may be open to the same SCSI device. [This is a way of side stepping the SG_MAX_QUEUE limit.] At the sg level separate state information is maintained. This means that even if multiple file descriptors are open to a single SCSI device their write() read() sequences are essentially independent.

Open() calls may be blocked due to exclusive locks (i.e. O_EXCL). An exclusive lock applies to a single SCSI device and only to sg's use of that device (i.e. it has no effect on access via sd, sr or st to that device). If the O_NONBLOCK flag is used then open() calls that would have otherwise blocked, yield EBUSY. Applications that scan sg devices trying to determine their identity (e.g. whether one is a scanner) should use the O_NONBLOCK flag otherwise they run the risk of blocking.

The driver will attempt to reserve SG_DEF_RESERVED_SIZE bytes (32KBytes in the current sg.h) on open(). The size of this reserved buffer can subsequently be modified with the SG_SET_RESERVED_SIZE ioctl(). In both cases these are requests subject to various dynamic constraints. The actual amount of memory obtained can be found by the SG_GET_RESERVED_SIZE ioctl(). The reserved buffer will be used if:

• it is not already in use (e.g. when command queuing is in use)

• a write() or ioctl(SG_IO) requests a data transfer size that is less than or equal to the reserved buffer size.

Returns a file descriptor if >= 0 , otherwise -1 implies an error.

7.2. write()

The 'buffer' should point to an object of type sg_io_hdr_t and 'count' should be sizeof(sg_io_hdr_t) [it can be larger but the excess is ignored]. If the write() call succeeds then the 'count' is returned as the result.

Up to SG_MAX_QUEUE (16) write()s can be queued up before any finished requests are completed by read(). An attempt to queue more than that will result in an EDOM error. [11] The write() command should return more or less immediately. [12]

The version 2 sg driver defaulted the maximum queue length to 1 (and made available the SG_SET_COMMAND_Q ioctl() to switch it to SG_MAX_QUEUE). So for backward compatibility a file descriptor that only receives sg_header structures in its write() will have a default "max" queue length of 1. As soon as a sg_io_hdr_t structure is seen by a write() then the maximum queue length is switched to SG_MAX_QUEUE on that file descriptor.

The "const" on the 'buffer' pointer is respected by the sg driver. Data is read in from the sg_io_hdr object that is pointed to. Significantly this is when the 'sbp' and the 'dxferp' are recorded internally (i.e. not from the sg_io_hdr object given to the corresponding read() ).

7.3. read()

The 'buffer' should point to an object of type sg_io_hdr_t and 'count' should be sizeof(sg_io_hdr_t) [it can be larger but the excess is ignored]. If the read() call succeeds then the 'count' is returned as the result.

By default, read() will return the oldest completed request that is queued up. A read() will not interfere with any request associated with the SG_IO ioctl() on this file descriptor except in a special case when a SG_IO ioctl() is interrupted by a signal.

If the SG_SET_FORCE_PACK_ID,1 ioctl() is active then read() will attempt to fetch the packet whose pack_id (given earlier to write()) matches the sg_io_hdr_t::pack_id given to this read(). If not available it will either wait or yield EAGAIN. As a special case, -1 in sg_io_hdr_t::pack_id given to read() will match the request whose response has been waiting for the longest time. Take care to also set 'dxfer_direction' to any valid value (e.g. SG_DXFER_NONE) when in this mode. The 'interface_id' member should also be set appropriately.

Apart from the SG_SET_FORCE_PACK_ID case (and then only for the 3 indicated fields), the sg_io_hdr_t object given to read() can be uninitialized. Note that the 'sbp' pointer value for optionally outputting a sense buffer was recorded from the earlier, corresponding write().

7.4. poll()

For file descriptors associated with sg devices:

• POLLIN one or more responses is awaiting a read()

• POLLOUT command can be sent to write() without causing an EDOM error (i.e. sufficient space on sg's queues)

• POLLHUP SCSI device has been detached, awaiting cleanup

• POLLERR internal structures are inconsistent

POLLOUT indicates the sg will not block a new write() or SG_IO ioctl(). However it is still possible (but unlikely) that the mid level or an adapter may block (or yield EAGAIN).

7.5. close()

When close() leaves outstanding SCSI commands still awaiting responses, the sg driver maintains its internal structures for the now defunct file descriptor. These internal structures are maintained until all outstanding responses (some might be timeouts) are received. When the sg driver is loaded as a module and has any open file descriptors or "defunct" file descriptors then it cannot be unloaded. An attempt to call rmmod sg will report the driver is busy. Defunct file descriptors that remain for some time, perhaps awaiting a timeout, can be observed with the cat /proc/scsi/sg/debug command. In this case "closed=1" will be set on the defunct file descriptor [see Section 11.1]. Defunct file descriptors do not impede attempts by applications to open() new file descriptors on the same SCSI device.

The kernel arranges for only the last close() on a file descriptor to be seen by a driver (and to emphasize this, the corresponding sg driver call is named sg_release() rather than sg_close()). This is only significant when an application uses fork() or dup().

Returns 0 if successful, otherwise -1 implies an error.

7.6. mmap()

The mmap() system call can be made multiple times on the same sg_fd. The munmap() system call is not required if close() is called on sg_fd. Mmap-ed IO is well-behaved when a process is fork()-ed (or the equivalent finer grained clone() system call is made). In the case of a fork(), 2 processes will be sharing the same memory mapped area together with the sg driver for a sg_fd and the last one to close the sg_fd (or exit) will cause the shared memory to be freed.

It is assumed that if the default reserved buffer size of 32 KB is not sufficient then a ioctl(SG_SET_RESERVED_SIZE) call is made prior to any calls to mmap(). If the required size is not a multiple of the kernel's page size (returned by getpagesize() system call) then the size passed to ioctl(SG_SET_RESERVED_SIZE) should be rounded up to the next page size multiple.

Mmap-ed IO is requested by setting (or or-ing in) the SG_FLAG_MMAP_IO constant into the flag member of the the sg_io_hdr structure prior to a call to write() or ioctl(SG_IO). The logic to do mmap-ed IO _assumes_ that an appropriate mmap() call has been made by the application. In other words it does not check. [13]

7.7. fcntl(sg_fd, F_SETFL, oflags | FASYNC)

The sg driver supports asynchronous notification. This is a non-blocking mode of operation in which, when the driver receives data back from a device so that a read() can be done, it sends a SIGPOLL (aka SIGIO) signal to the owning process. Here is a code snippet from the sg_poll test program.

sigemptyset(&sig_set) sigaddset(&sig_set, SIGPOLL) sigaction(SIGPOLL, &s_action, 0) fcntl(sg_fd, F_SETOWN, getpid()) flags = fcntl(sg_fd, F_GETFL); fcntl(sg_fd, F_SETFL, flags | O_ASYNC)

7.8. Errors reported in errno

With the original interface almost any string could be accidentally given to write() and potentially (but rarely) something nasty could happen. If some error was detected then more than likely EIO was placed in errno.

Unfortunately this can still happen with write() since it can accept both the original struct sg_header or the newer sg_io_hdr_t described in this note. However since the SG_IO ioctl() will only accept the sg_io_hdr_t structure there is less chance of a random string being interpreted as a command. Since the sg_io_hdr_t interface does a lot more error checking, it attempts to give out more precise errno values to help the user pinpoint the problem. [Admittedly some of these errno values are picked in an arbitrary way from the large set of available values.]

In most cases when a system call on a sg file descriptor fails, the call in question will return -1. After an application detects that a system call has failed it should read the value in the "errno" variable (prior to do any more system calls). Applications should include the <errno.h> header.

Below is a table of errno values indicating which calls to sg will generate them and the meaning of the error. A write() call is indicated by "w", a read() call by "r" and an open() call by "o".

errno which_calls Meaning ----- ----------- ---------------------------------------------- EACCES <some ioctls> Root permission (more precisely CAP_SYS_ADMIN or CAP_SYS_RAWIO) required. Also may occur during an attempted write to /proc/scsi/sg files. EAGAIN r The file descriptor is non-blocking and the request has not been completed yet. EAGAIN w,SG_IO SCSI sub-system has (temporarily) run out of command blocks. EBADF w File descriptor was not open()ed O_RDWR. EBUSY o Someone else has an O_EXCL lock on this device. EBUSY w With mmap-ed IO, the reserved buffer already in use. EBUSY <some ioctls> Attempt to change something (e.g. reserved buffer size) when the resource was in use. EDOM w,SG_IO Too many requests queued against this file descriptor. Limit is SG_MAX_QUEUE active requests. If sg_header interface is being used then the default queue depth is 1. Use SG_SET_COMMAND_Q ioctl() to increase it. EFAULT w,r,SG_IO Pointer to user space invalid. <most ioctls> EINVAL w,r Size given as 3rd argument not large enough for the sg_io_hdr_t structure. Both direct and mmap-ed IO selected. EIO w Size given as 3rd argument less than size of old header structure (sg_header). Additionally a write() with the old header will yield this error for most detected malformed requests. EIO r A read() with the older sg_header structure yields this value for some errors that it detects. EINTR o While waiting for the O_EXCL lock to clear this call was interrupted by a signal. EINTR r,SG_IO While waiting for the request to finish this call was interrupted by a signal. EINTR w [Very unlikely] While waiting for an internal SCSI resource this call was interrupted by a signal. EMSGSIZE w,SG_IO SCSI command size ('cmd_len') was too small (i.e. < 6) or too large ENODEV o Tried to open() a file with no associated device. [Perhaps sg has not been built into the kernel or is not available as a module?] ENODEV o,w,r,SG_IO SCSI device has detached, awaiting cleanup. User should close fd. Poll() will yield POLLHUP. ENOENT o Given filename not found. ENOMEM o [Very unlikely] Kernel was not even able to find enough memory for this file descriptor's context. ENOMEM w,SG_IO Kernel unable to find memory for internal buffers. This is usually associated with indirect IO. For mmap-ed IO 'dxfer_len' greater than reserved buffer size. Lower level (adapter) driver does not support enough scatter gather elements for requested data transfer. ENOSYS w,SG_IO 'interface_id' of a sg_io_hdr_t object was _not_ 'S'. ENXIO o "remove-single-device" may have removed this device. ENXIO o, w,r,SG_IO Internal error (including SCSI sub-system busy doing error processing - e.g. SCSI bus reset). When a SCSI device is offline, this is the response. This can be bypassed by opening O_NONBLOCK. EPERM o Can't use O_EXCL when open()ing with O_RDONLY EPERM w,SG_IO File descriptor open()-ed O_RDONLY but O_RDWR <some ioctls> access mode needed for this operation.

8.1. SG_IO

The same file descriptor can be used both for SG_IO synchronous calls and the write() read() sequences at the same time. The sg driver makes sure that the response to a SG_IO call will never accidentally be fetched by a read(). Even though a single file descriptor can be shared in this manner, it is probably more sensible (and results in cleaner code) if separate file descriptors to the same SCSI device are used in this case.

It is possible that the wait for the command completion is interrupted by a signal. In this case the SG_IO call will yield an EINTR error. This is reasonably complex to handle and is discussed in the ioctl(SG_SET_KEEP_ORPHAN) description below. The following SCSI commands will be permitted by SG_IO when the sg file descriptor was opened O_RDONLY:

• TEST UNIT READY

• REQUEST SENSE

• INQUIRY

• READ CAPACITY

• READ BUFFER

• READ(6) (10) and (12)

• MODE SENSE(6) and (10)

• LOG SENSE

8.2. SG_GET_ACCESS_COUNT

8.3. SG_SET_COMMAND_Q (and _GET_)

8.4. SG_SET_DEBUG

The other actions of debug mode performed in version 2 of the sg driver have been removed as they are no longer needed. The internal state of the sg driver can now be found by viewing the output of cat /proc/scsi/sg/debug.

8.5. SG_EMULATED_HOST

8.6. SG_SET_KEEP_ORPHAN (and _GET_)

8.7. SG_SET_FORCE_LOW_DMA

8.8. SG_GET_LOW_DMA

8.9. SG_NEXT_CMD_LEN

8.10. SG_GET_NUM_WAITING

8.11. SG_SET_FORCE_PACK_ID

8.12. SG_GET_PACK_ID

8.13. SG_GET_REQUEST_TABLE

8.14. SG_SET_RESERVED_SIZE (and _GET_ )

For ioctl(SG_SET_RESERVED_SIZE) the value will be used to request a new reserved buffer of that size. The previous reserved buffer is freed (if it is not in use; if it was in use then the ioctl() fails and EBUSY is placed in errno). A new reserved buffer is then allocated and its actual size can be found by calling the ioctl(SG_GET_RESERVED_SIZE). The reserved buffer is then used for DMA purposes by subsequent write() and ioctl(SG_IO) commands if it is not already in use and if the write() is not calling for a buffer size larger than that reserved. The reserved buffer may well be a series of kernel buffers if the adapter supports scatter-gather. Large buffers can be requested (e.g. 4 MB) but not necessarily granted. Once a mmap() call has been made on a sg file descriptor, subsequent calls to this ioctl() will fail with EBUSY placed in errno.

In the case of ioctl(SG_GET_RESERVED_SIZE) the size in bytes of the reserved buffer from open() or the most recent SG_SET_RESERVED_SIZE ioctl() call on this fd. The result can be 0 if memory is very tight. In this case it may not be wise to attempt something like burning a CD on this file descriptor.

8.15. SG_SCSI_RESET

Unfortunately this ioctl() doesn't currently do much (but may in the future after other issues are resolved). Yields an EBUSY error if the SCSI bus or the associated device is being reset when this ioctl() is called, otherwise returns 0. N.B. In some recent distributions there is a patch to the SCSI mid level code that activates this ioctl. Check your distribution.

8.16. SG_GET_SCSI_ID

8.17. SG_GET_SG_TABLESIZE

8.18. SG_GET_TIMEOUT

8.19. SG_SET_TIMEOUT

8.20. SG_SET_TRANSFORM

8.21. SG_GET_TRANSFORM

8.22. Sg ioctls removed in version 3

Some seldom used ioctl()s introduced in the sg 2.x series drivers have been withdrawn. They are:

• SG_SET_UNDERRUN_FLAG (and _GET_) [use 'resid' in this new interface]

• SG_SET_MERGE_FD (and _GET) [added complexity with little benefit]

8.23. SCSI_IOCTL_GET_IDLUN

The 'host_no' element is a change in lk 2.4 kernels. [In the lk 2.2 series and earlier, it was 'low_inode & 0xff' from the procfs entry corresponding to the host.] This change makes the use of the SCSI_IOCTL_GET_BUS_NUMBER ioctl() superfluous.

The advantage of this ioctl() is that it can be called on any SCSI file descriptor.

8.24. SCSI_IOCTL_GET_PCI

8.25. SCSI_IOCTL_PROBE_HOST

8.26. SCSI_IOCTL_SEND_COMMAND

The structure that we are passed should look like:

struct sdata { unsigned int inlen; [i] Length of data written to device unsigned int outlen; [i] Length of data read from device unsigned char cmd[x]; [i] SCSI command (6 <= x <= 16) [o] Data read from device starts here [o] On error, sense buffer starts here unsigned char wdata[y]; [i] Data written to device starts here };

Notes:

• The SCSI command length is determined by examining the 1st byte of the given command [15] . There is no way to override this.

• Data transfers are limited to PAGE_SIZE (4K on i386, 8K on alpha).

• The length (x + y) must be at least OMAX_SB_LEN bytes long to accommodate the sense buffer when an error occurs. The sense buffer is truncated to OMAX_SB_LEN (16) bytes so that old code will not be surprised.

• If a Unix error occurs (e.g. ENOMEM) then the user will receive a negative return and the Unix error code in 'errno'. If the SCSI command succeeds then 0 is returned. Positive numbers returned are the compacted SCSI error codes (4 bytes in one int) where the lowest byte is the SCSI status. See the drivers/scsi/scsi.h file for more information on this.

9.1. Direct IO

Direct IO uses the kiobuf mechanism [see the Linux Device Drivers book] to manipulate memory allocated within the user space so that a lower level (adapter) driver can DMA directly to or from that user space memory. Since the user can give a different data buffer to each SCSI command passed through the sg interface then the kiobuf mechanism needs to setup its structures (and undo that setup) for each SCSI command. [17] Direct IO is available as an option in sg 3.1.18 (before that the sg driver needed to be recompiled with an altered define). Direct IO support is designed in such a way that if it is requested and cannot be performed then the command will still be performed using indirect IO. If direct IO is requested and has been performed then the SG_INFO_DIRECT_IO bit will be set in the 'info' member of the sg_io_hdr_t control structure after the request has been completed. Direct IO is not supported on ISA SCSI adapters since they only can address a 24 bit address space.

One limit on direct IO is that sg_io_hdr_t::iovec_count==0. So the user cannot (currently) use application level scatter gather and direct IO on the same request.

For direct IO to be worthwhile, a reasonable amount of data should be requested for data transfer. For transfers less than 8 KByte it is probably not worth the trouble. On the other hand "locking down" a multiple 512 KB blocks of data for direct IO could adversely impact overall system performance. Remember that for the duration of a direct IO request, the data transfer buffer is mapped to a fixed memory location and locked in such a way that it won't be swapped out. This can "cramp the style" of the kernel if it is overdone.

Prior to sg 3.1.18 the direct IO code was commented out with the "SG_ALLOW_DIO" define. In sg 3.1.18 (available for lk 2.4.2 and later) the direct IO code is active but is defaulted off by a run time value. This value can be accessed via the "proc" file system at /proc/scsi/sg/allow_dio . Direct IO is enabled when a user with root permissions writes "1" to that file: echo 1 > /proc/scsi/sg/allow_dio . If SG_FLAG_DIRECT_IO is set in sg_io_hdr::flags but /proc/scsi/sg/allow_dio holds "0" then indirect IO will be performed (and this is indicated by ((sg_io_hdr::info & SG_INFO_DIRECT_IO_MASK) == SG_INFO_INDIRECT_IO) after the request is completed).

9.2. Mmap-ed IO

Memory-mapped IO takes a different approach from direct IO to removing the extra data copy performed by normal ("indirect") IO. With mmap-ed IO the application calls the mmap() system call to memory map sg's reserved buffer. The sg driver maintains one reserved buffer per file descriptor. The default size of the reserved buffer is 32 KB and it can be changed with the ioctl(SG_SET_RESERVED_SIZE). The mmap() system call only needs to be called once prior [18] to doing mmap-ed IO. For more details on the mmap() see Section 7.6. An application indicates that it wants mmap-ed on a SCSI request by setting the SG_FLAG_MMAP_IO value in 'flags'.

Since there is only reserved buffer per sg file descriptor then only one mmap-ed IO command can be active at one time. In order to perform command queuing with mmap-ed IO, an application will need to open() multiple file descriptors to the same SCSI device. With mmap-ed IO the various status values and the sense buffer (if required) are conveyed back to an application in the same fashion as normal ("indirect") IO.

Mmap-ed has very low per command latency since the reserved buffer mapping only needs to be done once per file descriptor. Also the reserved buffer is set up by the sg driver to aid the efficient construction of the internal scatter gather list used by the lower level (adapter) driver for DMA purposes. This tends to be more efficient than the user memory that direct IO requires the sg driver to process into an internal scatter gather list. So on both these counts, mmap-ed IO has the edge over direct IO.

11.1. /proc/scsi/sg/debug

This appendix explains the output from the /proc/scsi/sg/debug which is typically viewed by the command cat /proc/scsi/sg/debug. Below is the (slightly abridged) output while this command: sgp_dd if=/dev/sg0 of=/dev/null bs=512 is executing on the system. That sgp_dd command is using command queuing to read a disk (and the data is written to /dev/null which forgets it).

$ cat /proc/scsi/sg/debug dev_max(currently)=7 max_active_device=1 (origin 1) scsi_dma_free_sectors=416 sg_pool_secs_aval=320 def_reserved_size=32768 >>> device=sg0 scsi0 chan=0 id=0 lun=0 em=0 sg_tablesize=255 excl=0 FD(1): timeout=60000ms bufflen=65536 (res)sgat=2 low_dma=0 cmd_q=1 f_packid=1 k_orphan=0 closed=0 fin: id=3949312 blen=65536 dur=10ms sgat=2 op=0x28 act: id=3949440 blen=65536 t_o/elap=60000/10ms sgat=2 op=0x28 rb>> act: id=3949568 blen=65536 t_o/elap=60000/10ms sgat=2 op=0x28 act: id=3949696 blen=65536 t_o/elap=60000/0ms sgat=2 op=0x28

Those items output above that are significant to user applications are described below.

Broadly speaking the above output shows everything is going fine. Four SCSI READ(10) commands (SCSI opcode 0x28) for different ids are underway. Three commands are active while one is finished with its status and data read() and the request structure is pending deletion. The "id" corresponds to the pack_id given in the sg_io_hdr structure (or the sg_header structure). In the case if sgp_dd the pack_id value is the block number being given to the SCSI READ (or WRITE). You will notice the 4 ids are 128 apart.

The ">>>" line shows the sg device name followed by the linux scsi adapter, channel, scsi id and lun numbers. The "em=" argument indicates whether the driver emulates a SCSI HBA. The ide-scsi driver would set "em=1". The "sg_tablesize" is the maximum number of scatter gather elements supported by the adapter driver. The "excl=0" indicates no sg open() on this device is currently using the O_EXCL flag.

The next two lines starting with "FD(1)" supply data about the first (and only in this case) open file descriptor on /dev/sg0. The default timeout is 60 seconds however this is only significant if the sg_header interface is being used since the sg_io_hdr interface explicits sets the timeout on a per command basis. "bufflen=65536" is the reserved buffer size for this file descriptor. The "(res)sgat=2" indicates that this reserved buffer requires 2 scatter gather elements. The "low_dma" will be set to 1 for ISA HBAs indicating only the bottom 16 MB of RAM can be used for its kernel buffers. The "cmd_q=1" indicates command queuing is being allowed. The "f_packid=1" indicates the SG_SET_FORCE_PACK_ID mode is on. The "k_orphan" value is 1 in the rare cases when a SG_IO is interrupted while a SCSI command is "in flight". The "closed" value is 1 in the rare cases the file descriptor has been closed while a SCSI command is "in flight".

Each line indented with 5 spaces represents a SCSI command. The state of the command is either:

• prior: command hasn't been sent to mid level (rare)

• act: mid level (adapter driver or device) has command

• rcv: sg bottom half handler has received response to this command (awaiting read() or SG_IO ioctl to complete

• fin: SCSI response (and optionally data) has been or is being read but the command data structures have not been removed