4.9. Direct I/O
Direct I/O is a feature of the file system whereby file reads and writes go directly from the applications to the storage device, bypassing the operating system read and write caches. Direct I/O is used only by applications (such as databases) that manage their own caches.
An application invokes direct I/O by opening a file with the
O_DIRECT flag. Alternatively, GFS can attach a direct I/O attribute to a file, in which case direct I/O is used regardless of how the file is opened.
When a file is opened with
O_DIRECT, or when a GFS direct I/O attribute is attached to a file, all I/O operations must be done in block-size multiples of 512 bytes. The memory being read from or written to must also be 512-byte aligned.
Note
Performing I/O through a memory mapping and also via direct I/O to the same file at the same time may result in the direct I/O being failed with an I/O error. This occurs because the page invalidation required for the direct I/O can race with a page fault generated through the mapping. This is a problem only when the memory mapped I/O and the direct I/O are both performed on the same node as each other, and to the same file at the same point in time. A workaround is to use file locking to ensure that memory mapped (i.e., page faults) and direct I/O do not occur simultaneously on the same file.
The Oracle database, which is one of the main direct I/O using applications, does not memory map the files to which it uses direct I/O and thus is unaffected. In addition, writing to a file that is memory mapped will succeed, as expected, unless there are page faults in flight at that point in time. The
mmap system call on its own is safe when direct I/O is in use.
One of the following methods can be used to enable direct I/O on a file:
O_DIRECT- GFS file attribute
- GFS directory attribute
4.9.1. O_DIRECT
If an application uses the
O_DIRECT flag on an open() system call, direct I/O is used for the opened file.
To cause the
O_DIRECT flag to be defined with recent glibc libraries, define _GNU_SOURCE at the beginning of a source file before any includes, or define it on the cc line when compiling.
4.9.2. GFS File Attribute
The
gfs_tool command can be used to assign (set) a direct I/O attribute flag, directio, to a GFS file. The directio flag can also be cleared.
You can use the
gfs_tool stat filename to check what flags have been set for a GFS file. The output for this command includes a Flags: at the end of the display followed by a listing of the flags that are set for the indicated file.
Usage
Setting the
directio Flag
gfs_tool setflag directio File
Clearing the
directio Flag
gfs_tool clearflag directio FileFile- Specifies the file where the
directioflag is assigned.
Example
In this example, the command sets the
directio flag on the file named datafile in directory /mnt/gfs.
gfs_tool setflag directio /mnt/gfs/datafile
The following command checks whether the
directio flag is set for /mnt/gfs/datafile. The output has been elided to show only the relevant information.
[root@tng3-1 gfs]# gfs_tool stat /mnt/gfs/datafile
mh_magic = 0x01161970
...
Flags:
directio
4.9.3. GFS Directory Attribute
The
gfs_tool command can be used to assign (set) a direct I/O attribute flag, inherit_directio, to a GFS directory. Enabling the inherit_directio flag on a directory causes all newly created regular files in that directory to automatically inherit the directio flag. Also, the inherit_directio flag is inherited by any new subdirectories created in the directory. The inherit_directio flag can also be cleared.
Usage
Setting the
inherit_directio flag
gfs_tool setflag inherit_directio Directory
Clearing the
inherit_directio flag
gfs_tool clearflag inherit_directio DirectoryDirectory- Specifies the directory where the
inherit_directioflag is set.
Example
In this example, the command sets the
inherit_directio flag on the directory named /mnt/gfs/data.
gfs_tool setflag inherit_directio /mnt/gfs/data
This command displays the flags that have been set for the
/mnt/gfs/data directory. The full output has been truncated.
[root@tng3-1 gfs]# gfs_tool stat /mnt/gfs/data
...
Flags:
inherit_directio
