LLFIO  v2.00
Public Types | Public Member Functions | Protected Attributes | Friends | List of all members
llfio_v2_xxx::handle Class Reference

A native_handle_type which is managed by the lifetime of this object instance. More...

#include "handle.hpp"

Inheritance diagram for llfio_v2_xxx::handle:
llfio_v2_xxx::listening_socket_handle_buffer_types_injector< handle, byte_socket_handle > llfio_v2_xxx::awaitable_handle< IoHandleType > llfio_v2_xxx::byte_io_handle llfio_v2_xxx::byte_io_multiplexer llfio_v2_xxx::path_handle llfio_v2_xxx::process_handle llfio_v2_xxx::section_handle llfio_v2_xxx::symlink_handle

Public Types

enum class  mode : unsigned char {
  unchanged = 0 , none = 2 , attr_read = 4 , attr_write = 5 ,
  read = 6 , write , append = 9
}
 The behaviour of the handle: does it read, read and write, or atomic append? More...
 
enum class  creation : unsigned char {
  open_existing = 0 , only_if_not_exist , if_needed , truncate_existing ,
  always_new
}
 On opening, do we also create a new file or truncate an existing one? More...
 
enum class  caching : unsigned char {
  unchanged = 0 , none , only_metadata , reads ,
  reads_and_metadata , all , safety_barriers , temporary
}
 What i/o on the handle may complete immediately due to kernel caching. More...
 
using path_type = filesystem::path
 The path type used by this handle.
 
using extent_type = unsigned long long
 The file extent type used by this handle.
 
using size_type = size_t
 The memory extent type used by this handle.
 

Public Member Functions

 QUICKCPPLIB_BITFIELD_BEGIN_T (flag, uint16_t)
 Bitwise flags which can be specified. More...
 
constexpr handle ()
 Default constructor.
 
constexpr handle (native_handle_type h, flag flags=flag::none) noexcept
 Construct a handle from a supplied native handle.
 
 handle (const handle &)=delete
 No copy construction (use clone())
 
handle & operator= (const handle &o)=delete
 No copy assignment.
 
constexpr handle (handle &&o) noexcept
 Move the handle.
 
handle & operator= (handle &&o) noexcept
 Move assignment of handle.
 
void swap (handle &o) noexcept
 Swap with another instance.
 
virtual result< path_type > current_path () const noexcept
 
virtual result< void > close () noexcept
 Immediately close the native handle type managed by this handle.
 
result< handle > clone () const noexcept
 
virtual native_handle_type release () noexcept
 Release the native handle type managed by this handle.
 
bool is_valid () const noexcept
 True if the handle is valid (and usually open)
 
bool is_readable () const noexcept
 True if the handle is readable.
 
bool is_writable () const noexcept
 True if the handle is writable.
 
bool is_append_only () const noexcept
 True if the handle is append only.
 
virtual result< void > set_append_only (bool enable) noexcept
 EXTENSION: Changes whether this handle is append only or not. More...
 
bool is_multiplexable () const noexcept
 True if multiplexable.
 
bool is_nonblocking () const noexcept
 True if nonblocking.
 
bool is_seekable () const noexcept
 True if seekable.
 
bool requires_aligned_io () const noexcept
 True if requires aligned i/o.
 
bool is_kernel_handle () const noexcept
 True if native_handle() is a valid kernel handle.
 
bool is_regular () const noexcept
 True if a regular file or device.
 
bool is_directory () const noexcept
 True if a directory.
 
bool is_symlink () const noexcept
 True if a symlink.
 
bool is_pipe () const noexcept
 True if a pipe.
 
bool is_socket () const noexcept
 True if a socket.
 
bool is_multiplexer () const noexcept
 True if a multiplexer like BSD kqueues, Linux epoll or Windows IOCP.
 
bool is_process () const noexcept
 True if a process.
 
bool is_section () const noexcept
 True if a memory section.
 
bool is_allocation () const noexcept
 True if a memory allocation.
 
bool is_path () const noexcept
 True if a path or a directory.
 
bool is_tls_socket () const noexcept
 True if a TLS socket.
 
bool is_http_socket () const noexcept
 True if a HTTP socket.
 
caching kernel_caching () const noexcept
 Kernel cache strategy used by this handle.
 
bool are_reads_from_cache () const noexcept
 True if the handle uses the kernel page cache for reads.
 
bool are_writes_durable () const noexcept
 True if writes are safely on storage on completion.
 
bool are_safety_barriers_issued () const noexcept
 True if issuing safety fsyncs is on.
 
flag flags () const noexcept
 The flags this handle was opened with.
 
native_handle_type native_handle () const noexcept
 The native handle used by this handle.
 

Protected Attributes

union {
   native_handle_type   _v
 
   struct {
      intptr_t   _padding0_
 
      uint32_t   _padding1_
 
      flag   flags
 
      uint16_t   _padding2_
 
   }   _
 
}; 
 

Friends

class fs_handle
 
std::ostream & operator<< (std::ostream &s, const handle &v)
 

Detailed Description

A native_handle_type which is managed by the lifetime of this object instance.

Member Enumeration Documentation

◆ caching

enum llfio_v2_xxx::handle::caching : unsigned char
strong

What i/o on the handle may complete immediately due to kernel caching.

Enumerator
none 

No caching whatsoever, all reads and writes come from storage (i.e. O_DIRECT|O_SYNC). Align all i/o to 4Kb boundaries for this to work. disable_safety_barriers can be used here.

only_metadata 

Cache reads and writes of metadata but avoid caching data (O_DIRECT), thus i/o here does not affect other cached data for other handles. Align all i/o to 4Kb boundaries for this to work.

reads 

Cache reads only. Writes of data and metadata do not complete until reaching storage (O_SYNC). disable_safety_barriers can be used here.

reads_and_metadata 

Cache reads and writes of metadata, but writes of data do not complete until reaching storage (O_DSYNC). disable_safety_barriers can be used here.

all 

Cache reads and writes of data and metadata so they complete immediately, sending writes to storage at some point when the kernel decides (this is the default file system caching on a system).

safety_barriers 

Cache reads and writes of data and metadata so they complete immediately, but issue safety barriers at certain points. See documentation for disable_safety_barriers.

temporary 

Cache reads and writes of data and metadata so they complete immediately, only sending any updates to storage on last handle close in the system or if memory becomes tight as this file is expected to be temporary (Windows and FreeBSD only).

91  {
92  unchanged = 0,
93  none =
94  1, //!< No caching whatsoever, all reads and writes come from storage (i.e. <tt>O_DIRECT|O_SYNC</tt>). Align all i/o to 4Kb boundaries for this to work. <tt>disable_safety_barriers</tt> can be used here.
95  only_metadata =
96  2, //!< Cache reads and writes of metadata but avoid caching data (<tt>O_DIRECT</tt>), thus i/o here does not affect other cached data for other handles. Align all i/o to 4Kb boundaries for this to work.
97  reads =
98  3, //!< Cache reads only. Writes of data and metadata do not complete until reaching storage (<tt>O_SYNC</tt>). <tt>disable_safety_barriers</tt> can be used here.
99  reads_and_metadata =
100  5, //!< Cache reads and writes of metadata, but writes of data do not complete until reaching storage (<tt>O_DSYNC</tt>). <tt>disable_safety_barriers</tt> can be used here.
101  all =
102  6, //!< Cache reads and writes of data and metadata so they complete immediately, sending writes to storage at some point when the kernel decides (this is the default file system caching on a system).
103  safety_barriers =
104  7, //!< Cache reads and writes of data and metadata so they complete immediately, but issue safety barriers at certain points. See documentation for <tt>disable_safety_barriers</tt>.
105  temporary =
106  8 //!< Cache reads and writes of data and metadata so they complete immediately, only sending any updates to storage on last handle close in the system or if memory becomes tight as this file is expected to be temporary (Windows and FreeBSD only).
107  // NOTE: IF UPDATING THIS UPDATE THE std::ostream PRINTER BELOW!!!
108  };
llfio_v2_xxx::ip::none
@ none
No flags.
Definition: byte_socket_handle.hpp:235

◆ creation

enum llfio_v2_xxx::handle::creation : unsigned char
strong

On opening, do we also create a new file or truncate an existing one?

Enumerator
open_existing 

Filesystem entry must already exist.

only_if_not_exist 

Filesystem entry must NOT exist, and is atomically created by the success of this operation.

if_needed 

If filesystem entry exists that is used, else one is created.

truncate_existing 

Filesystem entry must already exist. It is atomically truncated on open, leaving creation date and unique identifier unmodified.

always_new 

If filesystem entry exists, it is atomically replaced with a new inode, else a new entry is created.

81  {
82  open_existing = 0, //!< Filesystem entry must already exist
83  only_if_not_exist, //!< Filesystem entry must NOT exist, and is atomically created by the success of this operation
84  if_needed, //!< If filesystem entry exists that is used, else one is created
85  truncate_existing, //!< Filesystem entry must already exist. It is atomically truncated on open, leaving creation date and unique identifier unmodified.
86  always_new //!< If filesystem entry exists, it is atomically replaced with a new inode, else a new entry is created.
87  // NOTE: IF UPDATING THIS UPDATE THE std::ostream PRINTER BELOW!!!
88  };

◆ mode

enum llfio_v2_xxx::handle::mode : unsigned char
strong

The behaviour of the handle: does it read, read and write, or atomic append?

Enumerator
none 

No ability to read or write anything, but can synchronise (SYNCHRONIZE or 0)

attr_read 

Ability to read attributes (FILE_READ_ATTRIBUTES|SYNCHRONIZE or O_RDONLY)

attr_write 

Ability to read and write attributes (FILE_READ_ATTRIBUTES|FILE_WRITE_ATTRIBUTES|SYNCHRONIZE or O_RDONLY)

read 

Ability to read (READ_CONTROL|FILE_READ_DATA|FILE_READ_ATTRIBUTES|FILE_READ_EA|SYNCHRONISE or O_RDONLY)

write 

Ability to read and write (READ_CONTROL|FILE_READ_DATA|FILE_READ_ATTRIBUTES|FILE_READ_EA|FILE_WRITE_DATA|FILE_WRITE_ATTRIBUTES|FILE_WRITE_EA|FILE_APPEND_DATA|SYNCHRONISE or O_RDWR)

append 

All mainstream OSs and CIFS guarantee this is atomic with respect to all other appenders (FILE_APPEND_DATA|SYNCHRONISE or O_APPEND)

68  {
69  unchanged = 0,
70  none = 2, //!< No ability to read or write anything, but can synchronise (SYNCHRONIZE or 0)
71  attr_read = 4, //!< Ability to read attributes (FILE_READ_ATTRIBUTES|SYNCHRONIZE or O_RDONLY)
72  attr_write = 5, //!< Ability to read and write attributes (FILE_READ_ATTRIBUTES|FILE_WRITE_ATTRIBUTES|SYNCHRONIZE or O_RDONLY)
73  read = 6, //!< Ability to read (READ_CONTROL|FILE_READ_DATA|FILE_READ_ATTRIBUTES|FILE_READ_EA|SYNCHRONISE or O_RDONLY)
74  write =
75  7, //!< Ability to read and write (READ_CONTROL|FILE_READ_DATA|FILE_READ_ATTRIBUTES|FILE_READ_EA|FILE_WRITE_DATA|FILE_WRITE_ATTRIBUTES|FILE_WRITE_EA|FILE_APPEND_DATA|SYNCHRONISE or O_RDWR)
76  append = 9 //!< All mainstream OSs and CIFS guarantee this is atomic with respect to all other appenders (FILE_APPEND_DATA|SYNCHRONISE or O_APPEND)
77  // NOTE: IF UPDATING THIS UPDATE THE std::ostream PRINTER BELOW!!!
78  };
llfio_v2_xxx::write
byte_io_handle::io_result< byte_io_handle::const_buffers_type > write(byte_io_handle &self, byte_io_handle::io_request< byte_io_handle::const_buffers_type > reqs, deadline d=deadline()) noexcept
Write data to the open handle.
Definition: byte_io_handle.hpp:681
llfio_v2_xxx::read
byte_io_handle::io_result< byte_io_handle::buffers_type > read(byte_io_handle &self, byte_io_handle::io_request< byte_io_handle::buffers_type > reqs, deadline d=deadline()) noexcept
Read data from the open handle.
Definition: byte_io_handle.hpp:654

Member Function Documentation

◆ clone()

result<handle> llfio_v2_xxx::handle::clone ( ) const
inlinenoexcept

Clone this handle (copy constructor is disabled to avoid accidental copying)

Errors returnable\n Any of the values POSIX dup() or DuplicateHandle() can return.

◆ current_path()

virtual result<path_type> llfio_v2_xxx::handle::current_path ( ) const
inlinevirtualnoexcept

Returns the current path of the open handle as said by the operating system. Note that you are NOT guaranteed that any path refreshed bears any resemblance to the original, some operating systems will return some different path which still reaches the same inode via some other route e.g. hardlinks, dereferenced symbolic links, etc. Windows and Linux correctly track changes to the specific path the handle was opened with, not getting confused by other hard links. MacOS nearly gets it right, but under some circumstances e.g. renaming may switch to a different hard link's path which is almost certainly a bug.

If LLFIO was not able to determine the current path for this open handle e.g. the inode has been unlinked, it returns an empty path. Be aware that FreeBSD can return an empty (deleted) path for file inodes no longer cached by the kernel path cache, LLFIO cannot detect the difference. FreeBSD will also return any path leading to the inode if it is hard linked. FreeBSD does implement path retrieval for directory inodes correctly however, and see algorithm::cached_parent_handle_adapter<T> for a handle adapter which makes use of that.

On Linux if /proc is not mounted, this call fails with an error. All APIs in LLFIO which require the use of current_path() can be told to not use it e.g. flag::disable_safety_unlinks. It is up to you to detect if current_path() is not working, and to change how you call LLFIO appropriately.

On Windows, you will almost certainly get back a path of the form \!!\Device\HarddiskVolume10\Users\ned\.... See path_view for what all the path prefix sequences mean, but to summarise the \!!\ prefix is LLFIO-only and will not be accepted by other Windows APIs. Pass LLFIO derived paths through the function to_win32_path() to Win32-ise them. This function is also available on Linux where it does nothing, so you can use it in portable code.

Warning
This call is expensive, it always asks the kernel for the current path, and no checking is done to ensure what the kernel returns is accurate or even sensible. Be aware that despite these precautions, paths are unstable and can change randomly at any moment. Most code written to use absolute file systems paths is racy, so don't do it, use path_handle to fix a base location on the file system and work from that anchor instead!
Memory Allocations\n At least one malloc for the path_type, likely several more.
See also
algorithm::cached_parent_handle_adapter<T> which overrides this with an implementation based on retrieving the current path of a cached handle to the parent directory. On platforms with instability or failure to retrieve the correct current path for regular files, the cached parent handle adapter works around the problem by taking advantage of directory inodes not having the same instability problems on any platform.

Reimplemented in llfio_v2_xxx::symlink_handle, and llfio_v2_xxx::process_handle.

◆ QUICKCPPLIB_BITFIELD_BEGIN_T()

llfio_v2_xxx::handle::QUICKCPPLIB_BITFIELD_BEGIN_T ( flag  ,
uint16_t   
)
inline

Bitwise flags which can be specified.

< No flags

Unlinks the file on handle close. On POSIX, this simply unlinks whatever is pointed to by path() upon the call of close() if and only if the inode matches. On Windows, if you are on Windows 10 1709 or later, exactly the same thing occurs. If on previous editions of Windows, the file entry does not disappears but becomes unavailable for anyone else to open with an errc::resource_unavailable_try_again error return. Because this is confusing, unless the win_disable_unlink_emulation flag is also specified, this POSIX behaviour is somewhat emulated by LLFIO on older Windows by renaming the file to a random name on close() causing it to appear to have been unlinked immediately.

Some kernel caching modes have unhelpfully inconsistent behaviours in getting your data onto storage, so by default unless this flag is specified LLFIO adds extra fsyncs to the following operations for the caching modes specified below: truncation of file length either explicitly or during file open. closing of the handle either explicitly or in the destructor.

Additionally on Linux only to prevent loss of file metadata: On the parent directory whenever a file might have been created. On the parent directory on file close.

This only occurs for these kernel caching modes: caching::none caching::reads caching::reads_and_metadata caching::safety_barriers

file_handle::unlink() could accidentally delete the wrong file if someone has renamed the open file handle since the time it was opened. To prevent this occuring, where the OS doesn't provide race free unlink-by-open-handle we compare the inode of the path we are about to unlink with that of the open handle before unlinking.

Warning
This does not prevent races where in between the time of checking the inode and executing the unlink a third party changes the item about to be unlinked. Only operating systems with a true race-free unlink syscall are race free.

Ask the OS to disable prefetching of data. This can improve random i/o performance.

Ask the OS to maximise prefetching of data, possibly prefetching the entire file into kernel cache. This can improve sequential i/o performance.

< See the documentation for unlink_on_first_close

Microsoft Windows NTFS, having been created in the late 1980s, did not originally implement extents-based storage and thus could only represent sparse files via efficient compression of intermediate zeros. With NTFS v3.0 (Microsoft Windows 2000), a proper extents-based on-storage representation was added, thus allowing only 64Kb extent chunks written to be stored irrespective of whatever the maximum file extent was set to.

For various historical reasons, extents-based storage is disabled by default in newly created files on NTFS, unlike in almost every other major filing system. You have to explicitly "opt in" to extents-based storage.

As extents-based storage is nearly cost free on NTFS, LLFIO by default opts in to extents-based storage for any empty file it creates. If you don't want this, you can specify this flag to prevent that happening.

Filesystems tend to be embarrassingly parallel for operations performed to different inodes. Where LLFIO performs i/o to multiple inodes at a time, it will use OpenMP or the Parallelism or Concurrency standard library extensions to usually complete the operation in constant rather than linear time. If you don't want this default, you can disable default using this flag.

Microsoft Windows NTFS has the option, when creating a directory, to set whether leafname lookup will be case sensitive. This is the only way of getting exact POSIX semantics on Windows without resorting to editing the system registry, however it also affects all code doing lookups within that directory, so we must default it to off.

Create the handle in a way where i/o upon it can be multiplexed with other i/o on the same initiating thread of execution i.e. you can perform more than one read concurrently, without using threads. The blocking operations .read() and .write() may have to use a less efficient, but cancellable, blocking implementation for handles created in this way. On Microsoft Windows, this creates handles with OVERLAPPED semantics. On POSIX, this creates handles with nonblocking semantics for non-file handles such as pipes and sockets, however for file, directory and symlink handles it does not set nonblocking, as it is non-portable.

< Using insane POSIX byte range locks

< This is an inode created with no representation on the filing system

110  {
111  none = uint16_t(0), //!< No flags
112  /*! Unlinks the file on handle close. On POSIX, this simply unlinks whatever is pointed
113  to by `path()` upon the call of `close()` if and only if the inode matches. On Windows,
114  if you are on Windows 10 1709 or later, exactly the same thing occurs. If on previous
115  editions of Windows, the file entry does not disappears but becomes unavailable for
116  anyone else to open with an `errc::resource_unavailable_try_again` error return. Because this is confusing, unless the
117  `win_disable_unlink_emulation` flag is also specified, this POSIX behaviour is
118  somewhat emulated by LLFIO on older Windows by renaming the file to a random name on `close()`
119  causing it to appear to have been unlinked immediately.
120  */
121  unlink_on_first_close = uint16_t(1U << 0U),
122 
123  /*! Some kernel caching modes have unhelpfully inconsistent behaviours
124  in getting your data onto storage, so by default unless this flag is
125  specified LLFIO adds extra fsyncs to the following operations for the
126  caching modes specified below:
127  * truncation of file length either explicitly or during file open.
128  * closing of the handle either explicitly or in the destructor.
129 
130  Additionally on Linux only to prevent loss of file metadata:
131  * On the parent directory whenever a file might have been created.
132  * On the parent directory on file close.
133 
134  This only occurs for these kernel caching modes:
135  * caching::none
136  * caching::reads
137  * caching::reads_and_metadata
138  * caching::safety_barriers
139  */
140  disable_safety_barriers = uint16_t(1U << 2U),
141  /*! `file_handle::unlink()` could accidentally delete the wrong file if someone has
142  renamed the open file handle since the time it was opened. To prevent this occuring,
143  where the OS doesn't provide race free unlink-by-open-handle we compare the inode of
144  the path we are about to unlink with that of the open handle before unlinking.
145  \warning This does not prevent races where in between the time of checking the inode
146  and executing the unlink a third party changes the item about to be unlinked. Only
147  operating systems with a true race-free unlink syscall are race free.
148  */
149  disable_safety_unlinks = uint16_t(1U << 3U),
150  /*! Ask the OS to disable prefetching of data. This can improve random
151  i/o performance.
152  */
153  disable_prefetching = uint16_t(1U << 4U),
154  /*! Ask the OS to maximise prefetching of data, possibly prefetching the entire file
155  into kernel cache. This can improve sequential i/o performance.
156  */
157  maximum_prefetching = uint16_t(1U << 5U),
158 
159  win_disable_unlink_emulation = uint16_t(1U << 9U), //!< See the documentation for `unlink_on_first_close`
160  /*! Microsoft Windows NTFS, having been created in the late 1980s, did not originally
161  implement extents-based storage and thus could only represent sparse files via
162  efficient compression of intermediate zeros. With NTFS v3.0 (Microsoft Windows 2000),
163  a proper extents-based on-storage representation was added, thus allowing only 64Kb
164  extent chunks written to be stored irrespective of whatever the maximum file extent
165  was set to.
166 
167  For various historical reasons, extents-based storage is disabled by default in newly
168  created files on NTFS, unlike in almost every other major filing system. You have to
169  explicitly "opt in" to extents-based storage.
170 
171  As extents-based storage is nearly cost free on NTFS, LLFIO by default opts in to
172  extents-based storage for any empty file it creates. If you don't want this, you
173  can specify this flag to prevent that happening.
174  */
175  win_disable_sparse_file_creation = uint16_t(1U << 10U),
176  /*! Filesystems tend to be embarrassingly parallel for operations performed to different
177  inodes. Where LLFIO performs i/o to multiple inodes at a time, it will use OpenMP or
178  the Parallelism or Concurrency standard library extensions to usually complete the
179  operation in constant rather than linear time. If you don't want this default, you can
180  disable default using this flag.
181  */
182  disable_parallelism = uint16_t(1U << 11U),
183  /*! Microsoft Windows NTFS has the option, when creating a directory, to set whether
184  leafname lookup will be case sensitive. This is the only way of getting exact POSIX
185  semantics on Windows without resorting to editing the system registry, however it also
186  affects all code doing lookups within that directory, so we must default it to off.
187  */
188  win_create_case_sensitive_directory = uint16_t(1U << 12U),
189 
190  /*! Create the handle in a way where i/o upon it can be multiplexed with other i/o
191  on the same initiating thread of execution i.e. you can perform more than one read
192  concurrently, without using threads. The blocking operations `.read()` and `.write()`
193  may have to use a less efficient, but cancellable, blocking implementation for handles created
194  in this way. On Microsoft Windows, this creates handles with `OVERLAPPED` semantics.
195  On POSIX, this creates handles with nonblocking semantics for non-file handles such
196  as pipes and sockets, however for file, directory and symlink handles it does not set
197  nonblocking, as it is non-portable.
198  */
199  multiplexable = uint16_t(1U << 13U),
200 
201  // NOTE: IF UPDATING THIS UPDATE THE std::ostream PRINTER BELOW!!!
202 
203  byte_lock_insanity = uint16_t(1U << 14U), //!< Using insane POSIX byte range locks
204  anonymous_inode = uint16_t(1U << 15U) //!< This is an inode created with no representation on the filing system
205  } QUICKCPPLIB_BITFIELD_END(flag)

◆ set_append_only()

virtual result<void> llfio_v2_xxx::handle::set_append_only ( bool  enable)
inlinevirtualnoexcept

EXTENSION: Changes whether this handle is append only or not.

Warning
On Windows this is implemented as a bit of a hack to make it fast like on POSIX, so make sure you open the handle for read/write originally. Note unlike on POSIX the append_only disposition will be the only one toggled, seekable and readable will remain turned on.
Errors returnable\n Whatever POSIX fcntl() returns. On Windows nothing is changed on the handle.
Memory Allocations\n No memory allocation.

Reimplemented in llfio_v2_xxx::process_handle.

The documentation for this class was generated from the following file: