LLFIO  v2.00
llfio_v2_xxx::directory_handle Class Reference

A handle to a directory which can be enumerated. More...

#include "directory_handle.hpp"

Inheritance diagram for llfio_v2_xxx::directory_handle:
llfio_v2_xxx::path_handle llfio_v2_xxx::fs_handle llfio_v2_xxx::handle

Classes

struct  buffers_type
 
struct  io_request
 The i/o request type used by this handle. More...
 

Public Types

enum class  filter : uint8_t { none , fastdeleted }
 How to do deleted file elimination on Windows. More...
 
using path_type = path_handle::path_type
 
using extent_type = path_handle::extent_type
 
using size_type = path_handle::size_type
 
using mode = path_handle::mode
 
using creation = path_handle::creation
 
using caching = path_handle::caching
 
using flag = path_handle::flag
 
using dev_t = fs_handle::dev_t
 
using ino_t = fs_handle::ino_t
 
using path_view_type = fs_handle::path_view_type
 
using buffer_type = directory_entry
 The buffer type used by this handle, which is a directory_entry
 
using const_buffer_type = directory_entry
 The const buffer type used by this handle, which is a directory_entry
 
using _flags_type = flags
 
using unique_id_type = QUICKCPPLIB_NAMESPACE::integers128::uint128
 The unique identifier type used by this handle.
 
using unique_id_type_hasher = QUICKCPPLIB_NAMESPACE::integers128::uint128_hasher
 A hasher for the unique identifier type used by this handle.
 

Public Member Functions

 QUICKCPPLIB_BITFIELD_BEGIN_T (flags, uint8_t)
 Flags for how to enumerate directory entries. More...
 
constexpr directory_handle ()
 Default constructor.
 
constexpr directory_handle (native_handle_type h, dev_t devid, ino_t inode, flag flags)
 Construct a directory_handle from a supplied native path_handle.
 
constexpr directory_handle (native_handle_type h, flag flags)
 Construct a directory_handle from a supplied native path_handle.
 
constexpr directory_handle (directory_handle &&o) noexcept
 Implicit move construction of directory_handle permitted.
 
 directory_handle (const directory_handle &)=delete
 No copy construction (use clone())
 
constexpr directory_handle (handle &&o, dev_t devid, ino_t inode) noexcept
 Explicit conversion from handle permitted.
 
directory_handleoperator= (directory_handle &&o) noexcept
 Move assignment of directory_handle permitted.
 
directory_handleoperator= (const directory_handle &)=delete
 No copy assignment.
 
void swap (directory_handle &o) noexcept
 Swap with another instance.
 
virtual result< void > close () noexcept override
 Immediately close the native handle type managed by this handle.
 
virtual result< directory_handlereopen (mode mode_=mode::unchanged, caching caching_=caching::unchanged, deadline d=std::chrono::seconds(30)) const noexcept
 
template<class... Args>
bool try_reopen (Args &&... args) noexcept
 
template<class... Args, class Rep , class Period >
bool try_reopen_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
 
template<class... Args, class Clock , class Duration >
bool try_reopen_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
 
result< path_handleclone_to_path_handle () const noexcept
 
virtual result< buffers_typeread (io_request< buffers_type > req, deadline d=std::chrono::seconds(30)) const noexcept
 
void swap (path_handle &o) noexcept
 Swap with another instance.
 
void swap (handle &o) noexcept
 Swap with another instance.
 
result< bool > exists (path_view_type path) const noexcept
 
flag flags () const noexcept
 The flags this handle was opened with.
 
 QUICKCPPLIB_BITFIELD_BEGIN_T (flag, uint16_t)
 Bitwise flags which can be specified. More...
 
virtual result< path_type > current_path () const noexcept
 
result< handleclone () const noexcept
 
virtual native_handle_type release () noexcept
 Release the native handle type managed by this handle.
 
bool is_valid () const noexcept
 True if the handle is valid (and usually open)
 
bool is_readable () const noexcept
 True if the handle is readable.
 
bool is_writable () const noexcept
 True if the handle is writable.
 
bool is_append_only () const noexcept
 True if the handle is append only.
 
virtual result< void > set_append_only (bool enable) noexcept
 EXTENSION: Changes whether this handle is append only or not. More...
 
bool is_multiplexable () const noexcept
 True if multiplexable.
 
bool is_nonblocking () const noexcept
 True if nonblocking.
 
bool is_seekable () const noexcept
 True if seekable.
 
bool requires_aligned_io () const noexcept
 True if requires aligned i/o.
 
bool is_kernel_handle () const noexcept
 True if native_handle() is a valid kernel handle.
 
bool is_regular () const noexcept
 True if a regular file or device.
 
bool is_directory () const noexcept
 True if a directory.
 
bool is_symlink () const noexcept
 True if a symlink.
 
bool is_pipe () const noexcept
 True if a pipe.
 
bool is_socket () const noexcept
 True if a socket.
 
bool is_multiplexer () const noexcept
 True if a multiplexer like BSD kqueues, Linux epoll or Windows IOCP.
 
bool is_process () const noexcept
 True if a process.
 
bool is_section () const noexcept
 True if a memory section.
 
bool is_allocation () const noexcept
 True if a memory allocation.
 
bool is_path () const noexcept
 True if a path or a directory.
 
bool is_tls_socket () const noexcept
 True if a TLS socket.
 
bool is_http_socket () const noexcept
 True if a HTTP socket.
 
caching kernel_caching () const noexcept
 Kernel cache strategy used by this handle.
 
bool are_reads_from_cache () const noexcept
 True if the handle uses the kernel page cache for reads.
 
bool are_writes_durable () const noexcept
 True if writes are safely on storage on completion.
 
bool are_safety_barriers_issued () const noexcept
 True if issuing safety fsyncs is on.
 
native_handle_type native_handle () const noexcept
 The native handle used by this handle.
 
dev_t st_dev () const noexcept
 Unless flag::disable_safety_unlinks is set, the device id of the file when opened.
 
ino_t st_ino () const noexcept
 Unless flag::disable_safety_unlinks is set, the inode of the file when opened. When combined with st_dev(), forms a unique identifer on this system.
 
unique_id_type unique_id () const noexcept
 A unique identifier for this handle across the entire system. Can be used in hash tables etc.
 
virtual result< path_handleparent_path_handle (deadline d=std::chrono::seconds(30)) const noexcept
 Obtain a handle to the path currently containing this handle's file entry. More...
 
template<class... Args>
bool try_parent_path_handle (Args &&... args) noexcept
 
template<class... Args, class Rep , class Period >
bool try_parent_path_handle_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
 
template<class... Args, class Clock , class Duration >
bool try_parent_path_handle_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
 
virtual result< void > relink (const path_handle &base, path_view_type path, bool atomic_replace=true, deadline d=std::chrono::seconds(30)) noexcept
 Relinks the current path of this open handle to the new path specified. If atomic_replace is true, the relink atomically and silently replaces any item at the new path specified. This operation is both atomic and matching POSIX behaviour even on Microsoft Windows where no Win32 API can match POSIX semantics. More...
 
template<class... Args>
bool try_relink (Args &&... args) noexcept
 
template<class... Args, class Rep , class Period >
bool try_relink_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
 
template<class... Args, class Clock , class Duration >
bool try_relink_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
 
virtual result< void > link (const path_handle &base, path_view_type path, deadline d=std::chrono::seconds(30)) noexcept
 Links the inode referred to by this open handle to the path specified. The current path of this open handle is not changed, unless it has no current path due to being unlinked. More...
 
template<class... Args>
bool try_link (Args &&... args) noexcept
 
template<class... Args, class Rep , class Period >
bool try_link_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
 
template<class... Args, class Clock , class Duration >
bool try_link_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
 
virtual result< void > unlink (deadline d=std::chrono::seconds(30)) noexcept
 Unlinks the current path of this open handle, causing its entry to immediately disappear from the filing system. More...
 
template<class... Args>
bool try_unlink (Args &&... args) noexcept
 
template<class... Args, class Rep , class Period >
bool try_unlink_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
 
template<class... Args, class Clock , class Duration >
bool try_unlink_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
 
virtual result< span< path_view_component > > list_extended_attributes (span< byte > tofill) const noexcept
 Fill the supplied buffer with the names of all extended attributes set on this file or directory, returning a span of path view components. More...
 
virtual result< span< byte > > get_extended_attribute (span< byte > tofill, path_view_component name) const noexcept
 Retrieve the value of an extended attribute set on this file or directory. More...
 
virtual result< void > set_extended_attribute (path_view_component name, span< const byte > value) noexcept
 Sets the value of an extended attribute on this file or directory. More...
 
virtual result< void > remove_extended_attribute (path_view_component) noexcept
 Removes the extended attribute set on this file or directory. More...
 
result< size_t > copy_extended_attributes (const fs_handle &src, bool replace_all_local_attributes=false) noexcept
 Copies the extended attributes from one entity to another, optionally replacing all the extended attributes on the destination. More...
 

Static Public Member Functions

static result< directory_handledirectory (const path_handle &base, path_view_type path, mode _mode=mode::read, creation _creation=creation::open_existing, caching _caching=caching::all, flag flags=flag::none) noexcept
 
static result< directory_handleuniquely_named_directory (const path_handle &dirpath, mode _mode=mode::write, caching _caching=caching::temporary, flag flags=flag::none) noexcept
 
static result< directory_handletemp_directory (path_view_type name=path_view_type(), mode _mode=mode::write, creation _creation=creation::if_needed, caching _caching=caching::all, flag flags=flag::none) noexcept
 
static result< bool > exists (const path_handle &base, path_view_type path) noexcept
 This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.
 
static result< path_handlepath (const path_handle &base, path_view_type path) noexcept
 
static result< path_handlepath (path_view_type _path) noexcept
 This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.
 

Protected Member Functions

result< void > _fetch_inode () const noexcept
 Fill in _devid and _inode from the handle via fstat()
 

Protected Attributes

union {
   native_handle_type   _v
 
   struct {
      intptr_t   _padding0_
 
      uint32_t   _padding1_
 
      flag   flags
 
      uint16_t   _padding2_
 
   }   _
 
}; 
 
dev_t _devid {0}
 
ino_t _inode {0}
 

Detailed Description

A handle to a directory which can be enumerated.

Note
For good performance, make sure you reuse buffers_type across calls to read(), including across different instances of directory_handle. This is because a kernel buffer is allocated within the first use of a buffers_type in a read(), so reusing buffers_type will save on an allocation-free cycle per directory enumeration.

Member Enumeration Documentation

◆ filter

How to do deleted file elimination on Windows.

Enumerator
none 

Do no filtering at all.

fastdeleted 

For Windows without POSIX delete semantics, filter out LLFIO deleted files based on their filename (fast and fairly reliable)

199  {
200  none, //!< Do no filtering at all
201  fastdeleted //!< For Windows without POSIX delete semantics, filter out LLFIO deleted files based on their filename (fast and fairly reliable)
202  };
@ none
No flags.
Definition: byte_socket_handle.hpp:235

Member Function Documentation

◆ clone()

result<handle> llfio_v2_xxx::handle::clone ( ) const
inlinenoexceptinherited

Clone this handle (copy constructor is disabled to avoid accidental copying)

Errors returnable\n Any of the values POSIX dup() or DuplicateHandle() can return.

◆ clone_to_path_handle()

result<path_handle> llfio_v2_xxx::directory_handle::clone_to_path_handle ( ) const
inlinenoexcept

Return a copy of this directory handle, but as a path handle.

Errors returnable\n Any of the values POSIX dup() or DuplicateHandle() can return.
Memory Allocations\n On POSIX, we must loop calling current_path() and
trying to open the path returned. Thus many allocations may occur.

◆ copy_extended_attributes()

result<size_t> llfio_v2_xxx::fs_handle::copy_extended_attributes ( const fs_handle src,
bool  replace_all_local_attributes = false 
)
inlinenoexceptinherited

Copies the extended attributes from one entity to another, optionally replacing all the extended attributes on the destination.

This convenience function is implemented using the APIs above, and therefore is racy with respect to concurrent users. If you specifiy an invalid source with replace_all_local_attributes = true, then this is a convenient way to remove all extended attributes on the local inode.

Note
This function uses 130Kb of stack and cannot handle attribute values larger than 64Kb.
446  {
447  auto &h = _get_handle();
448  (void) h;
449  LLFIO_LOG_FUNCTION_CALL(&h);
450  byte buffer[65536 + 4096];
451  if(replace_all_local_attributes)
452  {
453  for(;;)
454  {
455  OUTCOME_TRY(auto &&attribs, list_extended_attributes(buffer));
456  if(attribs.empty())
457  {
458  break;
459  }
460  for(auto &attrib : attribs)
461  {
462  OUTCOME_TRY(remove_extended_attribute(attrib));
463  }
464  }
465  }
466  if(src._get_handle().is_valid())
467  {
468  OUTCOME_TRY(auto &&attribs, src.list_extended_attributes(buffer));
469  for(auto &attrib : attribs)
470  {
471  byte buffer2[65536];
472  OUTCOME_TRY(auto &&value, src.get_extended_attribute(buffer2, attrib));
473  OUTCOME_TRY(set_extended_attribute(attrib, value));
474  }
475  }
476  return success();
477  }
virtual result< span< path_view_component > > list_extended_attributes(span< byte > tofill) const noexcept
Fill the supplied buffer with the names of all extended attributes set on this file or directory,...
virtual result< void > set_extended_attribute(path_view_component name, span< const byte > value) noexcept
Sets the value of an extended attribute on this file or directory.
virtual result< void > remove_extended_attribute(path_view_component) noexcept
Removes the extended attribute set on this file or directory.

◆ current_path()

virtual result<path_type> llfio_v2_xxx::handle::current_path ( ) const
inlinevirtualnoexceptinherited

Returns the current path of the open handle as said by the operating system. Note that you are NOT guaranteed that any path refreshed bears any resemblance to the original, some operating systems will return some different path which still reaches the same inode via some other route e.g. hardlinks, dereferenced symbolic links, etc. Windows and Linux correctly track changes to the specific path the handle was opened with, not getting confused by other hard links. MacOS nearly gets it right, but under some circumstances e.g. renaming may switch to a different hard link's path which is almost certainly a bug.

If LLFIO was not able to determine the current path for this open handle e.g. the inode has been unlinked, it returns an empty path. Be aware that FreeBSD can return an empty (deleted) path for file inodes no longer cached by the kernel path cache, LLFIO cannot detect the difference. FreeBSD will also return any path leading to the inode if it is hard linked. FreeBSD does implement path retrieval for directory inodes correctly however, and see algorithm::cached_parent_handle_adapter<T> for a handle adapter which makes use of that.

On Linux if /proc is not mounted, this call fails with an error. All APIs in LLFIO which require the use of current_path() can be told to not use it e.g. flag::disable_safety_unlinks. It is up to you to detect if current_path() is not working, and to change how you call LLFIO appropriately.

On Windows, you will almost certainly get back a path of the form \!!\Device\HarddiskVolume10\Users\ned\.... See path_view for what all the path prefix sequences mean, but to summarise the \!!\ prefix is LLFIO-only and will not be accepted by other Windows APIs. Pass LLFIO derived paths through the function to_win32_path() to Win32-ise them. This function is also available on Linux where it does nothing, so you can use it in portable code.

Warning
This call is expensive, it always asks the kernel for the current path, and no checking is done to ensure what the kernel returns is accurate or even sensible. Be aware that despite these precautions, paths are unstable and can change randomly at any moment. Most code written to use absolute file systems paths is racy, so don't do it, use path_handle to fix a base location on the file system and work from that anchor instead!
Memory Allocations\n At least one malloc for the path_type, likely several more.
See also
algorithm::cached_parent_handle_adapter<T> which overrides this with an implementation based on retrieving the current path of a cached handle to the parent directory. On platforms with instability or failure to retrieve the correct current path for regular files, the cached parent handle adapter works around the problem by taking advantage of directory inodes not having the same instability problems on any platform.

Reimplemented in llfio_v2_xxx::symlink_handle, and llfio_v2_xxx::process_handle.

◆ directory()

static result<directory_handle> llfio_v2_xxx::directory_handle::directory ( const path_handle base,
path_view_type  path,
mode  _mode = mode::read,
creation  _creation = creation::open_existing,
caching  _caching = caching::all,
flag  flags = flag::none 
)
inlinestaticnoexcept

Create a handle opening access to a directory on path.

Errors returnable\n Any of the values POSIX open() or CreateFile() can return.

◆ exists()

result<bool> llfio_v2_xxx::path_handle::exists ( path_view_type  path) const
inlinenoexceptinherited

Returns whether a file entry exists more efficiently that opening and closing a file_handle. Note that this can be a rich source of TOCTOU security attacks! Be aware that symbolic links are NOT dereferenced, so a subsequent file handle open may fail.

◆ get_extended_attribute()

virtual result<span<byte> > llfio_v2_xxx::fs_handle::get_extended_attribute ( span< byte >  tofill,
path_view_component  name 
) const
inlinevirtualnoexceptinherited

Retrieve the value of an extended attribute set on this file or directory.

Note
On Windows, this is the list of alternate streams on a file, NOT NTFS extended attributes.

◆ link()

virtual result<void> llfio_v2_xxx::fs_handle::link ( const path_handle base,
path_view_type  path,
deadline  d = std::chrono::seconds(30) 
)
inlinevirtualnoexceptinherited

Links the inode referred to by this open handle to the path specified. The current path of this open handle is not changed, unless it has no current path due to being unlinked.

Warning
Some operating systems provide a race free syscall for linking an open handle to a new location (Linux, Windows). On all other operating systems this call is racy and can result in the wrong inode being linked. Note that unless flag::disable_safety_unlinks is set, this implementation opens a path_handle to the source containing directory first, then checks before linking that the item about to be hard linked has the same inode as the open file handle. It will retry this matching until success until the deadline given. This should prevent most unmalicious accidental loss of data.
Parameters
baseBase for any relative path.
pathThe relative or absolute new path to hard link to.
dThe deadline by which the matching of the containing directory to the open handle's inode must succeed, else errc::timed_out will be returned.
Memory Allocations\n Except on platforms with race free syscalls for renaming open handles (Windows), calls
current_path() via parent_path_handle() and thus is both expensive and calls malloc many times.

◆ list_extended_attributes()

virtual result<span<path_view_component> > llfio_v2_xxx::fs_handle::list_extended_attributes ( span< byte >  tofill) const
inlinevirtualnoexceptinherited

Fill the supplied buffer with the names of all extended attributes set on this file or directory, returning a span of path view components.

Note that this routine is a very thin wrap of listxattr() on POSIX and NtQueryInformationFile() on Windows. If the supplied buffer is too small, the syscall typically returns failure rather than do a partial fill. Most implementations do not support more than 64Kb of extended attribute information per inode so maybe 70Kb is a safe default (to account for the return value storage), however properly written code will detect the buffer being too small and will auto-expand it until success.

Note
On Windows, this is the list of alternate streams on a file, NOT NTFS extended attributes.
Race Guarantees\n The list of extended attributes is fetched in a single syscall. This may be an
atomically consistent snapshot.

◆ parent_path_handle()

virtual result<path_handle> llfio_v2_xxx::fs_handle::parent_path_handle ( deadline  d = std::chrono::seconds(30)) const
inlinevirtualnoexceptinherited

Obtain a handle to the path currently containing this handle's file entry.

Warning
This call is racy and can result in the wrong path handle being returned. Note that unless flag::disable_safety_unlinks is set, this implementation opens a path_handle to the source containing directory, then checks if the file entry within has the same inode as the open file handle. It will retry this matching until success until the deadline given.
Memory Allocations\n Calls current_path() and thus is both expensive and calls malloc many times.
See also
algorithm::cached_parent_handle_adapter<T> which overrides this with a zero cost implementation, thus making unlinking and relinking very considerably quicker.

◆ path()

static result<path_handle> llfio_v2_xxx::path_handle::path ( const path_handle base,
path_view_type  path 
)
inlinestaticnoexceptinherited

Create a path handle opening access to some location on the filing system. Some operating systems provide a particularly lightweight method of doing this (Linux: O_PATH, Windows: no access perms) which is much faster than opening a directory. For other systems, we open a directory with read only permissions.

Errors returnable\n Any of the values POSIX open() or CreateFile() can return.

◆ QUICKCPPLIB_BITFIELD_BEGIN_T() [1/2]

llfio_v2_xxx::handle::QUICKCPPLIB_BITFIELD_BEGIN_T ( flag  ,
uint16_t   
)
inlineinherited

Bitwise flags which can be specified.

< No flags

Unlinks the file on handle close. On POSIX, this simply unlinks whatever is pointed to by path() upon the call of close() if and only if the inode matches. On Windows, if you are on Windows 10 1709 or later, exactly the same thing occurs. If on previous editions of Windows, the file entry does not disappears but becomes unavailable for anyone else to open with an errc::resource_unavailable_try_again error return. Because this is confusing, unless the win_disable_unlink_emulation flag is also specified, this POSIX behaviour is somewhat emulated by LLFIO on older Windows by renaming the file to a random name on close() causing it to appear to have been unlinked immediately.

Some kernel caching modes have unhelpfully inconsistent behaviours in getting your data onto storage, so by default unless this flag is specified LLFIO adds extra fsyncs to the following operations for the caching modes specified below: truncation of file length either explicitly or during file open. closing of the handle either explicitly or in the destructor.

Additionally on Linux only to prevent loss of file metadata: On the parent directory whenever a file might have been created. On the parent directory on file close.

This only occurs for these kernel caching modes: caching::none caching::reads caching::reads_and_metadata caching::safety_barriers

file_handle::unlink() could accidentally delete the wrong file if someone has renamed the open file handle since the time it was opened. To prevent this occuring, where the OS doesn't provide race free unlink-by-open-handle we compare the inode of the path we are about to unlink with that of the open handle before unlinking.

Warning
This does not prevent races where in between the time of checking the inode and executing the unlink a third party changes the item about to be unlinked. Only operating systems with a true race-free unlink syscall are race free.

Ask the OS to disable prefetching of data. This can improve random i/o performance.

Ask the OS to maximise prefetching of data, possibly prefetching the entire file into kernel cache. This can improve sequential i/o performance.

< See the documentation for unlink_on_first_close

Microsoft Windows NTFS, having been created in the late 1980s, did not originally implement extents-based storage and thus could only represent sparse files via efficient compression of intermediate zeros. With NTFS v3.0 (Microsoft Windows 2000), a proper extents-based on-storage representation was added, thus allowing only 64Kb extent chunks written to be stored irrespective of whatever the maximum file extent was set to.

For various historical reasons, extents-based storage is disabled by default in newly created files on NTFS, unlike in almost every other major filing system. You have to explicitly "opt in" to extents-based storage.

As extents-based storage is nearly cost free on NTFS, LLFIO by default opts in to extents-based storage for any empty file it creates. If you don't want this, you can specify this flag to prevent that happening.

Filesystems tend to be embarrassingly parallel for operations performed to different inodes. Where LLFIO performs i/o to multiple inodes at a time, it will use OpenMP or the Parallelism or Concurrency standard library extensions to usually complete the operation in constant rather than linear time. If you don't want this default, you can disable default using this flag.

Microsoft Windows NTFS has the option, when creating a directory, to set whether leafname lookup will be case sensitive. This is the only way of getting exact POSIX semantics on Windows without resorting to editing the system registry, however it also affects all code doing lookups within that directory, so we must default it to off.

Create the handle in a way where i/o upon it can be multiplexed with other i/o on the same initiating thread of execution i.e. you can perform more than one read concurrently, without using threads. The blocking operations .read() and .write() may have to use a less efficient, but cancellable, blocking implementation for handles created in this way. On Microsoft Windows, this creates handles with OVERLAPPED semantics. On POSIX, this creates handles with nonblocking semantics for non-file handles such as pipes and sockets, however for file, directory and symlink handles it does not set nonblocking, as it is non-portable.

< Using insane POSIX byte range locks

< This is an inode created with no representation on the filing system

110  {
111  none = uint16_t(0), //!< No flags
112  /*! Unlinks the file on handle close. On POSIX, this simply unlinks whatever is pointed
113  to by `path()` upon the call of `close()` if and only if the inode matches. On Windows,
114  if you are on Windows 10 1709 or later, exactly the same thing occurs. If on previous
115  editions of Windows, the file entry does not disappears but becomes unavailable for
116  anyone else to open with an `errc::resource_unavailable_try_again` error return. Because this is confusing, unless the
117  `win_disable_unlink_emulation` flag is also specified, this POSIX behaviour is
118  somewhat emulated by LLFIO on older Windows by renaming the file to a random name on `close()`
119  causing it to appear to have been unlinked immediately.
120  */
121  unlink_on_first_close = uint16_t(1U << 0U),
122 
123  /*! Some kernel caching modes have unhelpfully inconsistent behaviours
124  in getting your data onto storage, so by default unless this flag is
125  specified LLFIO adds extra fsyncs to the following operations for the
126  caching modes specified below:
127  * truncation of file length either explicitly or during file open.
128  * closing of the handle either explicitly or in the destructor.
129 
130  Additionally on Linux only to prevent loss of file metadata:
131  * On the parent directory whenever a file might have been created.
132  * On the parent directory on file close.
133 
134  This only occurs for these kernel caching modes:
135  * caching::none
136  * caching::reads
137  * caching::reads_and_metadata
138  * caching::safety_barriers
139  */
140  disable_safety_barriers = uint16_t(1U << 2U),
141  /*! `file_handle::unlink()` could accidentally delete the wrong file if someone has
142  renamed the open file handle since the time it was opened. To prevent this occuring,
143  where the OS doesn't provide race free unlink-by-open-handle we compare the inode of
144  the path we are about to unlink with that of the open handle before unlinking.
145  \warning This does not prevent races where in between the time of checking the inode
146  and executing the unlink a third party changes the item about to be unlinked. Only
147  operating systems with a true race-free unlink syscall are race free.
148  */
149  disable_safety_unlinks = uint16_t(1U << 3U),
150  /*! Ask the OS to disable prefetching of data. This can improve random
151  i/o performance.
152  */
153  disable_prefetching = uint16_t(1U << 4U),
154  /*! Ask the OS to maximise prefetching of data, possibly prefetching the entire file
155  into kernel cache. This can improve sequential i/o performance.
156  */
157  maximum_prefetching = uint16_t(1U << 5U),
158 
159  win_disable_unlink_emulation = uint16_t(1U << 9U), //!< See the documentation for `unlink_on_first_close`
160  /*! Microsoft Windows NTFS, having been created in the late 1980s, did not originally
161  implement extents-based storage and thus could only represent sparse files via
162  efficient compression of intermediate zeros. With NTFS v3.0 (Microsoft Windows 2000),
163  a proper extents-based on-storage representation was added, thus allowing only 64Kb
164  extent chunks written to be stored irrespective of whatever the maximum file extent
165  was set to.
166 
167  For various historical reasons, extents-based storage is disabled by default in newly
168  created files on NTFS, unlike in almost every other major filing system. You have to
169  explicitly "opt in" to extents-based storage.
170 
171  As extents-based storage is nearly cost free on NTFS, LLFIO by default opts in to
172  extents-based storage for any empty file it creates. If you don't want this, you
173  can specify this flag to prevent that happening.
174  */
175  win_disable_sparse_file_creation = uint16_t(1U << 10U),
176  /*! Filesystems tend to be embarrassingly parallel for operations performed to different
177  inodes. Where LLFIO performs i/o to multiple inodes at a time, it will use OpenMP or
178  the Parallelism or Concurrency standard library extensions to usually complete the
179  operation in constant rather than linear time. If you don't want this default, you can
180  disable default using this flag.
181  */
182  disable_parallelism = uint16_t(1U << 11U),
183  /*! Microsoft Windows NTFS has the option, when creating a directory, to set whether
184  leafname lookup will be case sensitive. This is the only way of getting exact POSIX
185  semantics on Windows without resorting to editing the system registry, however it also
186  affects all code doing lookups within that directory, so we must default it to off.
187  */
188  win_create_case_sensitive_directory = uint16_t(1U << 12U),
189 
190  /*! Create the handle in a way where i/o upon it can be multiplexed with other i/o
191  on the same initiating thread of execution i.e. you can perform more than one read
192  concurrently, without using threads. The blocking operations `.read()` and `.write()`
193  may have to use a less efficient, but cancellable, blocking implementation for handles created
194  in this way. On Microsoft Windows, this creates handles with `OVERLAPPED` semantics.
195  On POSIX, this creates handles with nonblocking semantics for non-file handles such
196  as pipes and sockets, however for file, directory and symlink handles it does not set
197  nonblocking, as it is non-portable.
198  */
199  multiplexable = uint16_t(1U << 13U),
200 
201  // NOTE: IF UPDATING THIS UPDATE THE std::ostream PRINTER BELOW!!!
202 
203  byte_lock_insanity = uint16_t(1U << 14U), //!< Using insane POSIX byte range locks
204  anonymous_inode = uint16_t(1U << 15U) //!< This is an inode created with no representation on the filing system
205  } QUICKCPPLIB_BITFIELD_END(flag)

◆ QUICKCPPLIB_BITFIELD_BEGIN_T() [2/2]

llfio_v2_xxx::directory_handle::QUICKCPPLIB_BITFIELD_BEGIN_T ( flags  ,
uint8_t   
)
inline

Flags for how to enumerate directory entries.

< No flags

< Do not error out if a atomic snapshot cannot be performed

193  {
194  none = 0u, //!< No flags
195  permit_racy_reads = (1u << 0u) //!< Do not error out if a atomic snapshot cannot be performed
196  } QUICKCPPLIB_BITFIELD_END(flags) using _flags_type = flags;

◆ read()

virtual result<buffers_type> llfio_v2_xxx::directory_handle::read ( io_request< buffers_type req,
deadline  d = std::chrono::seconds(30) 
) const
inlinevirtualnoexcept

Fill the buffers type with as many directory entries as will fit into any optionally supplied buffer. This operation returns a snapshot, without races, of the directory contents at the moment of the call, unless flags::permit_racy_reads is set.

Returns
Returns the buffers filled, what metadata was filled in and whether the entire directory was read or not. You should always examine .metadata() for the metadata you are about to use, fetching it with stat_t::fill() if not yet present.
Parameters
reqA buffer fill (directory enumeration) request.
dAn optional deadline by which the i/o must complete, else it is cancelled. Note function may return significantly after this deadline if the i/o takes long to cancel.
Errors returnable\n todo
Memory Allocations\n If the kernelbuffer parameter is set in the request, no memory allocations.
If unset, at least one memory allocation, possibly more is performed. MAKE SURE you reuse the buffers_type across calls once you are no longer using the buffers filled (simply restamp its span range, the internal kernel buffer will then get reused).

◆ relink()

virtual result<void> llfio_v2_xxx::fs_handle::relink ( const path_handle base,
path_view_type  path,
bool  atomic_replace = true,
deadline  d = std::chrono::seconds(30) 
)
inlinevirtualnoexceptinherited

Relinks the current path of this open handle to the new path specified. If atomic_replace is true, the relink atomically and silently replaces any item at the new path specified. This operation is both atomic and matching POSIX behaviour even on Microsoft Windows where no Win32 API can match POSIX semantics.

Note that if atomic_replace is false, the operation may be implemented as creating a hard link to the destination (which fails if the destination exists), opening a new file descriptor to the destination, closing the existing file descriptor, replacing the existing file descriptor with the new one (this is to ensure path tracking continues to work), then unlinking the previous link. Thus native_handle()'s value may change. This is not the case on Microsoft Windows nor Linux, both of which provide syscalls capable of refusing to rename if the destination exists.

If the handle refers to a pipe, on Microsoft Windows the base path handle is ignored as there is a single global named pipe namespace. Unless the path fragment begins with \, the string \??\ is prefixed to the name before passing it to the NT kernel API which performs the rename. This is because \\.\ in Win32 maps onto \??\ in the NT kernel.

Warning
Some operating systems provide a race free syscall for renaming an open handle (Windows). On all other operating systems this call is racy and can result in the wrong file entry being relinked. Note that unless flag::disable_safety_unlinks is set, this implementation opens a path_handle to the source containing directory first, then checks before relinking that the item about to be relinked has the same inode as the open file handle. It will retry this matching until success until the deadline given. This should prevent most unmalicious accidental loss of data.
Parameters
baseBase for any relative path.
pathThe relative or absolute new path to relink to.
atomic_replaceAtomically replace the destination if a file entry already is present there. Choosing false for this will fail if a file entry is already present at the destination, and may not be an atomic operation on some platforms (i.e. both the old and new names may be linked to the same inode for a very short period of time). Windows and recent Linuxes are always atomic.
dThe deadline by which the matching of the containing directory to the open handle's inode must succeed, else errc::timed_out will be returned.
Memory Allocations\n Except on platforms with race free syscalls for renaming open handles (Windows), calls
current_path() via parent_path_handle() and thus is both expensive and calls malloc many times.

Reimplemented in llfio_v2_xxx::symlink_handle, and llfio_v2_xxx::mapped_file_handle.

◆ remove_extended_attribute()

virtual result<void> llfio_v2_xxx::fs_handle::remove_extended_attribute ( path_view_component  )
inlinevirtualnoexceptinherited

Removes the extended attribute set on this file or directory.

Note
On Windows, this is the list of alternate streams on a file, NOT NTFS extended attributes. We do not prevent you trying to remove internal alternate streams, either.

◆ reopen()

virtual result<directory_handle> llfio_v2_xxx::directory_handle::reopen ( mode  mode_ = mode::unchanged,
caching  caching_ = caching::unchanged,
deadline  d = std::chrono::seconds(30) 
) const
inlinevirtualnoexcept

Reopen this handle (copy constructor is disabled to avoid accidental copying), optionally race free reopening the handle with different access or caching.

Microsoft Windows provides a syscall for cloning an existing handle but with new access. On POSIX, we must loop calling current_path(), trying to open the path returned and making sure it is the same inode.

Errors returnable\n Any of the values POSIX dup() or DuplicateHandle() can return.
Memory Allocations\n On POSIX if changing the mode, we must loop calling current_path() and
trying to open the path returned. Thus many allocations may occur.

◆ set_append_only()

virtual result<void> llfio_v2_xxx::handle::set_append_only ( bool  enable)
inlinevirtualnoexceptinherited

EXTENSION: Changes whether this handle is append only or not.

Warning
On Windows this is implemented as a bit of a hack to make it fast like on POSIX, so make sure you open the handle for read/write originally. Note unlike on POSIX the append_only disposition will be the only one toggled, seekable and readable will remain turned on.
Errors returnable\n Whatever POSIX fcntl() returns. On Windows nothing is changed on the handle.
Memory Allocations\n No memory allocation.

Reimplemented in llfio_v2_xxx::process_handle.

◆ set_extended_attribute()

virtual result<void> llfio_v2_xxx::fs_handle::set_extended_attribute ( path_view_component  name,
span< const byte >  value 
)
inlinevirtualnoexceptinherited

Sets the value of an extended attribute on this file or directory.

To prevent collision in a globally visible resource, there is a convention whereby you ought to namespace the names of your values as namespace.attribute e.g. appname.setting to prevent unintentional collision with other programs. Obviously, do choose a unique appname if there is any chance another program might use the same namespace name.

On POSIX, there are additional namespacing requirements: before your value name, you need to prefix one of user or system, so the actual name you might set would be user.appname.propname. Windows does not have the user/system prefix requirement, but it does no harm to do the exact same on Windows as on POSIX.

The host OS and target filing system choose the limits on value size, and will fail accordingly. Some impose a maximum of 64Kb for all names and values per inode, others have a 4Kb maximum value size, there are lots of combinations. You are probably safest not setting many names, and keep the values short.

Warning
Extended attributes are 'brittle' because they can get silently wiped at any moment. Never store anything in extended attributes which cannot be recalculated if missing. The ideal use case for extended attributes is as a cache of additional metadata about a file or directory e.g. "I last checked this directory at timestamp X", or "the MD5 hash at last modified timestamp X for this file was Y". Also remember that other processes can and do arbitrarily modify extended attributes concurrent to you.

Windows only

This API is implemented as file alternate data streams, rather than the Extended Attributes API as accessed via NtSetEaFile() and NtQueryEaFile() (which actually modify the file alternate data stream $EA in any case).

The reason why is that NtSetEaFile() can only append new records to EA storage. It cannot deallocate any existing EA records, if you try to do so you will get STATUS_EA_CORRUPT_ERROR. You can append setting the same name to a different value, which can include a null value which then appears as if the name is no longer there. But there is a cap of 64kB for the EA record, and once it is consumed, it is gone forever for that inode.

Obviously that doesn't map at all well onto POSIX extended attributes, where you can set the value of an attribute as frequently as you like. The closest equivalent on Windows is therefore file alternate data streams, even though the attribute's value is then worked with as a whole proper file with all the attendant performance consequences.

As a result, name must be a valid filename and not contain any characters not permitted in a filename. We use the NT API here, so the character restrictions are far fewer than for the Win32 API e.g. single character names do NOT cause misoperation like on Win32.

◆ temp_directory()

static result<directory_handle> llfio_v2_xxx::directory_handle::temp_directory ( path_view_type  name = path_view_type(),
mode  _mode = mode::write,
creation  _creation = creation::if_needed,
caching  _caching = caching::all,
flag  flags = flag::none 
)
inlinestaticnoexcept

Create a directory handle creating the named directory on some path which the OS declares to be suitable for temporary files. Note also that an empty name is equivalent to calling uniquely_named_file(path_discovery::storage_backed_temporary_files_directory()) and the creation parameter is ignored.

Errors returnable\n Any of the values POSIX open() or CreateFile() can return.
347  {
349  return name.empty() ? uniquely_named_directory(tempdirh, _mode, _caching, flags) : directory(tempdirh, name, _mode, _creation, _caching, flags);
350  }
static result< directory_handle > directory(const path_handle &base, path_view_type path, mode _mode=mode::read, creation _creation=creation::open_existing, caching _caching=caching::all, flag flags=flag::none) noexcept
static result< directory_handle > uniquely_named_directory(const path_handle &dirpath, mode _mode=mode::write, caching _caching=caching::temporary, flag flags=flag::none) noexcept
Definition: directory_handle.hpp:315
const path_handle & storage_backed_temporary_files_directory() noexcept
Returns a reference to an open handle to a verified temporary directory where files created are store...

◆ uniquely_named_directory()

static result<directory_handle> llfio_v2_xxx::directory_handle::uniquely_named_directory ( const path_handle dirpath,
mode  _mode = mode::write,
caching  _caching = caching::temporary,
flag  flags = flag::none 
)
inlinestaticnoexcept

Create a directory handle creating a uniquely named file on a path. The file is opened exclusively with creation::only_if_not_exist so it will never collide with nor overwrite any existing entry.

Errors returnable\n Any of the values POSIX open() or CreateFile() can return.
317  {
318  try
319  {
320  for(;;)
321  {
322  auto randomname = utils::random_string(32);
323  result<directory_handle> ret = directory(dirpath, randomname, _mode, creation::only_if_not_exist, _caching, flags);
324  if(ret || (!ret && ret.error() != errc::file_exists))
325  {
326  return ret;
327  }
328  }
329  }
330  catch(...)
331  {
332  return error_from_exception();
333  }
334  }
@ only_if_not_exist
Filesystem entry must NOT exist, and is atomically created by the success of this operation.
std::string random_string(size_t randomlen)
Returns a cryptographically random string capable of being used as a filename. Essentially random_fil...
Definition: utils.hpp:170

◆ unlink()

virtual result<void> llfio_v2_xxx::fs_handle::unlink ( deadline  d = std::chrono::seconds(30))
inlinevirtualnoexceptinherited

Unlinks the current path of this open handle, causing its entry to immediately disappear from the filing system.

On Windows before Windows 10 1709 unless flag::win_disable_unlink_emulation is set, this behaviour is simulated by renaming the file to something random and setting its delete-on-last-close flag. Note that Windows may prevent the renaming of a file in use by another process, if so it will NOT be renamed. After the next handle to that file closes, it will become permanently unopenable by anyone else until the last handle is closed, whereupon the entry will be eventually removed by the operating system.

Warning
Some operating systems provide a race free syscall for unlinking an open handle (Windows). On all other operating systems this call is racy and can result in the wrong file entry being unlinked. Note that unless flag::disable_safety_unlinks is set, this implementation opens a path_handle to the containing directory first, then checks that the item about to be unlinked has the same inode as the open file handle. It will retry this matching until success until the deadline given. This should prevent most unmalicious accidental loss of data.
Parameters
dThe deadline by which the matching of the containing directory to the open handle's inode must succeed, else errc::timed_out will be returned.
Memory Allocations\n Except on platforms with race free syscalls for unlinking open handles (Windows), calls
current_path() and thus is both expensive and calls malloc many times. On Windows, also calls current_path() if flag::disable_safety_unlinks is not set.

Reimplemented in llfio_v2_xxx::symlink_handle.


The documentation for this class was generated from the following file: