llfio_v2_xxx::mapped_file_handle Class Reference

A memory mapped regular file or device. More...

#include "mapped_file_handle.hpp"

Inheritance diagram for llfio_v2_xxx::mapped_file_handle:

Public Types
using	dev_t = file_handle::dev_t
using	ino_t = file_handle::ino_t
using	path_view_type = file_handle::path_view_type
using	path_type = byte_io_handle::path_type
using	extent_type = byte_io_handle::extent_type
using	size_type = byte_io_handle::size_type
using	mode = byte_io_handle::mode
using	creation = byte_io_handle::creation
using	caching = byte_io_handle::caching
using	flag = byte_io_handle::flag
using	buffer_type = byte_io_handle::buffer_type
using	const_buffer_type = byte_io_handle::const_buffer_type
using	buffers_type = byte_io_handle::buffers_type
using	const_buffers_type = byte_io_handle::const_buffers_type
template<class T >
using	io_request = byte_io_handle::io_request< T >
template<class T >
using	io_result = byte_io_handle::io_result< T >
using	barrier_kind = byte_io_multiplexer::barrier_kind
using	registered_buffer_type = byte_io_multiplexer::registered_buffer_type
template<class T >
using	awaitable = byte_io_multiplexer::awaitable< T >
using	unique_id_type = QUICKCPPLIB_NAMESPACE::integers128::uint128
	The unique identifier type used by this handle.
using	unique_id_type_hasher = QUICKCPPLIB_NAMESPACE::integers128::uint128_hasher
	A hasher for the unique identifier type used by this handle.

Public Member Functions
constexpr	mapped_file_handle ()
	Default constructor.
	mapped_file_handle (mapped_file_handle &&o) noexcept
	Implicit move construction of mapped_file_handle permitted.
	mapped_file_handle (const mapped_file_handle &)=delete
	No copy construction (use clone())
template<class SHF , typename std::enable_if<(std::is_same< typename std::decay< SHF >::type, section_handle::flag >::value||std::is_same< typename std::decay< SHF >::type, section_handle::flag::enum_type >::value), bool >::type = true>
constexpr	mapped_file_handle (file_handle &&o, SHF sflags, extent_type offset) noexcept
	Explicit conversion from file_handle permitted.
template<class SHF , typename std::enable_if<(std::is_same< typename std::decay< SHF >::type, section_handle::flag >::value||std::is_same< typename std::decay< SHF >::type, section_handle::flag::enum_type >::value), bool >::type = true>
	mapped_file_handle (file_handle &&o, size_type reservation, SHF sflags, extent_type offset)
	Explicit conversion from file_handle permitted, this overload also attempts to map the file.
mapped_file_handle &	operator= (mapped_file_handle &&o) noexcept
	Move assignment of mapped_file_handle permitted.
mapped_file_handle &	operator= (const mapped_file_handle &)=delete
	No copy assignment.
void	swap (mapped_file_handle &o) noexcept
	Swap with another instance.
const section_handle &	section () const noexcept
	The memory section this handle is using.
section_handle &	section () noexcept
	The memory section this handle is using.
const map_handle &	map () const noexcept
	The map this handle is using.
map_handle &	map () noexcept
	The map this handle is using.
byte *	address () const noexcept
	The address in memory where this mapped file currently resides.
extent_type	starting_offset () const noexcept
	The offset into the backing file from which this mapped file begins.
size_type	page_size () const noexcept
	The page size used by the map, in bytes.
bool	is_nvram () const noexcept
	True if the map is of non-volatile RAM.
result< extent_type >	underlying_file_maximum_extent () const noexcept
	The maximum extent of the underlying file, minus any offset.
size_type	capacity () const noexcept
	The address space (to be) reserved for future expansion of this file.
result< size_type >	reserve (size_type reservation=0) noexcept
	Reserve a new amount of address space for mapping future expansion of this file. More...
virtual result< void >	close () noexcept override
	Immediately close the native handle type managed by this handle.
virtual native_handle_type	release () noexcept override
	Release the native handle type managed by this handle.
result< mapped_file_handle >	reopen (size_type reservation, extent_type offset=0, mode mode_=mode::unchanged, caching caching_=caching::unchanged, deadline d=std::chrono::seconds(30)) const noexcept
template<class... Args>
bool	try_reopen (Args &&... args) noexcept
template<class... Args, class Rep , class Period >
bool	try_reopen_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
template<class... Args, class Clock , class Duration >
bool	try_reopen_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
virtual result< void >	set_multiplexer (byte_io_multiplexer *c=this_thread::multiplexer()) noexcept override
	Sets the i/o multiplexer this handle will use to implement read(), write() and barrier(). More...
virtual result< extent_type >	maximum_extent () const noexcept override
	Return the current maximum permitted extent of the file, after updating the map. More...
virtual result< extent_type >	truncate (extent_type newsize) noexcept override
	Resize the current maximum permitted extent of the mapped file to the given extent, avoiding any new allocation of physical storage where supported, and mapping or unmapping any new pages up to the reservation to reflect the new maximum extent. If the new size exceeds the reservation, reserve() will be called to increase the reservation. More...
result< extent_type >	update_map () noexcept
	Efficiently update the mapping to match that of the underlying file, returning the new current maximum permitted extent of the file. More...
virtual result< extent_type >	zero (extent_pair extent, deadline=deadline()) noexcept override
	Efficiently zero, and possibly deallocate, data on storage. More...
virtual result< void >	relink (const path_handle &base, path_view_type path, bool atomic_replace=true, deadline d=std::chrono::seconds(30)) noexcept override
	Relinks the current path of this open handle to the new path specified. If atomic_replace is true, the relink atomically and silently replaces any item at the new path specified. This operation is both atomic and matching POSIX behaviour even on Microsoft Windows where no Win32 API can match POSIX semantics. More...
template<class... Args>
bool	try_relink (Args &&... args) noexcept
template<class... Args, class Rep , class Period >
bool	try_relink_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
template<class... Args, class Clock , class Duration >
bool	try_relink_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
void	swap (file_handle &o) noexcept
	Swap with another instance.
void	swap (handle &o) noexcept
	Swap with another instance.
result< file_handle >	reopen (mode mode_=mode::unchanged, caching caching_=caching::unchanged, deadline d=std::chrono::seconds(30)) const noexcept
virtual result< std::vector< extent_pair > >	extents () const noexcept
	Returns a list of currently valid extents for this open file. WARNING: racy! More...
virtual result< extent_pair >	clone_extents_to (extent_pair extent, byte_io_handle &dest, byte_io_handle::extent_type destoffset, deadline d={}, bool force_copy_now=false, bool emulate_if_unsupported=true) noexcept
	Clones the extents referred to by extent to dest at destoffset. This is how you ought to copy file content, including within the same file. This is fundamentally a racy call with respect to concurrent modification of the files. More...
result< extent_pair >	clone_extents_to (byte_io_handle &dest, deadline d={}, bool force_copy_now=false, bool emulate_if_unsupported=true) noexcept
	This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.
result< extent_type >	zero (extent_type offset, extent_type bytes, deadline d=deadline()) noexcept
	This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.
template<class... Args>
bool	try_zero (Args &&... args) noexcept
template<class... Args, class Rep , class Period >
bool	try_zero_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
template<class... Args, class Clock , class Duration >
bool	try_zero_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
virtual result< void >	lock_file () noexcept
	Locks the inode referred to by the open handle for exclusive access. More...
virtual bool	try_lock_file () noexcept
	Tries to lock the inode referred to by the open handle for exclusive access, returning false if lock is currently unavailable. More...
virtual void	unlock_file () noexcept
	Unlocks a previously acquired exclusive lock.
virtual result< void >	lock_file_shared () noexcept
	Locks the inode referred to by the open handle for shared access. More...
virtual bool	try_lock_file_shared () noexcept
	Tries to lock the inode referred to by the open handle for shared access, returning false if lock is currently unavailable. More...
virtual void	unlock_file_shared () noexcept
	Unlocks a previously acquired shared lock.
virtual result< extent_guard >	lock_file_range (extent_type offset, extent_type bytes, lock_kind kind, deadline d=deadline()) noexcept
	EXTENSION: Tries to lock the range of bytes specified for shared or exclusive access. Note that this may, or MAY NOT, observe whole file locks placed with lock(), lock_shared() etc. More...
result< extent_guard >	lock_file_range (io_request< buffers_type > reqs, deadline d=deadline()) noexcept
result< extent_guard >	lock_file_range (io_request< const_buffers_type > reqs, deadline d=deadline()) noexcept
template<class... Args>
bool	try_lock_file_range (Args &&... args) noexcept
template<class... Args, class Rep , class Period >
bool	try_lock_file_range_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
template<class... Args, class Clock , class Duration >
bool	try_lock_file_range_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
virtual void	unlock_file_range (extent_type offset, extent_type bytes) noexcept
	EXTENSION: Unlocks a byte range previously locked. More...
byte_io_multiplexer *	multiplexer () const noexcept
	The i/o multiplexer this handle will use to multiplex i/o. If this returns null, then this handle has not been registered with an i/o multiplexer yet.
size_t	max_buffers () const noexcept
	The maximum number of buffers which a single read or write syscall can (atomically) process at a time for this specific open handle. On POSIX, this is known as IOV_MAX. Preferentially uses any i/o multiplexer set over the virtually overridable per-class implementation. More...
result< registered_buffer_type >	allocate_registered_buffer (size_t &bytes) noexcept
	Request the allocation of a new registered i/o buffer with the system suitable for maximum performance i/o, preferentially using any i/o multiplexer set over the virtually overridable per-class implementation. More...
io_result< buffers_type >	read (io_request< buffers_type > reqs, deadline d=deadline()) noexcept
	Read data from the open handle, preferentially using any i/o multiplexer set over the virtually overridable per-class implementation. More...
io_result< buffers_type >	read (registered_buffer_type base, io_request< buffers_type > reqs, deadline d=deadline()) noexcept
io_result< size_type >	read (extent_type offset, std::initializer_list< buffer_type > lst, deadline d=deadline()) noexcept
template<class... Args>
bool	try_read (Args &&... args) noexcept
template<class... Args, class Rep , class Period >
bool	try_read_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
template<class... Args, class Clock , class Duration >
bool	try_read_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
io_result< const_buffers_type >	write (io_request< const_buffers_type > reqs, deadline d=deadline()) noexcept
	Write data to the open handle, preferentially using any i/o multiplexer set over the virtually overridable per-class implementation. More...
io_result< const_buffers_type >	write (registered_buffer_type base, io_request< const_buffers_type > reqs, deadline d=deadline()) noexcept
io_result< size_type >	write (extent_type offset, std::initializer_list< const_buffer_type > lst, deadline d=deadline()) noexcept
template<class... Args>
bool	try_write (Args &&... args) noexcept
template<class... Args, class Rep , class Period >
bool	try_write_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
template<class... Args, class Clock , class Duration >
bool	try_write_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
virtual io_result< const_buffers_type >	barrier (io_request< const_buffers_type > reqs=io_request< const_buffers_type >(), barrier_kind kind=barrier_kind::nowait_data_only, deadline d=deadline()) noexcept
	Issue a write reordering barrier such that writes preceding the barrier will reach storage before writes after this barrier, preferentially using any i/o multiplexer set over the virtually overridable per-class implementation. More...
io_result< const_buffers_type >	barrier (barrier_kind kind, deadline d=deadline()) noexcept
template<class... Args>
bool	try_barrier (Args &&... args) noexcept
template<class... Args, class Rep , class Period >
bool	try_barrier_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
template<class... Args, class Clock , class Duration >
bool	try_barrier_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
awaitable< io_result< buffers_type > >	co_read (io_request< buffers_type > reqs, deadline d=deadline()) noexcept
	A coroutinised equivalent to .read() which suspends the coroutine until the i/o finishes. Blocks execution i.e is equivalent to .read() if no i/o multiplexer has been set on this handle! More...
awaitable< io_result< buffers_type > >	co_read (registered_buffer_type base, io_request< buffers_type > reqs, deadline d=deadline()) noexcept
awaitable< io_result< const_buffers_type > >	co_write (io_request< const_buffers_type > reqs, deadline d=deadline()) noexcept
	A coroutinised equivalent to .write() which suspends the coroutine until the i/o finishes. Blocks execution i.e is equivalent to .write() if no i/o multiplexer has been set on this handle! More...
awaitable< io_result< const_buffers_type > >	co_write (registered_buffer_type base, io_request< const_buffers_type > reqs, deadline d=deadline()) noexcept
awaitable< io_result< const_buffers_type > >	co_barrier (io_request< const_buffers_type > reqs=io_request< const_buffers_type >(), barrier_kind kind=barrier_kind::nowait_data_only, deadline d=deadline()) noexcept
	A coroutinised equivalent to .barrier() which suspends the coroutine until the i/o finishes. Blocks execution i.e is equivalent to .barrier() if no i/o multiplexer has been set on this handle! More...
flag	flags () const noexcept
	The flags this handle was opened with.
	QUICKCPPLIB_BITFIELD_BEGIN_T (flag, uint16_t)
	Bitwise flags which can be specified. More...
virtual result< path_type >	current_path () const noexcept
result< handle >	clone () const noexcept
bool	is_valid () const noexcept
	True if the handle is valid (and usually open)
bool	is_readable () const noexcept
	True if the handle is readable.
bool	is_writable () const noexcept
	True if the handle is writable.
bool	is_append_only () const noexcept
	True if the handle is append only.
virtual result< void >	set_append_only (bool enable) noexcept
	EXTENSION: Changes whether this handle is append only or not. More...
bool	is_multiplexable () const noexcept
	True if multiplexable.
bool	is_nonblocking () const noexcept
	True if nonblocking.
bool	is_seekable () const noexcept
	True if seekable.
bool	requires_aligned_io () const noexcept
	True if requires aligned i/o.
bool	is_kernel_handle () const noexcept
	True if native_handle() is a valid kernel handle.
bool	is_regular () const noexcept
	True if a regular file or device.
bool	is_directory () const noexcept
	True if a directory.
bool	is_symlink () const noexcept
	True if a symlink.
bool	is_pipe () const noexcept
	True if a pipe.
bool	is_socket () const noexcept
	True if a socket.
bool	is_multiplexer () const noexcept
	True if a multiplexer like BSD kqueues, Linux epoll or Windows IOCP.
bool	is_process () const noexcept
	True if a process.
bool	is_section () const noexcept
	True if a memory section.
bool	is_allocation () const noexcept
	True if a memory allocation.
bool	is_path () const noexcept
	True if a path or a directory.
bool	is_tls_socket () const noexcept
	True if a TLS socket.
bool	is_http_socket () const noexcept
	True if a HTTP socket.
caching	kernel_caching () const noexcept
	Kernel cache strategy used by this handle.
bool	are_reads_from_cache () const noexcept
	True if the handle uses the kernel page cache for reads.
bool	are_writes_durable () const noexcept
	True if writes are safely on storage on completion.
bool	are_safety_barriers_issued () const noexcept
	True if issuing safety fsyncs is on.
native_handle_type	native_handle () const noexcept
	The native handle used by this handle.
dev_t	st_dev () const noexcept
	Unless flag::disable_safety_unlinks is set, the device id of the file when opened.
ino_t	st_ino () const noexcept
	Unless flag::disable_safety_unlinks is set, the inode of the file when opened. When combined with st_dev(), forms a unique identifer on this system.
unique_id_type	unique_id () const noexcept
	A unique identifier for this handle across the entire system. Can be used in hash tables etc.
virtual result< path_handle >	parent_path_handle (deadline d=std::chrono::seconds(30)) const noexcept
	Obtain a handle to the path currently containing this handle's file entry. More...
template<class... Args>
bool	try_parent_path_handle (Args &&... args) noexcept
template<class... Args, class Rep , class Period >
bool	try_parent_path_handle_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
template<class... Args, class Clock , class Duration >
bool	try_parent_path_handle_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
virtual result< void >	link (const path_handle &base, path_view_type path, deadline d=std::chrono::seconds(30)) noexcept
	Links the inode referred to by this open handle to the path specified. The current path of this open handle is not changed, unless it has no current path due to being unlinked. More...
template<class... Args>
bool	try_link (Args &&... args) noexcept
template<class... Args, class Rep , class Period >
bool	try_link_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
template<class... Args, class Clock , class Duration >
bool	try_link_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
virtual result< void >	unlink (deadline d=std::chrono::seconds(30)) noexcept
	Unlinks the current path of this open handle, causing its entry to immediately disappear from the filing system. More...
template<class... Args>
bool	try_unlink (Args &&... args) noexcept
template<class... Args, class Rep , class Period >
bool	try_unlink_for (Args &&... args, const std::chrono::duration< Rep, Period > &duration) noexcept
template<class... Args, class Clock , class Duration >
bool	try_unlink_until (Args &&... args, const std::chrono::time_point< Clock, Duration > &timeout) noexcept
virtual result< span< path_view_component > >	list_extended_attributes (span< byte > tofill) const noexcept
	Fill the supplied buffer with the names of all extended attributes set on this file or directory, returning a span of path view components. More...
virtual result< span< byte > >	get_extended_attribute (span< byte > tofill, path_view_component name) const noexcept
	Retrieve the value of an extended attribute set on this file or directory. More...
virtual result< void >	set_extended_attribute (path_view_component name, span< const byte > value) noexcept
	Sets the value of an extended attribute on this file or directory. More...
virtual result< void >	remove_extended_attribute (path_view_component) noexcept
	Removes the extended attribute set on this file or directory. More...
result< size_t >	copy_extended_attributes (const fs_handle &src, bool replace_all_local_attributes=false) noexcept
	Copies the extended attributes from one entity to another, optionally replacing all the extended attributes on the destination. More...

Static Public Member Functions
static result< mapped_file_handle >	mapped_file (size_type reservation, const path_handle &base, path_view_type _path, mode _mode=mode::read, creation _creation=creation::open_existing, caching _caching=caching::all, flag flags=flag::none, section_handle::flag sflags=section_handle::flag::none, extent_type offset=0) noexcept
static result< mapped_file_handle >	mapped_file (const path_handle &base, path_view_type _path, mode _mode=mode::read, creation _creation=creation::open_existing, caching _caching=caching::all, flag flags=flag::none, section_handle::flag sflags=section_handle::flag::none, extent_type offset=0) noexcept
	This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.
static result< mapped_file_handle >	mapped_uniquely_named_file (size_type reservation, const path_handle &dirpath, mode _mode=mode::write, caching _caching=caching::temporary, flag flags=flag::none, section_handle::flag sflags=section_handle::flag::none) noexcept
static result< mapped_file_handle >	mapped_temp_file (size_type reservation, path_view_type name=path_view_type(), mode _mode=mode::write, creation _creation=creation::if_needed, caching _caching=caching::temporary, flag flags=flag::unlink_on_first_close, section_handle::flag sflags=section_handle::flag::none) noexcept
static result< mapped_file_handle >	mapped_temp_inode (size_type reservation=0, const path_handle &dir=path_discovery::storage_backed_temporary_files_directory(), mode _mode=mode::write, caching _caching=caching::temporary, flag flags=flag::none, section_handle::flag sflags=section_handle::flag::none) noexcept
static result< file_handle >	file (const path_handle &base, path_view_type path, mode _mode=mode::read, creation _creation=creation::open_existing, caching _caching=caching::all, flag flags=flag::none) noexcept
static result< file_handle >	uniquely_named_file (const path_handle &dirpath, mode _mode=mode::write, caching _caching=caching::temporary, flag flags=flag::none) noexcept
static result< file_handle >	temp_file (path_view_type name=path_view_type(), mode _mode=mode::write, creation _creation=creation::if_needed, caching _caching=caching::temporary, flag flags=flag::unlink_on_first_close) noexcept
static result< file_handle >	temp_inode (const path_handle &dirh=path_discovery::storage_backed_temporary_files_directory(), mode _mode=mode::write, caching _caching=caching::temporary, flag flags=flag::none) noexcept

Protected Member Functions
virtual size_t	_do_max_buffers () const noexcept override
	The virtualised implementation of max_buffers() used if no multiplexer has been set.
virtual io_result< const_buffers_type >	_do_barrier (io_request< const_buffers_type > reqs=io_request< const_buffers_type >(), barrier_kind kind=barrier_kind::nowait_data_only, deadline d=deadline()) noexcept override
	The virtualised implementation of barrier() used if no multiplexer has been set.
virtual io_result< buffers_type >	_do_read (io_request< buffers_type > reqs, deadline d=deadline()) noexcept override
	The virtualised implementation of read() used if no multiplexer has been set.
virtual io_result< const_buffers_type >	_do_write (io_request< const_buffers_type > reqs, deadline d=deadline()) noexcept override
	The virtualised implementation of write() used if no multiplexer has been set.
result< size_type >	_reserve (extent_type &length, size_type reservation) noexcept
virtual result< registered_buffer_type >	_do_allocate_registered_buffer (size_t &bytes) noexcept
	The virtualised implementation of allocate_registered_buffer() used if no multiplexer has been set.
virtual io_result< buffers_type >	_do_read (registered_buffer_type base, io_request< buffers_type > reqs, deadline d) noexcept
	The virtualised implementation of read() used if no multiplexer has been set.
virtual io_result< const_buffers_type >	_do_write (registered_buffer_type base, io_request< const_buffers_type > reqs, deadline d) noexcept
	The virtualised implementation of write() used if no multiplexer has been set.
io_result< buffers_type >	_do_multiplexer_read (registered_buffer_type &&base, io_request< buffers_type > reqs, deadline d) noexcept
io_result< const_buffers_type >	_do_multiplexer_write (registered_buffer_type &&base, io_request< const_buffers_type > reqs, deadline d) noexcept
io_result< const_buffers_type >	_do_multiplexer_barrier (registered_buffer_type &&base, io_request< const_buffers_type > reqs, barrier_kind kind, deadline d) noexcept
result< void >	_fetch_inode () const noexcept
	Fill in _devid and _inode from the handle via fstat()

Protected Attributes
size_type	_reservation {0}
extent_type	_offset {0}
section_handle	_sh
map_handle	_mh
byte_io_multiplexer *	_ctx {nullptr}
union {
native_handle_type   _v
struct {
intptr_t   _padding0_
uint32_t   _padding1_
flag   flags
uint16_t   _padding2_
}   _
};
dev_t	_devid {0}
ino_t	_inode {0}

Detailed Description

A memory mapped regular file or device.

	Cost of opening	Cost of i/o	Concurrency and Atomicity	Other remarks
file_handle	Least	Syscall	POSIX guarantees (usually)	Least gotcha
mapped_file_handle	Most	Least	None	Cannot be used with uncached i/o

All the major OSs on all the major 64 bit CPU architectures now offer at least 127 Tb of address spaces to user mode processes. This makes feasible mapping multi-Tb files directly into memory, and thus avoiding the syscall overhead involved when reading and writing. This becames especially important with next-gen storage devices capable of Direct Access Storage (DAX) like Optane from 2018 onwards, performance via syscalls will always be but a fraction of speaking directly to the storage device via directly mapped memory.

As an example of the gains, on Microsoft Windows to read or write 1Kb using the standard syscalls takes about fifteen times longer than the exact same i/o via mapped memory. On Linux, OS X or FreeBSD the gain is considerably lower, a 1Kb i/o might only be 50% slower via syscalls than memory maps. However for lots of say 64 byte i/o, the gain of memory maps over syscalls is unsurpassable.

This class combines a file_handle with a section_handle and a map_handle to implement a fully memory mapped file_handle. The portion of the file between starting_offset() and capacity() is mapped into memory, and read() and write() i/o is performed directly with the map. Reads always return the original mapped data, and do not fill any buffers passed in. For obvious reasons the utility of this class on 32-bit systems is limited, but can be useful when used with smaller files.

Note that zero length files cannot be memory mapped on most platforms, and writes past the maximum extent do NOT auto-extend the size of the file, rather the data written beyond the maximum valid extent has undefined kernel-specific behaviour, which includes segfaulting. You must therefore always truncate(newsize) to resize the file and its maps before you can read or write to it, and be VERY careful to not read or write beyond the maximum extent of the file.

On most platforms, when a file is created or is otherwise of zero length, address() will return a null pointer. Similarly, calling truncate(0) will close the map and section handles, they will be recreated on next truncation to a non-zero size.

For better performance when handling files which are growing, there is a concept of "address space reservation" via reserve() and capacity(), which on some kernels is automatically and efficiently expanded into when the underlying file grows. The implementation asks the kernel to set out a contiguous region of pages matching that reservation, and to map the file into the beginning of the reservation. The remainder of the pages may be inaccessible and may generate a segfault, or they may automatically reflect any growth in the underlying file. This is why read() and write() only know about the reservation size, and will read and write memory up to that reservation size, without checking if the memory involved exists or not yet.

You are guaranteed on POSIX only that address() will not return a new value unless:

1. You truncate from a bigger length to a smaller length.
2. You call reserve() with a new reservation.
3. You call truncate() with a value bigger than the reservation.
4. You call relink() with atomic_replace = false, which may on some platforms require a close-open file descriptor cycle as part of its implementation.

You are guaranteed on Windows only that address() will not return a new value unless:

1. You truncate from a bigger length to a smaller length.
2. You call reserve() with a new reservation.
3. You call truncate() with a value bigger than the reservation.
4. You call relink(), which requires closing and reopening the map because you cannot rename a file with an open map on Windows.

maximum_extent() in mapped file handle is an alias for update_map(). update_map() fetches the maximum extent of the underlying file, and if it has changed from the map's length, the map is updated to match the underlying file, up to the reservation limit. You can of course explicitly call update_map() whenever you need the map to reflect changes to the maximum extent of the underlying file.

starting_offset() offsets all map i/o, truncation, and sizing, it is always as if the underlying file starts from that offset. This only applies to the map_handle portion, the section_handle portion has no concept of starting offsets. Starting offsets can be byte granularity, the underlying map will use an appropriate platform specific granularity (typically the page size on POSIX and 64Kb on Windows). You might note that combined with how reservations work, this allows a mapped_file_handle to wholly reflect a subset of a very large file – only the relevant portion of that very large file will be mapped into memory, read() and write() will not exceed the boundaries of starting_offset() - capacity(), and maximum_extent() will not return a value exceeding the reservation. This effectively means such a subset file quacks like a file_handle of a complete file reflecting just that subset proportion.

It is up to you to detect that the reservation has been exhausted, and to reserve a new reservation which will change the value returned by address(). This entirely manual system is a bit tedious and cumbersome to use, but as mapping files is an expensive operation given TLB shootdown, we leave it up to the end user to decide when to expend the cost of mapping.

Warning

You must be cautious when the file is being extended by third parties which are not using this mapped_file_handle to write the new data. With unified page cache kernels, mixing mapped and normal i/o is generally safe except at the end of a file where race conditions and outright kernel bugs tend to abound. To avoid these, solely and exclusively use a dedicated handle configured to atomic append only to do the appends.

For 64-bit systems under heavy load, or all 32-bit systems, one can run out of enough contiguous virtual memory address space to map all of a large file. This generally presents itself as an error code comparing equal to errc::not_enough_memory, and it can appear from the constructor, truncate(), reserve() and most of the other functions in this class not inherited from base classes. update_map() never returns errc::not_enough_memory, but relink() may do so, due to the potential map teardown and recreate.

Microsoft Windows only

Microsoft Windows can have quite different semantics to POSIX, which are important to be aware of.

On Windows, the length of any section object for a file can never exceed the maximum extent of the file. Open section objects therefore clamp the valid maximum extent of a file to no lower than the largest of the section objects open upon the file. This is enforced by Windows, which will return an error if you try to truncate a file to smaller than any of its section objects on the system.

Windows only implements address space reservation for sections opened with write permissions. LLFIO names the section object backing every mapped file uniquely to the file's inode, and all those section objects are shared between all LLFIO processes for the current Windows session. This means that if any one process updates a section object's length, the section's length gets updated for all processes, and all maps of that section in all processes get automatically updated. This is far more efficient than each process updating a section individually (like, orders of magnitude better).

This works great if, and only if, the very first mapping of any inode is with a writable file handle, as that permits the global section object all mapped files of that inode (including read only maps) to use the same section object. If however the very first mapping of any file is with a read-only file handle, that creates a read-only section object, and that forces the first writable map to create a second section object. Now, both section objects must be updated individually when the backing file length changes, potentially halving your performance.

This is different to on POSIX, where the ordering of the permissions of how you open a mapped file does not silently impair performance.

Member Function Documentation

allocate_registered_buffer()

result<registered_buffer_type> llfio_v2_xxx::byte_io_handle::allocate_registered_buffer ( size_t &  bytes )

inlinenoexceptinherited

Request the allocation of a new registered i/o buffer with the system suitable for maximum performance i/o, preferentially using any i/o multiplexer set over the virtually overridable per-class implementation.

Returns

A shared pointer to the i/o buffer. Note that the pointer returned is not the resource under management, using shared ptr's aliasing feature.

Parameters

bytes

The size of the i/o buffer requested. This may be rounded (considerably) upwards, you should always use the value returned.

Some i/o multiplexer implementations have the ability to allocate i/o buffers in special memory shared between the i/o hardware and user space processes. Using registered i/o buffers can entirely eliminate all kernel transitions and memory copying during i/o, and can saturate very high end hardware from a single kernel thread.

If no multiplexer is set, the default implementation uses map_handle to allocate raw memory pages from the OS kernel. If the requested buffer size is a multiple of one of the larger page sizes from utils::page_sizes(), an attempt to satisfy the request using the larger page size will be attempted first.

  {
    if(_ctx == nullptr)
    {
      return _do_allocate_registered_buffer(bytes);
    }
    return _ctx->do_byte_io_handle_allocate_registered_buffer(this, bytes);
  }

barrier()

virtual io_result<const_buffers_type> llfio_v2_xxx::byte_io_handle::barrier ( io_request< const_buffers_type >  reqs = io_request<const_buffers_type>(), barrier_kind  kind = barrier_kind::nowait_data_only, deadline  d = deadline()  )

inlinevirtualnoexceptinherited

Issue a write reordering barrier such that writes preceding the barrier will reach storage before writes after this barrier, preferentially using any i/o multiplexer set over the virtually overridable per-class implementation.

Warning

Assume that this call is a no-op. It is not reliably implemented in many common use cases, for example if your code is running inside a LXC container, or if the user has mounted the filing system with non-default options. Instead open the handle with caching::reads which means that all writes form a strict sequential order not completing until acknowledged by the storage device. Filing system can and do use different algorithms to give much better performance with caching::reads, some (e.g. ZFS) spectacularly better.

Let me repeat again: consider this call to be a hint to poke the kernel with a stick to go start to do some work sooner rather than later. It may be ignored entirely.

For portability, you can only assume that barriers write order for a single handle instance. You cannot assume that barriers write order across multiple handles to the same inode, or across processes.

Returns

The buffers barriered, which may not be the buffers input. The size of each scatter-gather buffer is updated with the number of bytes of that buffer barriered.

Parameters

reqs	A scatter-gather and offset request for what range to barrier. May be ignored on some platforms which always write barrier the entire file. Supplying a default initialised reqs write barriers the entire file.
kind	Which kind of write reordering barrier to perform.
d	An optional deadline by which the i/o must complete, else it is cancelled. Note function may return significantly after this deadline if the i/o takes long to cancel.

Errors returnable\n Any of the values POSIX fdatasync() or Windows NtFlushBuffersFileEx() can return.

Memory Allocations\n None.

  {
    return (_ctx == nullptr) ? _do_barrier(reqs, kind, d) : _do_multiplexer_barrier({}, std::move(reqs), kind, d);
  }

clone()

result<handle> llfio_v2_xxx::handle::clone ( ) const

inlinenoexceptinherited

Clone this handle (copy constructor is disabled to avoid accidental copying)

Errors returnable\n Any of the values POSIX dup() or DuplicateHandle() can return.

clone_extents_to()

virtual result<extent_pair> llfio_v2_xxx::file_handle::clone_extents_to ( extent_pair  extent, byte_io_handle &  dest, byte_io_handle::extent_type  destoffset, deadline  d = {}, bool  force_copy_now = false, bool  emulate_if_unsupported = true  )

inlinevirtualnoexceptinherited

Clones the extents referred to by extent to dest at destoffset. This is how you ought to copy file content, including within the same file. This is fundamentally a racy call with respect to concurrent modification of the files.

Some of the filesystems on the major operating systems implement copy-on-write extent reference counting, and thus can very cheaply link a "copy" of extents in one file into another file (at the time of writing - Linux: XFS, btrfs, ocfs2, smbfs; Mac OS: APFS; Windows: ReFS, CIFS). Upon first write into an extent, only then is a new copy formed for the specific extents being modified. Note that extent cloning is usually only possible in cluster sized amounts, so if the portion you clone is not so aligned, new extents will be allocated for the spill into non-aligned portions. Obviously, cloning an entire file in a single shot does not have that problem.

Networked filing systems typically can also implement remote extent copying, such that extents can be copied between files entirely upon the remote server, and avoiding the copy going over the network. This is usually far more efficient.

This implementation first enumerates the valid extents for the region requested, and only clones extents which are reported as valid. It then iterates the platform specific syscall to cause the extents to be cloned in utils::page_allocator<T> sized chunks (i.e. the next large page greater or equal to 1Mb). Generally speaking, if the dedicated syscalls fail, the implementation falls back to a user space emulation, unless emulate_if_unsupported is false.

If the region being cloned does not exist in the source file, the region is truncated to what is available. If the destination file is not big enough to receive the cloned region, it is extended. If the clone is occurring within the same inode, you should ensure that the regions do not overlap, as cloning regions which overlap has platform-specific semantics. If they do overlap, you should always set force_copy_now for portable code.

Note

The current implementation does not permit overlapping clones within the same inode to differ by less than utils::page_allocator<T> sized chunks. It will fail with an error code comparing equal to errc::invalid_parameter.

If you really want the copy to happen now, and not later via copy-on-write, set force_copy_now. Note that this forces emulate_if_unsupported to true.

If dest is not a file_handle, sendfile() is used and the destination offset and gaps in the source valid extents are ignored.

co_barrier()

awaitable<io_result<const_buffers_type> > llfio_v2_xxx::byte_io_handle::co_barrier ( io_request< const_buffers_type >  reqs = io_request<const_buffers_type>(), barrier_kind  kind = barrier_kind::nowait_data_only, deadline  d = deadline()  )

inlinenoexceptinherited

A coroutinised equivalent to .barrier() which suspends the coroutine until the i/o finishes. Blocks execution i.e is equivalent to .barrier() if no i/o multiplexer has been set on this handle!

The awaitable returned is eager i.e. it immediately begins the i/o. If the i/o completes and finishes immediately, no coroutine suspension occurs.

  {
    if(_ctx == nullptr)
    {
      return awaitable<io_result<const_buffers_type>>(barrier(std::move(reqs), kind, d));
    }
    awaitable<io_result<const_buffers_type>> ret;
    ret.set_state(_ctx->construct(ret._state_storage, this, nullptr, {}, d, std::move(reqs), kind));
    return ret;
  }

co_read()

awaitable<io_result<buffers_type> > llfio_v2_xxx::byte_io_handle::co_read ( io_request< buffers_type >  reqs, deadline  d = deadline()  )

inlinenoexceptinherited

A coroutinised equivalent to .read() which suspends the coroutine until the i/o finishes. Blocks execution i.e is equivalent to .read() if no i/o multiplexer has been set on this handle!

The awaitable returned is eager i.e. it immediately begins the i/o. If the i/o completes and finishes immediately, no coroutine suspension occurs.

  {
    if(_ctx == nullptr)
    {
      return awaitable<io_result<buffers_type>>(read(std::move(reqs), d));
    }
    awaitable<io_result<buffers_type>> ret;
    ret.set_state(_ctx->construct(ret._state_storage, this, nullptr, {}, d, std::move(reqs)));
    return ret;
  }

co_write()

awaitable<io_result<const_buffers_type> > llfio_v2_xxx::byte_io_handle::co_write ( io_request< const_buffers_type >  reqs, deadline  d = deadline()  )

inlinenoexceptinherited

A coroutinised equivalent to .write() which suspends the coroutine until the i/o finishes. Blocks execution i.e is equivalent to .write() if no i/o multiplexer has been set on this handle!

The awaitable returned is eager i.e. it immediately begins the i/o. If the i/o completes and finishes immediately, no coroutine suspension occurs.

  {
    if(_ctx == nullptr)
    {
      return awaitable<io_result<const_buffers_type>>(write(std::move(reqs), d));
    }
    awaitable<io_result<const_buffers_type>> ret;
    ret.set_state(_ctx->construct(ret._state_storage, this, nullptr, {}, d, std::move(reqs)));
    return ret;
  }

copy_extended_attributes()

result<size_t> llfio_v2_xxx::fs_handle::copy_extended_attributes ( const fs_handle &  src, bool  replace_all_local_attributes = false  )

inlinenoexceptinherited

Copies the extended attributes from one entity to another, optionally replacing all the extended attributes on the destination.

This convenience function is implemented using the APIs above, and therefore is racy with respect to concurrent users. If you specifiy an invalid source with replace_all_local_attributes = true, then this is a convenient way to remove all extended attributes on the local inode.

Note

This function uses 130Kb of stack and cannot handle attribute values larger than 64Kb.

  {
    auto &h = _get_handle();
    (void) h;
    LLFIO_LOG_FUNCTION_CALL(&h);
    byte buffer[65536 + 4096];
    if(replace_all_local_attributes)
    {
      for(;;)
      {
        OUTCOME_TRY(auto &&attribs, list_extended_attributes(buffer));
        if(attribs.empty())
        {
          break;
        }
        for(auto &attrib : attribs)
        {
          OUTCOME_TRY(remove_extended_attribute(attrib));
        }
      }
    }
    if(src._get_handle().is_valid())
    {
      OUTCOME_TRY(auto &&attribs, src.list_extended_attributes(buffer));
      for(auto &attrib : attribs)
      {
        byte buffer2[65536];
        OUTCOME_TRY(auto &&value, src.get_extended_attribute(buffer2, attrib));
        OUTCOME_TRY(set_extended_attribute(attrib, value));
      }
    }
    return success();
  }

current_path()

virtual result<path_type> llfio_v2_xxx::handle::current_path ( ) const

inlinevirtualnoexceptinherited

Returns the current path of the open handle as said by the operating system. Note that you are NOT guaranteed that any path refreshed bears any resemblance to the original, some operating systems will return some different path which still reaches the same inode via some other route e.g. hardlinks, dereferenced symbolic links, etc. Windows and Linux correctly track changes to the specific path the handle was opened with, not getting confused by other hard links. MacOS nearly gets it right, but under some circumstances e.g. renaming may switch to a different hard link's path which is almost certainly a bug.

If LLFIO was not able to determine the current path for this open handle e.g. the inode has been unlinked, it returns an empty path. Be aware that FreeBSD can return an empty (deleted) path for file inodes no longer cached by the kernel path cache, LLFIO cannot detect the difference. FreeBSD will also return any path leading to the inode if it is hard linked. FreeBSD does implement path retrieval for directory inodes correctly however, and see algorithm::cached_parent_handle_adapter<T> for a handle adapter which makes use of that.

On Linux if /proc is not mounted, this call fails with an error. All APIs in LLFIO which require the use of current_path() can be told to not use it e.g. flag::disable_safety_unlinks. It is up to you to detect if current_path() is not working, and to change how you call LLFIO appropriately.

On Windows, you will almost certainly get back a path of the form \!!\Device\HarddiskVolume10\Users\ned\.... See path_view for what all the path prefix sequences mean, but to summarise the \!!\ prefix is LLFIO-only and will not be accepted by other Windows APIs. Pass LLFIO derived paths through the function to_win32_path() to Win32-ise them. This function is also available on Linux where it does nothing, so you can use it in portable code.

Warning

This call is expensive, it always asks the kernel for the current path, and no checking is done to ensure what the kernel returns is accurate or even sensible. Be aware that despite these precautions, paths are unstable and can change randomly at any moment. Most code written to use absolute file systems paths is racy, so don't do it, use path_handle to fix a base location on the file system and work from that anchor instead!

Memory Allocations\n At least one malloc for the path_type, likely several more.

See also

algorithm::cached_parent_handle_adapter<T> which overrides this with an implementation based on retrieving the current path of a cached handle to the parent directory. On platforms with instability or failure to retrieve the correct current path for regular files, the cached parent handle adapter works around the problem by taking advantage of directory inodes not having the same instability problems on any platform.

Reimplemented in llfio_v2_xxx::symlink_handle, and llfio_v2_xxx::process_handle.

extents()

virtual result<std::vector<extent_pair> > llfio_v2_xxx::file_handle::extents ( ) const

inlinevirtualnoexceptinherited

Returns a list of currently valid extents for this open file. WARNING: racy!

Returns

A vector of pairs of extent offset + extent length representing the valid extents in this file. Filing systems which do not support extents return a single extent matching the length of the file rather than returning an error.

Reimplemented in llfio_v2_xxx::fast_random_file_handle.

file()

static result<file_handle> llfio_v2_xxx::file_handle::file ( const path_handle &  base, path_view_type  path, mode  _mode = mode::read, creation  _creation = creation::open_existing, caching  _caching = caching::all, flag  flags = flag::none  )

inlinestaticnoexceptinherited

Create a file handle opening access to a file on path

Parameters

base	Handle to a base location on the filing system. Pass {} to indicate that path will be absolute.
path	The path relative to base to open.
_mode	How to open the file.
_creation	How to create the file.
_caching	How to ask the kernel to cache the file.
flags	Any additional custom behaviours.

Errors returnable\n Any of the values POSIX open() or CreateFile() can return.

get_extended_attribute()

virtual result<span<byte> > llfio_v2_xxx::fs_handle::get_extended_attribute ( span< byte >  tofill, path_view_component  name  ) const

inlinevirtualnoexceptinherited

Retrieve the value of an extended attribute set on this file or directory.

Note

On Windows, this is the list of alternate streams on a file, NOT NTFS extended attributes.

link()

virtual result<void> llfio_v2_xxx::fs_handle::link ( const path_handle &  base, path_view_type  path, deadline  d = std::chrono::seconds(30)  )

inlinevirtualnoexceptinherited

Links the inode referred to by this open handle to the path specified. The current path of this open handle is not changed, unless it has no current path due to being unlinked.

Warning

Some operating systems provide a race free syscall for linking an open handle to a new location (Linux, Windows). On all other operating systems this call is racy and can result in the wrong inode being linked. Note that unless flag::disable_safety_unlinks is set, this implementation opens a path_handle to the source containing directory first, then checks before linking that the item about to be hard linked has the same inode as the open file handle. It will retry this matching until success until the deadline given. This should prevent most unmalicious accidental loss of data.

Parameters

base	Base for any relative path.
path	The relative or absolute new path to hard link to.
d	The deadline by which the matching of the containing directory to the open handle's inode must succeed, else errc::timed_out will be returned.

Memory Allocations\n Except on platforms with race free syscalls for renaming open handles (Windows), calls

current_path() via parent_path_handle() and thus is both expensive and calls malloc many times.

list_extended_attributes()

virtual result<span<path_view_component> > llfio_v2_xxx::fs_handle::list_extended_attributes ( span< byte >  tofill ) const

inlinevirtualnoexceptinherited

Fill the supplied buffer with the names of all extended attributes set on this file or directory, returning a span of path view components.

Note that this routine is a very thin wrap of listxattr() on POSIX and NtQueryInformationFile() on Windows. If the supplied buffer is too small, the syscall typically returns failure rather than do a partial fill. Most implementations do not support more than 64Kb of extended attribute information per inode so maybe 70Kb is a safe default (to account for the return value storage), however properly written code will detect the buffer being too small and will auto-expand it until success.

Note

On Windows, this is the list of alternate streams on a file, NOT NTFS extended attributes.

Race Guarantees\n The list of extended attributes is fetched in a single syscall. This may be an

atomically consistent snapshot.

lock_file()

virtual result<void> llfio_v2_xxx::lockable_byte_io_handle::lock_file ( )

inlinevirtualnoexceptinherited

Locks the inode referred to by the open handle for exclusive access.

Note that this may, or may not, interact with the byte range lock extensions. See unique_file_lock for a RAII locker.

Errors returnable\n Any of the values POSIX flock() can return.

Memory Allocations\n The default synchronous implementation in file_handle performs no memory allocation.

lock_file_range()

virtual result<extent_guard> llfio_v2_xxx::lockable_byte_io_handle::lock_file_range ( extent_type  offset, extent_type  bytes, lock_kind  kind, deadline  d = deadline()  )

inlinevirtualnoexceptinherited

EXTENSION: Tries to lock the range of bytes specified for shared or exclusive access. Note that this may, or MAY NOT, observe whole file locks placed with lock(), lock_shared() etc.

Be aware this passes through the same semantics as the underlying OS call, including any POSIX insanity present on your platform:

• Any fd closed on an inode must release all byte range locks on that inode for all other fds. If your OS isn't new enough to support the non-insane lock API, flag::byte_lock_insanity will be set in flags() after the first call to this function.
• Threads replace each other's locks, indeed locks replace each other's locks.

You almost cetainly should use your choice of an algorithm::shared_fs_mutex::* instead of this as those are more portable and performant, or use the SharedMutex modelling member functions which lock the whole inode for exclusive or shared access.

Warning

This is a low-level API which you should not use directly in portable code. Another issue is that atomic lock upgrade/downgrade, if your platform implements that (you should assume it does not in portable code), means that on POSIX you need to release the old extent_guard after creating a new one over the same byte range, otherwise the old extent_guard's destructor will simply unlock the range entirely. On Windows however upgrade/downgrade locks overlay, so on that platform you must not release the old extent_guard. Look into algorithm::shared_fs_mutex::safe_byte_ranges for a portable solution.

Returns

An extent guard, the destruction of which will call unlock().

Parameters

offset	The offset to lock. Note that on POSIX the top bit is always cleared before use as POSIX uses signed transport for offsets. If you want an advisory rather than mandatory lock on Windows, one technique is to force top bit set so the region you lock is not the one you will i/o - obviously this reduces maximum file size to (2^63)-1.
bytes	The number of bytes to lock. Setting this and the offset to zero causes the whole file to be locked.
kind	Whether the lock is to be shared or exclusive.
d	An optional deadline by which the lock must complete, else it is cancelled.

Errors returnable\n Any of the values POSIX fcntl() can return, errc::timed_out, errc::not_supported may be

returned if deadline i/o is not possible with this particular handle configuration (e.g. non-overlapped HANDLE on Windows).

Memory Allocations\n The default synchronous implementation in file_handle performs no memory allocation.

Reimplemented in llfio_v2_xxx::fast_random_file_handle.

lock_file_shared()

virtual result<void> llfio_v2_xxx::lockable_byte_io_handle::lock_file_shared ( )

inlinevirtualnoexceptinherited

Locks the inode referred to by the open handle for shared access.

Note that this may, or may not, interact with the byte range lock extensions. See unique_file_lock for a RAII locker.

Errors returnable\n Any of the values POSIX flock() can return.

Memory Allocations\n The default synchronous implementation in file_handle performs no memory allocation.

mapped_file()

static result<mapped_file_handle> llfio_v2_xxx::mapped_file_handle::mapped_file ( size_type  reservation, const path_handle &  base, path_view_type  _path, mode  _mode = mode::read, creation  _creation = creation::open_existing, caching  _caching = caching::all, flag  flags = flag::none, section_handle::flag  sflags = section_handle::flag::none, extent_type  offset = 0  )

inlinestaticnoexcept

Create a memory mapped file handle opening access to a file on path.

Parameters

reservation	The number of bytes to reserve for later expansion when mapping. Zero means reserve only the current file length.
base	Handle to a base location on the filing system. Pass {} to indicate that path will be absolute.
_path	The path relative to base to open.
_mode	How to open the file.
_creation	How to create the file.
_caching	How to ask the kernel to cache the file.
flags	Any additional custom behaviours.
sflags	Any additional custom behaviours for the internal section_handle.
offset	The offset into the backing file for the map.

Note that if the file is currently zero sized, no mapping occurs now, but later when truncate() or update_map() is called.

Errors returnable\n Any of the values which the constructors for file_handle, section_handle and map_handle can return.

  {
    try
    {
      if(_mode == mode::append)
      {
        return errc::invalid_argument;
      }
      OUTCOME_TRY(auto &&fh, file_handle::file(base, _path, _mode, _creation, _caching, flags));
      switch(_creation)
      {
      default:
      {
        // Attempt mapping now (may silently fail if file is empty)
        mapped_file_handle mfh(std::move(fh), reservation, sflags, offset);
        return {std::move(mfh)};
      }
      case creation::only_if_not_exist:
      case creation::truncate_existing:
      case creation::always_new:
      {
        // Don't attempt mapping now as file will be empty
        mapped_file_handle mfh(std::move(fh), sflags, offset);
        mfh._reservation = reservation;
        return {std::move(mfh)};
      }
      }
    }
    catch(...)
    {
      return error_from_exception();
    }
  }

mapped_temp_file()

static result<mapped_file_handle> llfio_v2_xxx::mapped_file_handle::mapped_temp_file ( size_type  reservation, path_view_type  name = path_view_type(), mode  _mode = mode::write, creation  _creation = creation::if_needed, caching  _caching = caching::temporary, flag  flags = flag::unlink_on_first_close, section_handle::flag  sflags = section_handle::flag::none  )

inlinestaticnoexcept

Create a mapped file handle creating the named file on some path which the OS declares to be suitable for temporary files. Most OSs are very lazy about flushing changes made to these temporary files. Note the default flags are to have the newly created file deleted on first handle close. Note also that an empty name is equivalent to calling mapped_uniquely_named_file(path_discovery::storage_backed_temporary_files_directory()) and the creation parameter is ignored.

Note

If the temporary file you are creating is not going to have its path sent to another process for usage, this is the WRONG function to use. Use temp_inode() instead, it is far more secure.

Errors returnable\n Any of the values POSIX open() or CreateFile() can return.

  {
    auto &tempdirh = path_discovery::storage_backed_temporary_files_directory();
    return name.empty() ? mapped_uniquely_named_file(reservation, tempdirh, _mode, _caching, flags, sflags) :
                          mapped_file(reservation, tempdirh, name, _mode, _creation, _caching, flags, sflags);
  }

mapped_temp_inode()

static result<mapped_file_handle> llfio_v2_xxx::mapped_file_handle::mapped_temp_inode ( size_type  reservation = 0, const path_handle &  dir = path_discovery::storage_backed_temporary_files_directory(), mode  _mode = mode::write, caching  _caching = caching::temporary, flag  flags = flag::none, section_handle::flag  sflags = section_handle::flag::none  )

inlinestaticnoexcept

Securely create a mapped file handle creating a temporary anonymous inode in the filesystem referred to by dirpath. The inode created has no name nor accessible path on the filing system and ceases to exist as soon as the last handle is closed, making it ideal for use as a temporary file where other processes do not need to have access to its contents via some path on the filing system (a classic use case is for backing shared memory maps).

Errors returnable\n Any of the values POSIX open() or CreateFile() can return.

  {
    try
    {
      OUTCOME_TRY(auto &&v, file_handle::temp_inode(dir, _mode, _caching, flags));
      mapped_file_handle ret(std::move(v), sflags, 0);
      ret._reservation = reservation;
      return {std::move(ret)};
    }
    catch(...)
    {
      return error_from_exception();
    }
  }

mapped_uniquely_named_file()

static result<mapped_file_handle> llfio_v2_xxx::mapped_file_handle::mapped_uniquely_named_file ( size_type  reservation, const path_handle &  dirpath, mode  _mode = mode::write, caching  _caching = caching::temporary, flag  flags = flag::none, section_handle::flag  sflags = section_handle::flag::none  )

inlinestaticnoexcept

Create an mapped file handle creating a uniquely named file on a path. The file is opened exclusively with creation::only_if_not_exist so it will never collide with nor overwrite any existing file. Note also that caching defaults to temporary which hints to the OS to only flush changes to physical storage as lately as possible.

Errors returnable\n Any of the values POSIX open() or CreateFile() can return.

  {
    try
    {
      for(;;)
      {
        auto randomname = utils::random_string(32);
        randomname.append(".random");
        result<mapped_file_handle> ret = mapped_file(reservation, dirpath, randomname, _mode, creation::only_if_not_exist, _caching, flags, sflags);
        if(ret || (!ret && ret.error() != errc::file_exists))
        {
          return ret;
        }
      }
    }
    catch(...)
    {
      return error_from_exception();
    }
  }

max_buffers()

size_t llfio_v2_xxx::byte_io_handle::max_buffers ( ) const

inlinenoexceptinherited

The maximum number of buffers which a single read or write syscall can (atomically) process at a time for this specific open handle. On POSIX, this is known as IOV_MAX. Preferentially uses any i/o multiplexer set over the virtually overridable per-class implementation.

Note that the actual number of buffers accepted for a read or a write may be significantly lower than this system-defined limit, depending on available resources. The read() or write() call will return the buffers accepted at the time of invoking the syscall.

Note also that some OSs will error out if you supply more than this limit to read() or write(), but other OSs do not. Some OSs guarantee that each i/o syscall has effects atomically visible or not to other i/o, other OSs do not.

OS X does not implement scatter-gather file i/o syscalls. Thus this function will always return 1 in that situation.

Microsoft Windows may implement scatter-gather i/o under certain handle configurations. Most of the time for non-socket handles this function will return 1.

For handles which implement i/o entirely in user space, and thus syscalls are not involved, this function will return 0.

  {
    if(_ctx == nullptr)
    {
      return _do_max_buffers();
    }
    return _ctx->do_byte_io_handle_max_buffers(this);
  }

maximum_extent()

virtual result<extent_type> llfio_v2_xxx::mapped_file_handle::maximum_extent ( ) const

inlineoverridevirtualnoexcept

Return the current maximum permitted extent of the file, after updating the map.

Firstly calls update_map() to efficiently update the map to match that of the underlying file, then returns the number of bytes in the map which are valid to access. Because of the call to update_map(), this call is not particularly efficient, and you ought to cache its value where possible.

Reimplemented from llfio_v2_xxx::file_handle.

  {
    return (0 == _reservation) ? underlying_file_maximum_extent() : const_cast<mapped_file_handle *>(this)->update_map();
  }

parent_path_handle()

virtual result<path_handle> llfio_v2_xxx::fs_handle::parent_path_handle ( deadline  d = std::chrono::seconds(30) ) const

inlinevirtualnoexceptinherited

Obtain a handle to the path currently containing this handle's file entry.

Warning

This call is racy and can result in the wrong path handle being returned. Note that unless flag::disable_safety_unlinks is set, this implementation opens a path_handle to the source containing directory, then checks if the file entry within has the same inode as the open file handle. It will retry this matching until success until the deadline given.

Memory Allocations\n Calls current_path() and thus is both expensive and calls malloc many times.

See also

algorithm::cached_parent_handle_adapter<T> which overrides this with a zero cost implementation, thus making unlinking and relinking very considerably quicker.

QUICKCPPLIB_BITFIELD_BEGIN_T()

llfio_v2_xxx::handle::QUICKCPPLIB_BITFIELD_BEGIN_T ( flag  , uint16_t    )

inlineinherited

Bitwise flags which can be specified.

< No flags

Unlinks the file on handle close. On POSIX, this simply unlinks whatever is pointed to by path() upon the call of close() if and only if the inode matches. On Windows, if you are on Windows 10 1709 or later, exactly the same thing occurs. If on previous editions of Windows, the file entry does not disappears but becomes unavailable for anyone else to open with an errc::resource_unavailable_try_again error return. Because this is confusing, unless the win_disable_unlink_emulation flag is also specified, this POSIX behaviour is somewhat emulated by LLFIO on older Windows by renaming the file to a random name on close() causing it to appear to have been unlinked immediately.

Some kernel caching modes have unhelpfully inconsistent behaviours in getting your data onto storage, so by default unless this flag is specified LLFIO adds extra fsyncs to the following operations for the caching modes specified below: truncation of file length either explicitly or during file open. closing of the handle either explicitly or in the destructor.

Additionally on Linux only to prevent loss of file metadata: On the parent directory whenever a file might have been created. On the parent directory on file close.

This only occurs for these kernel caching modes: caching::none caching::reads caching::reads_and_metadata caching::safety_barriers

file_handle::unlink() could accidentally delete the wrong file if someone has renamed the open file handle since the time it was opened. To prevent this occuring, where the OS doesn't provide race free unlink-by-open-handle we compare the inode of the path we are about to unlink with that of the open handle before unlinking.

Warning

This does not prevent races where in between the time of checking the inode and executing the unlink a third party changes the item about to be unlinked. Only operating systems with a true race-free unlink syscall are race free.

Ask the OS to disable prefetching of data. This can improve random i/o performance.

Ask the OS to maximise prefetching of data, possibly prefetching the entire file into kernel cache. This can improve sequential i/o performance.

< See the documentation for unlink_on_first_close

Microsoft Windows NTFS, having been created in the late 1980s, did not originally implement extents-based storage and thus could only represent sparse files via efficient compression of intermediate zeros. With NTFS v3.0 (Microsoft Windows 2000), a proper extents-based on-storage representation was added, thus allowing only 64Kb extent chunks written to be stored irrespective of whatever the maximum file extent was set to.

For various historical reasons, extents-based storage is disabled by default in newly created files on NTFS, unlike in almost every other major filing system. You have to explicitly "opt in" to extents-based storage.

As extents-based storage is nearly cost free on NTFS, LLFIO by default opts in to extents-based storage for any empty file it creates. If you don't want this, you can specify this flag to prevent that happening.

Filesystems tend to be embarrassingly parallel for operations performed to different inodes. Where LLFIO performs i/o to multiple inodes at a time, it will use OpenMP or the Parallelism or Concurrency standard library extensions to usually complete the operation in constant rather than linear time. If you don't want this default, you can disable default using this flag.

Microsoft Windows NTFS has the option, when creating a directory, to set whether leafname lookup will be case sensitive. This is the only way of getting exact POSIX semantics on Windows without resorting to editing the system registry, however it also affects all code doing lookups within that directory, so we must default it to off.

Create the handle in a way where i/o upon it can be multiplexed with other i/o on the same initiating thread of execution i.e. you can perform more than one read concurrently, without using threads. The blocking operations .read() and .write() may have to use a less efficient, but cancellable, blocking implementation for handles created in this way. On Microsoft Windows, this creates handles with OVERLAPPED semantics. On POSIX, this creates handles with nonblocking semantics for non-file handles such as pipes and sockets, however for file, directory and symlink handles it does not set nonblocking, as it is non-portable.

< Using insane POSIX byte range locks

< This is an inode created with no representation on the filing system

                                              {
  none = uint16_t(0),  //!< No flags
  /*! Unlinks the file on handle close. On POSIX, this simply unlinks whatever is pointed
  to by `path()` upon the call of `close()` if and only if the inode matches. On Windows,
  if you are on Windows 10 1709 or later, exactly the same thing occurs. If on previous
  editions of Windows, the file entry does not disappears but becomes unavailable for
  anyone else to open with an `errc::resource_unavailable_try_again` error return. Because this is confusing, unless the
  `win_disable_unlink_emulation` flag is also specified, this POSIX behaviour is
  somewhat emulated by LLFIO on older Windows by renaming the file to a random name on `close()`
  causing it to appear to have been unlinked immediately.
  */
  unlink_on_first_close = uint16_t(1U << 0U),
 
  /*! Some kernel caching modes have unhelpfully inconsistent behaviours
  in getting your data onto storage, so by default unless this flag is
  specified LLFIO adds extra fsyncs to the following operations for the
  caching modes specified below:
  * truncation of file length either explicitly or during file open.
  * closing of the handle either explicitly or in the destructor.
 
  Additionally on Linux only to prevent loss of file metadata:
  * On the parent directory whenever a file might have been created.
  * On the parent directory on file close.
 
  This only occurs for these kernel caching modes:
  * caching::none
  * caching::reads
  * caching::reads_and_metadata
  * caching::safety_barriers
  */
  disable_safety_barriers = uint16_t(1U << 2U),
  /*! `file_handle::unlink()` could accidentally delete the wrong file if someone has
  renamed the open file handle since the time it was opened. To prevent this occuring,
  where the OS doesn't provide race free unlink-by-open-handle we compare the inode of
  the path we are about to unlink with that of the open handle before unlinking.
  \warning This does not prevent races where in between the time of checking the inode
  and executing the unlink a third party changes the item about to be unlinked. Only
  operating systems with a true race-free unlink syscall are race free.
  */
  disable_safety_unlinks = uint16_t(1U << 3U),
  /*! Ask the OS to disable prefetching of data. This can improve random
  i/o performance.
  */
  disable_prefetching = uint16_t(1U << 4U),
  /*! Ask the OS to maximise prefetching of data, possibly prefetching the entire file
  into kernel cache. This can improve sequential i/o performance.
  */
  maximum_prefetching = uint16_t(1U << 5U),
 
  win_disable_unlink_emulation = uint16_t(1U << 9U),  //!< See the documentation for `unlink_on_first_close`
  /*! Microsoft Windows NTFS, having been created in the late 1980s, did not originally
  implement extents-based storage and thus could only represent sparse files via
  efficient compression of intermediate zeros. With NTFS v3.0 (Microsoft Windows 2000),
  a proper extents-based on-storage representation was added, thus allowing only 64Kb
  extent chunks written to be stored irrespective of whatever the maximum file extent
  was set to.
 
  For various historical reasons, extents-based storage is disabled by default in newly
  created files on NTFS, unlike in almost every other major filing system. You have to
  explicitly "opt in" to extents-based storage.
 
  As extents-based storage is nearly cost free on NTFS, LLFIO by default opts in to
  extents-based storage for any empty file it creates. If you don't want this, you
  can specify this flag to prevent that happening.
  */
  win_disable_sparse_file_creation = uint16_t(1U << 10U),
  /*! Filesystems tend to be embarrassingly parallel for operations performed to different
  inodes. Where LLFIO performs i/o to multiple inodes at a time, it will use OpenMP or
  the Parallelism or Concurrency standard library extensions to usually complete the
  operation in constant rather than linear time. If you don't want this default, you can
  disable default using this flag.
  */
  disable_parallelism = uint16_t(1U << 11U),
  /*! Microsoft Windows NTFS has the option, when creating a directory, to set whether
  leafname lookup will be case sensitive. This is the only way of getting exact POSIX
  semantics on Windows without resorting to editing the system registry, however it also
  affects all code doing lookups within that directory, so we must default it to off.
  */
  win_create_case_sensitive_directory = uint16_t(1U << 12U),
 
  /*! Create the handle in a way where i/o upon it can be multiplexed with other i/o
  on the same initiating thread of execution i.e. you can perform more than one read
  concurrently, without using threads. The blocking operations `.read()` and `.write()`
  may have to use a less efficient, but cancellable, blocking implementation for handles created
  in this way. On Microsoft Windows, this creates handles with `OVERLAPPED` semantics.
  On POSIX, this creates handles with nonblocking semantics for non-file handles such
  as pipes and sockets, however for file, directory and symlink handles it does not set
  nonblocking, as it is non-portable.
  */
  multiplexable = uint16_t(1U << 13U),
 
  // NOTE: IF UPDATING THIS UPDATE THE std::ostream PRINTER BELOW!!!
 
  byte_lock_insanity = uint16_t(1U << 14U),  //!< Using insane POSIX byte range locks
  anonymous_inode = uint16_t(1U << 15U)      //!< This is an inode created with no representation on the filing system
  } QUICKCPPLIB_BITFIELD_END(flag)

read()

io_result<buffers_type> llfio_v2_xxx::byte_io_handle::read ( io_request< buffers_type >  reqs, deadline  d = deadline()  )

inlinenoexceptinherited

Read data from the open handle, preferentially using any i/o multiplexer set over the virtually overridable per-class implementation.

Warning

Depending on the implementation backend, very different buffers may be returned than you supplied. You should always use the buffers returned and assume that they point to different memory and that each buffer's size will have changed.

Returns

The buffers read, which may not be the buffers input. The size of each scatter-gather buffer returned is updated with the number of bytes of that buffer transferred, and the pointer to the data may be completely different to what was submitted (e.g. it may point into a memory map).

Parameters

reqs	A scatter-gather and offset request.
d	An optional deadline by which the i/o must complete, else it is cancelled. Note function may return significantly after this deadline if the i/o takes long to cancel.

Errors returnable\n Any of the values POSIX read() can return, errc::timed_out, errc::operation_canceled. errc::not_supported may be

returned if deadline i/o is not possible with this particular handle configuration (e.g. reading from regular files on POSIX or reading from a non-overlapped HANDLE on Windows).

Memory Allocations\n The default synchronous implementation in file_handle performs no memory allocation.

  {
    return (_ctx == nullptr) ? _do_read(reqs, d) : _do_multiplexer_read({}, reqs, d);
  }

relink()

virtual result<void> llfio_v2_xxx::mapped_file_handle::relink ( const path_handle &  base, path_view_type  path, bool  atomic_replace = true, deadline  d = std::chrono::seconds(30)  )

inlineoverridevirtualnoexcept

Relinks the current path of this open handle to the new path specified. If atomic_replace is true, the relink atomically and silently replaces any item at the new path specified. This operation is both atomic and matching POSIX behaviour even on Microsoft Windows where no Win32 API can match POSIX semantics.

Note that if atomic_replace is false, the operation may be implemented as creating a hard link to the destination (which fails if the destination exists), opening a new file descriptor to the destination, closing the existing file descriptor, replacing the existing file descriptor with the new one (this is to ensure path tracking continues to work), then unlinking the previous link. Thus native_handle()'s value may change. This is not the case on Microsoft Windows nor Linux, both of which provide syscalls capable of refusing to rename if the destination exists.

If the handle refers to a pipe, on Microsoft Windows the base path handle is ignored as there is a single global named pipe namespace. Unless the path fragment begins with \, the string \??\ is prefixed to the name before passing it to the NT kernel API which performs the rename. This is because \\.\ in Win32 maps onto \??\ in the NT kernel.

Warning

Some operating systems provide a race free syscall for renaming an open handle (Windows). On all other operating systems this call is racy and can result in the wrong file entry being relinked. Note that unless flag::disable_safety_unlinks is set, this implementation opens a path_handle to the source containing directory first, then checks before relinking that the item about to be relinked has the same inode as the open file handle. It will retry this matching until success until the deadline given. This should prevent most unmalicious accidental loss of data.

Parameters

base	Base for any relative path.
path	The relative or absolute new path to relink to.
atomic_replace	Atomically replace the destination if a file entry already is present there. Choosing false for this will fail if a file entry is already present at the destination, and may not be an atomic operation on some platforms (i.e. both the old and new names may be linked to the same inode for a very short period of time). Windows and recent Linuxes are always atomic.
d	The deadline by which the matching of the containing directory to the open handle's inode must succeed, else errc::timed_out will be returned.

Memory Allocations\n Except on platforms with race free syscalls for renaming open handles (Windows), calls

current_path() via parent_path_handle() and thus is both expensive and calls malloc many times.

Reimplemented from llfio_v2_xxx::fs_handle.

remove_extended_attribute()

virtual result<void> llfio_v2_xxx::fs_handle::remove_extended_attribute ( path_view_component  )

inlinevirtualnoexceptinherited

Removes the extended attribute set on this file or directory.

Note

On Windows, this is the list of alternate streams on a file, NOT NTFS extended attributes. We do not prevent you trying to remove internal alternate streams, either.

reopen()

result<file_handle> llfio_v2_xxx::file_handle::reopen ( mode  mode_ = mode::unchanged, caching  caching_ = caching::unchanged, deadline  d = std::chrono::seconds(30)  ) const

inlinenoexceptinherited

Reopen this handle (copy constructor is disabled to avoid accidental copying), optionally race free reopening the handle with different access or caching.

Microsoft Windows provides a syscall for cloning an existing handle but with new access. On POSIX, if not changing the mode, we change caching via fcntl(), if changing the mode we must loop calling current_path(), trying to open the path returned and making sure it is the same inode.

Errors returnable\n Any of the values POSIX dup() or DuplicateHandle() can return.

Memory Allocations\n On POSIX if changing the mode, we must loop calling current_path() and

trying to open the path returned. Thus many allocations may occur.

reserve()

result<size_type> llfio_v2_xxx::mapped_file_handle::reserve ( size_type  reservation = 0 )

inlinenoexcept

Reserve a new amount of address space for mapping future expansion of this file.

Parameters

reservation

The number of bytes of virtual address space to reserve. Zero means reserve the current length of the underlying file.

Note that this is an expensive call, and address() may return a different value afterwards. This call will fail if the underlying file has zero length.

  {
    auto length = (extent_type) -1;
    return _reserve(length, reservation);
  }

set_append_only()

virtual result<void> llfio_v2_xxx::handle::set_append_only ( bool  enable )

inlinevirtualnoexceptinherited

EXTENSION: Changes whether this handle is append only or not.

Warning

On Windows this is implemented as a bit of a hack to make it fast like on POSIX, so make sure you open the handle for read/write originally. Note unlike on POSIX the append_only disposition will be the only one toggled, seekable and readable will remain turned on.

Errors returnable\n Whatever POSIX fcntl() returns. On Windows nothing is changed on the handle.

Memory Allocations\n No memory allocation.

Reimplemented in llfio_v2_xxx::process_handle.

set_extended_attribute()

virtual result<void> llfio_v2_xxx::fs_handle::set_extended_attribute ( path_view_component  name, span< const byte >  value  )

inlinevirtualnoexceptinherited

Sets the value of an extended attribute on this file or directory.

To prevent collision in a globally visible resource, there is a convention whereby you ought to namespace the names of your values as namespace.attribute e.g. appname.setting to prevent unintentional collision with other programs. Obviously, do choose a unique appname if there is any chance another program might use the same namespace name.

On POSIX, there are additional namespacing requirements: before your value name, you need to prefix one of user or system, so the actual name you might set would be user.appname.propname. Windows does not have the user/system prefix requirement, but it does no harm to do the exact same on Windows as on POSIX.

The host OS and target filing system choose the limits on value size, and will fail accordingly. Some impose a maximum of 64Kb for all names and values per inode, others have a 4Kb maximum value size, there are lots of combinations. You are probably safest not setting many names, and keep the values short.

Warning

Extended attributes are 'brittle' because they can get silently wiped at any moment. Never store anything in extended attributes which cannot be recalculated if missing. The ideal use case for extended attributes is as a cache of additional metadata about a file or directory e.g. "I last checked this directory at timestamp X", or "the MD5 hash at last modified timestamp X for this file was Y". Also remember that other processes can and do arbitrarily modify extended attributes concurrent to you.

Windows only

This API is implemented as file alternate data streams, rather than the Extended Attributes API as accessed via NtSetEaFile() and NtQueryEaFile() (which actually modify the file alternate data stream $EA in any case).

The reason why is that NtSetEaFile() can only append new records to EA storage. It cannot deallocate any existing EA records, if you try to do so you will get STATUS_EA_CORRUPT_ERROR. You can append setting the same name to a different value, which can include a null value which then appears as if the name is no longer there. But there is a cap of 64kB for the EA record, and once it is consumed, it is gone forever for that inode.

Obviously that doesn't map at all well onto POSIX extended attributes, where you can set the value of an attribute as frequently as you like. The closest equivalent on Windows is therefore file alternate data streams, even though the attribute's value is then worked with as a whole proper file with all the attendant performance consequences.

As a result, name must be a valid filename and not contain any characters not permitted in a filename. We use the NT API here, so the character restrictions are far fewer than for the Win32 API e.g. single character names do NOT cause misoperation like on Win32.

set_multiplexer()

virtual result<void> llfio_v2_xxx::mapped_file_handle::set_multiplexer ( byte_io_multiplexer *  c = this_thread::multiplexer() )

inlineoverridevirtualnoexcept

Sets the i/o multiplexer this handle will use to implement read(), write() and barrier().

Note that this call deregisters this handle from any existing i/o multiplexer, and registers it with the new i/o multiplexer. You must therefore not call it if any i/o is currently outstanding on this handle. You should also be aware that multiple dynamic memory allocations and deallocations may occur, as well as multiple syscalls (i.e. this is an expensive call, try to do it from cold code).

If the handle was not created as multiplexable, this call always fails.

Memory Allocations\n Multiple dynamic memory allocations and deallocations.

Reimplemented from llfio_v2_xxx::byte_io_handle.

  {
    OUTCOME_TRY(file_handle::set_multiplexer(c));
    return _mh.set_multiplexer(file_handle::multiplexer());
  }

temp_file()

static result<file_handle> llfio_v2_xxx::file_handle::temp_file ( path_view_type  name = path_view_type(), mode  _mode = mode::write, creation  _creation = creation::if_needed, caching  _caching = caching::temporary, flag  flags = flag::unlink_on_first_close  )

inlinestaticnoexceptinherited

Create a file handle creating the named file on some path which the OS declares to be suitable for temporary files. Most OSs are very lazy about flushing changes made to these temporary files. Note the default flags are to have the newly created file deleted on first handle close. Note also that an empty name is equivalent to calling uniquely_named_file(path_discovery::storage_backed_temporary_files_directory()) and the creation parameter is ignored.

Note

If the temporary file you are creating is not going to have its path sent to another process for usage, this is the WRONG function to use. Use temp_inode() instead, it is far more secure.

Errors returnable\n Any of the values POSIX open() or CreateFile() can return.

  {
    auto &tempdirh = path_discovery::storage_backed_temporary_files_directory();
    return name.empty() ? uniquely_named_file(tempdirh, _mode, _caching, flags) : file(tempdirh, name, _mode, _creation, _caching, flags);
  }

temp_inode()

static result<file_handle> llfio_v2_xxx::file_handle::temp_inode ( const path_handle &  dirh = path_discovery::storage_backed_temporary_files_directory(), mode  _mode = mode::write, caching  _caching = caching::temporary, flag  flags = flag::none  )

inlinestaticnoexceptinherited

Securely create a file handle creating a temporary anonymous inode in the filesystem referred to by dirpath. The inode created has no name nor accessible path on the filing system and ceases to exist as soon as the last handle is closed, making it ideal for use as a temporary file where other processes do not need to have access to its contents via some path on the filing system (a classic use case is for backing shared memory maps).

Errors returnable\n Any of the values POSIX open() or CreateFile() can return.

truncate()

virtual result<extent_type> llfio_v2_xxx::mapped_file_handle::truncate ( extent_type  newsize )

inlineoverridevirtualnoexcept

Resize the current maximum permitted extent of the mapped file to the given extent, avoiding any new allocation of physical storage where supported, and mapping or unmapping any new pages up to the reservation to reflect the new maximum extent. If the new size exceeds the reservation, reserve() will be called to increase the reservation.

Note that on extents based filing systems this will succeed even if there is insufficient free space on the storage medium. Only when pages are written to will the lack of sufficient free space be realised, resulting in an operating system specific exception.

Note

On Microsoft Windows you cannot shrink a file with a section handle open on it in any process in the system. We therefore always destroy the internal map and section before truncating, and then recreate the map and section afterwards if the new size is not zero. address() therefore may change. You will need to ensure all other users of the same file close their section and map handles before any process can shrink the underlying file.

Returns

The bytes actually truncated to.

Parameters

newsize

The bytes to truncate the file to. Zero causes the maps to be closed before truncation.

Reimplemented from llfio_v2_xxx::file_handle.

try_lock_file()

virtual bool llfio_v2_xxx::lockable_byte_io_handle::try_lock_file ( )

inlinevirtualnoexceptinherited

Tries to lock the inode referred to by the open handle for exclusive access, returning false if lock is currently unavailable.

Note that this may, or may not, interact with the byte range lock extensions. See unique_file_lock for a RAII locker.

Errors returnable\n Any of the values POSIX flock() can return.

Memory Allocations\n The default synchronous implementation in file_handle performs no memory allocation.

try_lock_file_shared()

virtual bool llfio_v2_xxx::lockable_byte_io_handle::try_lock_file_shared ( )

inlinevirtualnoexceptinherited

Tries to lock the inode referred to by the open handle for shared access, returning false if lock is currently unavailable.

Note that this may, or may not, interact with the byte range lock extensions. See unique_file_lock for a RAII locker.

Errors returnable\n Any of the values POSIX flock() can return.

Memory Allocations\n The default synchronous implementation in file_handle performs no memory allocation.

uniquely_named_file()

static result<file_handle> llfio_v2_xxx::file_handle::uniquely_named_file ( const path_handle &  dirpath, mode  _mode = mode::write, caching  _caching = caching::temporary, flag  flags = flag::none  )

inlinestaticnoexceptinherited

Create a file handle creating a uniquely named file on a path. The file is opened exclusively with creation::only_if_not_exist so it will never collide with nor overwrite any existing file. Note also that caching defaults to temporary which hints to the OS to only flush changes to physical storage as lately as possible.

Errors returnable\n Any of the values POSIX open() or CreateFile() can return.

  {
    try
    {
      for(;;)
      {
        auto randomname = utils::random_string(32);
        randomname.append(".random");
        result<file_handle> ret = file(dirpath, randomname, _mode, creation::only_if_not_exist, _caching, flags);
        if(ret || (!ret && ret.error() != errc::file_exists))
        {
          return ret;
        }
      }
    }
    catch(...)
    {
      return error_from_exception();
    }
  }

unlink()

virtual result<void> llfio_v2_xxx::fs_handle::unlink ( deadline  d = std::chrono::seconds(30) )

inlinevirtualnoexceptinherited

Unlinks the current path of this open handle, causing its entry to immediately disappear from the filing system.

On Windows before Windows 10 1709 unless flag::win_disable_unlink_emulation is set, this behaviour is simulated by renaming the file to something random and setting its delete-on-last-close flag. Note that Windows may prevent the renaming of a file in use by another process, if so it will NOT be renamed. After the next handle to that file closes, it will become permanently unopenable by anyone else until the last handle is closed, whereupon the entry will be eventually removed by the operating system.

Warning

Some operating systems provide a race free syscall for unlinking an open handle (Windows). On all other operating systems this call is racy and can result in the wrong file entry being unlinked. Note that unless flag::disable_safety_unlinks is set, this implementation opens a path_handle to the containing directory first, then checks that the item about to be unlinked has the same inode as the open file handle. It will retry this matching until success until the deadline given. This should prevent most unmalicious accidental loss of data.

Parameters

d	The deadline by which the matching of the containing directory to the open handle's inode must succeed, else errc::timed_out will be returned.

Memory Allocations\n Except on platforms with race free syscalls for unlinking open handles (Windows), calls

current_path() and thus is both expensive and calls malloc many times. On Windows, also calls current_path() if flag::disable_safety_unlinks is not set.

Reimplemented in llfio_v2_xxx::symlink_handle.

unlock_file_range()

virtual void llfio_v2_xxx::lockable_byte_io_handle::unlock_file_range ( extent_type  offset, extent_type  bytes  )

inlinevirtualnoexceptinherited

EXTENSION: Unlocks a byte range previously locked.

Parameters

offset	The offset to unlock. This should be an offset previously locked.
bytes	The number of bytes to unlock. This should be a byte extent previously locked.

Errors returnable\n Any of the values POSIX fcntl() can return.

Memory Allocations\n None.

Reimplemented in llfio_v2_xxx::fast_random_file_handle.

update_map()

result<extent_type> llfio_v2_xxx::mapped_file_handle::update_map ( )

inlinenoexcept

Efficiently update the mapping to match that of the underlying file, returning the new current maximum permitted extent of the file.

This call is often considerably less heavyweight than truncate(newsize), and should be used where possible.

If the internal section and map handle are invalid, they are restored unless the underlying file is zero length.

If the size of the underlying file has become zero length, the internal section and map handle are closed.

This function never returns errc::not_enough_memory, even if it calls reserve().

write()

io_result<const_buffers_type> llfio_v2_xxx::byte_io_handle::write ( io_request< const_buffers_type >  reqs, deadline  d = deadline()  )

inlinenoexceptinherited

Write data to the open handle, preferentially using any i/o multiplexer set over the virtually overridable per-class implementation.

Warning

Depending on the implementation backend, not all of the buffers input may be written. For example, with a zeroed deadline, some backends may only consume as many buffers as the system has available write slots for, thus for those backends this call is "non-blocking" in the sense that it will return immediately even if it could not schedule a single buffer write. Another example is that some implementations will not auto-extend the length of a file when a write exceeds the maximum extent, you will need to issue a truncate(newsize) first.

Returns

The buffers written, which may not be the buffers input. The size of each scatter-gather buffer returned is updated with the number of bytes of that buffer transferred.

Parameters

reqs	A scatter-gather and offset request.
d	An optional deadline by which the i/o must complete, else it is cancelled. Note function may return significantly after this deadline if the i/o takes long to cancel.

Errors returnable\n Any of the values POSIX write() can return, errc::timed_out, errc::operation_canceled. errc::not_supported may be

returned if deadline i/o is not possible with this particular handle configuration (e.g. writing to regular files on POSIX or writing to a non-overlapped HANDLE on Windows).

Memory Allocations\n The default synchronous implementation in file_handle performs no memory allocation.

  {
    return (_ctx == nullptr) ? _do_write(reqs, d) : _do_multiplexer_write({}, std::move(reqs), d);
  }

zero()

virtual result<extent_type> llfio_v2_xxx::mapped_file_handle::zero ( extent_pair  extent, deadline  d = deadline()  )

inlineoverridevirtualnoexcept

Efficiently zero, and possibly deallocate, data on storage.

On most major operating systems and with recent filing systems which are "extents based", one can deallocate the physical storage of a file, causing the space deallocated to appear all bits zero. This call attempts to deallocate whole pages (usually 4Kb) entirely, and memset's any excess to all bits zero. This call works on most Linux filing systems with a recent kernel, Microsoft Windows with NTFS, and FreeBSD with ZFS. On other systems it simply writes zeros.

Returns

The bytes zeroed.

Parameters

extent	The offset to start zeroing from and the number of bytes to zero.
d	An optional deadline by which the i/o must complete, else it is cancelled. Note function may return significantly after this deadline if the i/o takes long to cancel.

Errors returnable\n Any of the values POSIX write() can return, errc::timed_out, errc::operation_canceled.

errc::not_supported may be returned if deadline i/o is not possible with this particular handle configuration (e.g. writing to regular files on POSIX or writing to a non-overlapped HANDLE on Windows).

Memory Allocations\n The default synchronous implementation in file_handle performs no memory allocation.

Reimplemented from llfio_v2_xxx::file_handle.

  {
    OUTCOME_TRYV(_mh.zero_memory({_mh.address() + extent.offset, (size_type) extent.length}));
    return extent.length;
  }

---

The documentation for this class was generated from the following file:

• include/llfio/v2.0/mapped_file_handle.hpp