llfio_v2_xxx::utils Namespace Reference

Utility routines often useful when using LLFIO. More...

Classes
struct	process_memory_usage
	Memory usage statistics for a process. More...
struct	process_cpu_usage
	CPU usage statistics for a process. More...
class	page_allocator
	An STL allocator which allocates large TLB page memory. More...
class	page_allocator< void >

Functions
size_t	page_size () noexcept
	Returns the smallest page size of this architecture which is useful for calculating direct i/o multiples. More...
template<class T >
T	round_down_to_page_size (T i, size_t pagesize) noexcept
	Round a value to its next lowest page size multiple.
template<class T >
T	round_up_to_page_size (T i, size_t pagesize) noexcept
	Round a value to its next highest page size multiple.
template<class T , typename = decltype( std::declval<T>().data() ), typename = decltype( std::declval<T>().size() )>
T	round_to_page_size_larger (T i, size_t pagesize) noexcept
	Round a pair of a pointer and a size_t to their nearest page size multiples. The pointer will be rounded down, the size_t upwards.
template<class T , typename = decltype( std::declval<T>().data() ), typename = decltype( std::declval<T>().size() )>
T	round_to_page_size_smaller (T i, size_t pagesize) noexcept
	Round a pair of a pointer and a size_t to their nearest page size multiples. The pointer will be rounded upwards, the size_t downwards.
template<class A , class B >
std::pair< A, B >	round_to_page_size_larger (std::pair< A, B > i, size_t pagesize) noexcept
	Round a pair of values to their nearest page size multiples. The first will be rounded down, the second upwards.
template<class A , class B >
std::pair< A, B >	round_to_page_size_smaller (std::pair< A, B > i, size_t pagesize) noexcept
	Round a pair of values to their nearest page size multiples. The first will be rounded upwards, the second downwards.
const std::vector< size_t > &	page_sizes (bool only_actually_available=true)
	Returns the page sizes of this architecture which is useful for calculating direct i/o multiples. More...
size_t	file_buffer_default_size ()
	Returns a reasonable default size for page_allocator, typically the closest page size from page_sizes() to 1Mb. More...
void	random_fill (char *buffer, size_t bytes) noexcept
	Fills the buffer supplied with cryptographically strong randomness. Uses the OS kernel API. More...
std::string	random_string (size_t randomlen)
	Returns a cryptographically random string capable of being used as a filename. Essentially random_fill() + to_hex_string(). More...
result< void >	flush_modified_data () noexcept
	Tries to flush all modified data to the physical device.
result< void >	drop_filesystem_cache () noexcept
	Tries to flush all modified data to the physical device, and then drop the OS filesystem cache, thus making all future reads come from the physical device. Currently only implemented for Microsoft Windows and Linux. More...
bool	running_under_wsl () noexcept
	Returns true if this POSIX is running under Microsoft's Subsystem for Linux.
result< process_memory_usage >	current_process_memory_usage (process_memory_usage::want want=process_memory_usage::want::this_process) noexcept
	Retrieve the current memory usage statistics for this process. More...
result< process_cpu_usage >	current_process_cpu_usage () noexcept
	Retrieve the current CPU usage statistics for this system and this process. These are unsigned counters which always increment, and so may eventually wrap. More...
template<class T , class U >
bool	operator== (const page_allocator< T > &, const page_allocator< U > &) noexcept

Detailed Description

Utility routines often useful when using LLFIO.

Function Documentation

current_process_cpu_usage()

result<process_cpu_usage> llfio_v2_xxx::utils::current_process_cpu_usage ( )

inlinenoexcept

Retrieve the current CPU usage statistics for this system and this process. These are unsigned counters which always increment, and so may eventually wrap.

The simplest way to use this API is to call it whilst also taking the current monotonic clock/CPU TSC and then calculating the delta change over that period of time.

Note

The returned values may not be a snapshot accurate against one another as they may get derived from multiple sources. Also, granularity is probably either a lot more than one nanosecond on most platforms, but may be CPU TSC based on others (you can test it to be sure).

Within some versions of Docker, the per-process counters are not available.

current_process_memory_usage()

result<process_memory_usage> llfio_v2_xxx::utils::current_process_memory_usage ( process_memory_usage::want  want = process_memory_usage::want::this_process )

inlinenoexcept

Retrieve the current memory usage statistics for this process.

Be aware that because Linux provides no summary counter for private_committed, we have to manually parse through /proc/pid/smaps to calculate it. This can start to take seconds for a process with a complex virtual memory space. If you are sure that you never use section_handle::flag::nocommit without section_handle::flag::none (i.e. you don't nocommit accessible memory), then specifying the flag process_memory_usage::want::private_committed_inaccurate can yield significant performance gains. If you set process_memory_usage::want::private_committed_inaccurate, we use /proc/pid/smaps_rollup and /proc/pid/maps to calculate the results. This cannot distinguish between regions with the accounted flag enabled or disabled, and be aware that glibc's malloc() for some inexplicable reason doesn't set the accounted flag on regions it commits, so the inaccurate flag will always yield higher numbers for private commited on Linux. By default, this fast path is enabled.

Note

/proc/pid/smaps_rollup was added in Linux kernel 3.16, so the default specifying process_memory_usage::want::private_committed_inaccurate will always fail on Linux kernels preceding that with an error code comparing equal to errc::operation_not_supported. As one would assume users would prefer this operation to fail on older kernels rather than silently go slowly in complex memory spaces, it is left opt-in to request the accurate implementation which works on older Linux kernels. Or, just don't request private_committed at all, and pretend private_paged_in means the same thing.

Mac OS provides no way of reading how much memory a process has committed. We therefore supply as private_committed the same value as private_paged_in.

drop_filesystem_cache()

result<void> llfio_v2_xxx::utils::drop_filesystem_cache ( )

inlinenoexcept

Tries to flush all modified data to the physical device, and then drop the OS filesystem cache, thus making all future reads come from the physical device. Currently only implemented for Microsoft Windows and Linux.

Note that the OS specific magic called by this routine generally requires elevated privileges for the calling process. For obvious reasons, calling this will have a severe negative impact on performance, but it's very useful for benchmarking cold cache vs warm cache performance.

file_buffer_default_size()

size_t llfio_v2_xxx::utils::file_buffer_default_size ( )

inline

Returns a reasonable default size for page_allocator, typically the closest page size from page_sizes() to 1Mb.

Returns

A value of a TLB large page size close to 1Mb.

Complexity\nWhatever the system API takes (one would hope constant time).

Errors returnable\nThrows any error from the operating system or std::bad_alloc.

  {
    static size_t size;
    if(size == 0u)
    {
      const std::vector<size_t> &sizes = page_sizes(true);
      for(auto &i : sizes)
      {
        if(i >= 1024 * 1024)
        {
          size = i;
          break;
        }
      }
      if(size == 0u)
      {
        size = 1024 * 1024;
      }
    }
    return size;
  }

page_size()

size_t llfio_v2_xxx::utils::page_size ( )

inlinenoexcept

Returns the smallest page size of this architecture which is useful for calculating direct i/o multiples.

Returns

The page size of this architecture.

Complexity\nWhatever the system API takes (one would hope constant time).

page_sizes()

const std::vector<size_t>& llfio_v2_xxx::utils::page_sizes ( bool  only_actually_available = true )

inline

Returns the page sizes of this architecture which is useful for calculating direct i/o multiples.

Parameters

only_actually_available

Only return page sizes actually available to the user running this process

Returns

The page sizes of this architecture.

Complexity\nFirst call performs multiple memory allocations, mutex locks and system calls. Subsequent calls

lock mutexes.

Errors returnable\nThrows any error from the operating system or std::bad_alloc.

random_fill()

void llfio_v2_xxx::utils::random_fill ( char *  buffer, size_t  bytes  )

inlinenoexcept

Fills the buffer supplied with cryptographically strong randomness. Uses the OS kernel API.

Parameters

buffer	A buffer to fill
bytes	How many bytes to fill

Complexity\nWhatever the system API takes.

Errors returnable\nAny error from the operating system.

random_string()

std::string llfio_v2_xxx::utils::random_string ( size_t  randomlen )

inline

Returns a cryptographically random string capable of being used as a filename. Essentially random_fill() + to_hex_string().

Parameters

randomlen

The number of bytes of randomness to use for the string.

Returns

A string representing the randomness at a 2x ratio, so if 32 bytes were requested, this string would be 64 bytes long.

Complexity\nWhatever the system API takes.

Errors returnable\nAny error from the operating system.

  {
    size_t outlen = randomlen * 2;
    std::string ret(outlen, 0);
    random_fill(const_cast<char *>(ret.data()), randomlen);
    QUICKCPPLIB_NAMESPACE::algorithm::string::to_hex_string(const_cast<char *>(ret.data()), outlen, ret.data(), randomlen);
    return ret;
  }