LLFIO v2.00
Loading...
Searching...
No Matches
llfio_v2_xxx::algorithm::contents_visitor Struct Reference

A visitor for the filesystem contents algorithm. More...

#include "contents.hpp"

Inheritance diagram for llfio_v2_xxx::algorithm::contents_visitor:
llfio_v2_xxx::algorithm::traverse_visitor

Classes

struct  contents_type
 Enumerated contents, and what parts of their stat_t is valid. More...
struct  _state_type

Public Member Functions

 contents_visitor ()=default
 Default construtor.
 contents_visitor (stat_t::want _metadata, bool _include_files=true, bool _include_directories=true, bool _include_symlinks=true)
 Construct an instance.
virtual result< void > post_enumeration (void *data, const directory_handle &dirh, directory_handle::buffers_type &contents, size_t depth) noexcept
 The default implementation accumulates the contents into thread local storage. At traverse end, all the thread local storages are coalesced into a single result, the member variable contents.
virtual result< size_t > finished (void *data, result< size_t > result) noexcept
 Called when a traversal finishes, this default implementation merges all the thread local results into contents, and deallocates the thread local results.
virtual result< directory_handle > directory_open_failed (void *data, result< void >::error_type &&error, const directory_handle &dirh, path_view leaf, size_t depth) noexcept
 Called when we failed to open a directory for enumeration. The default fails the traversal with that error. Return a default constructed instance to ignore the failure.
virtual result< bool > pre_enumeration (void *data, const directory_handle &dirh, size_t depth) noexcept
 Called to decide whether to enumerate a directory.
virtual result< void > stack_updated (void *data, size_t dirs_processed, size_t known_dirs_remaining, size_t depth_processed, size_t known_depth_remaining) noexcept
 Called whenever the traversed stack of directory hierarchy is updated. This can act as an estimated progress indicator, or to give an accurate progress indicator by matching it against a previous traversal.

Public Attributes

bool contents_include_files {true}
 Whether to include files in the contents.
bool contents_include_directories {true}
 Whether to include directories in the contents.
bool contents_include_symlinks {true}
 Whether to include symlinks in the contents.
stat_t::want contents_include_metadata {stat_t::want::none}
 What stat_t::want to include, if enumeration doesn't provide these, they will be additionally fetched.

Static Protected Member Functions

static std::shared_ptr< contents_type > _thread_contents (_state_type *state) noexcept

Friends

result< contents_type > contents (const path_handle &dirh, contents_visitor *visitor=nullptr, size_t threads=0, bool force_slow_path=false)
 Calculate the contents of everything within and under dirh. What is returned is unordered.

Detailed Description

A visitor for the filesystem contents algorithm.

Constructor & Destructor Documentation

◆ contents_visitor()

llfio_v2_xxx::algorithm::contents_visitor::contents_visitor ( stat_t::want _metadata,
bool _include_files = true,
bool _include_directories = true,
bool _include_symlinks = true )
inlineexplicit

Construct an instance.

64 : contents_include_files(_include_files)
65 , contents_include_directories(_include_directories)
66 , contents_include_symlinks(_include_symlinks)
67 , contents_include_metadata(_metadata)
68 {
69 }
stat_t::want contents_include_metadata
What stat_t::want to include, if enumeration doesn't provide these, they will be additionally fetched...
Definition contents.hpp:50
bool contents_include_symlinks
Whether to include symlinks in the contents.
Definition contents.hpp:48
bool contents_include_directories
Whether to include directories in the contents.
Definition contents.hpp:46
bool contents_include_files
Whether to include files in the contents.
Definition contents.hpp:44

Member Function Documentation

◆ _thread_contents()

std::shared_ptr< contents_type > llfio_v2_xxx::algorithm::contents_visitor::_thread_contents ( _state_type * state)
inlinestaticprotectednoexcept
91 {
92 LLFIO_EXCEPTION_TRY
93 {
94 static thread_local std::weak_ptr<contents_type> mycontents;
95 auto ret = mycontents.lock();
96 if(ret)
97 {
98 return ret;
99 }
100 ret = std::make_unique<contents_type>();
101 mycontents = ret;
102 std::lock_guard<std::mutex> g(state->lock);
103 state->all_thread_contents.push_back(ret);
104 return ret;
105 }
106 LLFIO_EXCEPTION_CATCH_ALL
107 {
108 return {};
109 }
110 }

◆ directory_open_failed()

virtual result< directory_handle > llfio_v2_xxx::algorithm::traverse_visitor::directory_open_failed ( void * data,
result< void >::error_type && error,
const directory_handle & dirh,
path_view leaf,
size_t depth )
inlinevirtualnoexceptinherited

Called when we failed to open a directory for enumeration. The default fails the traversal with that error. Return a default constructed instance to ignore the failure.

Note
May be called from multiple kernel threads concurrently.

Reimplemented in llfio_v2_xxx::algorithm::reduce_visitor, and llfio_v2_xxx::algorithm::summarize_visitor.

54 {
55 (void) data;
56 (void) dirh;
57 (void) leaf;
58 (void) depth;
59 return failure(std::move(error));
60 }

◆ finished()

virtual result< size_t > llfio_v2_xxx::algorithm::contents_visitor::finished ( void * data,
result< size_t > result )
inlinevirtualnoexcept

Called when a traversal finishes, this default implementation merges all the thread local results into contents, and deallocates the thread local results.

Reimplemented from llfio_v2_xxx::algorithm::traverse_visitor.

231 {
232 LLFIO_EXCEPTION_TRY
233 {
234 auto *state = (_state_type *) data;
235 state->contents.clear();
236 state->contents.metadata = state->metadata.load(std::memory_order_relaxed);
237 size_t count = 0;
238 for(auto &i : state->all_thread_contents)
239 {
240 count += i->size();
241 }
242 state->contents.reserve(count);
243 for(auto &i : state->all_thread_contents)
244 {
245 state->contents.insert(state->contents.end(), std::make_move_iterator(i->begin()), std::make_move_iterator(i->end()));
246 }
247 state->all_thread_contents.clear();
248 return result;
249 }
250 LLFIO_EXCEPTION_CATCH_ALL
251 {
252 return error_from_exception();
253 }
254 }

◆ post_enumeration()

virtual result< void > llfio_v2_xxx::algorithm::contents_visitor::post_enumeration ( void * data,
const directory_handle & dirh,
directory_handle::buffers_type & contents,
size_t depth )
inlinevirtualnoexcept

The default implementation accumulates the contents into thread local storage. At traverse end, all the thread local storages are coalesced into a single result, the member variable contents.

Reimplemented from llfio_v2_xxx::algorithm::traverse_visitor.

118 {
119 LLFIO_EXCEPTION_TRY
120 {
121 auto *state = (_state_type *) data;
122 (void) depth;
123 if(!contents.empty())
124 {
125 contents_type toadd;
126 toadd.reserve(contents.size());
127 filesystem::path dirhpath;
128 // Compute the path of `dirh` relative to the root handle. `rootdirpathlen`
129 // caches the length of the root handle's current path, which is stale if the
130 // root was relocated after it was cached, so a mismatch refreshes it and
131 // retries. There is no progress guarantee across a concurrent relocation, and
132 // the retry used to be unbounded, so it is now bounded at two attempts: if the
133 // refreshed length still does not resolve, the entries are reported against
134 // `dirh`'s absolute path, which is still correct.
135 for(size_t attempt = 0;; attempt++)
136 {
137 OUTCOME_TRY(dirhpath, dirh.current_path());
138 const size_t dirhpathlen = dirhpath.native().size();
139 const size_t rootdirpathlen = state->rootdirpathlen.load(std::memory_order_relaxed);
140 if(dirhpathlen >= rootdirpathlen)
141 {
142 // Strip the cached root path, then the separator joining it to the
143 // remainder. A filesystem root's current path ("/" or "C:\\") ends in a
144 // separator already, so the cached length deliberately excludes one:
145 // stripping a fixed length + 1 ate the first character of every relative
146 // path under such a root, which neither resolved nor ever converged.
147 filesystem::path relpath = dirhpath.native().substr(rootdirpathlen);
148 while(!relpath.native().empty() && (relpath.native()[0] == '/' || relpath.native()[0] == '\\'))
149 {
150 relpath = filesystem::path(relpath.native().substr(1));
151 }
152 if(relpath.native().empty())
153 {
154 // `dirh` is at or above the root. Treat it as the root itself when the two
155 // handles name the same directory. Comparing the root handle's own current
156 // path needs no new handle and is what works on Windows, where enumerating
157 // reports /!!/Device/... paths: opening "." relative to the root handle and
158 // comparing unique ids there always failed, so every top level entry was
159 // reported with its absolute path instead of its leafname. The id check is
160 // kept for the case of two differently spelled handles onto one directory.
161 OUTCOME_TRY(auto &&rootnow, state->rootdirh.current_path());
162 if(rootnow.native() == dirhpath.native())
163 {
164 dirhpath = filesystem::path();
165 break;
166 }
167 auto r = directory_handle::directory(state->rootdirh, filesystem::path("."));
168 if(r && r.value().unique_id() == dirh.unique_id())
169 {
170 dirhpath = filesystem::path();
171 break;
172 }
173 }
174 else
175 {
176 auto r = directory_handle::directory(state->rootdirh, relpath);
177 if(r && r.value().unique_id() == dirh.unique_id())
178 {
179 dirhpath = std::move(relpath);
180 break;
181 }
182 }
183 }
184 if(attempt > 0)
185 {
186 break;
187 }
188 OUTCOME_TRY(auto &&rootpath, state->rootdirh.current_path());
189 state->rootdirpathlen.store(rootpath.native().size(), std::memory_order_relaxed);
190 }
191 auto _metadata_ = state->metadata.load(std::memory_order_relaxed);
192 if((_metadata_ & (contents_include_metadata | contents.metadata())) != _metadata_)
193 {
194 state->metadata.store(_metadata_ & (contents_include_metadata | contents.metadata()), std::memory_order_relaxed);
195 }
196 auto into = _thread_contents(state);
197 for(auto &entry : contents)
198 {
199 auto need_stat = contents_include_metadata & ~contents.metadata();
200 if((contents_include_files && entry.stat.st_type == filesystem::file_type::regular) ||
201 (contents_include_directories && entry.stat.st_type == filesystem::file_type::directory) ||
202 (contents_include_symlinks && entry.stat.st_type == filesystem::file_type::symlink))
203 {
204 if(!need_stat)
205 {
206 into->emplace_back(dirhpath / entry.leafname, entry.stat);
207 continue;
208 }
209 auto r = file_handle::file(dirh, entry.leafname, file_handle::mode::attr_read);
210 if(r)
211 {
212 OUTCOME_TRY(entry.stat.fill(r.value(), need_stat));
213 into->emplace_back(dirhpath / entry.leafname, entry.stat);
214 }
215 }
216 }
217 }
218 return success();
219 }
220 LLFIO_EXCEPTION_CATCH_ALL
221 {
222 return error_from_exception();
223 }
224 }
static result< directory_handle > directory(const path_handle &base, path_view_type path, mode _mode=mode::read, creation _creation=creation::open_existing, caching _caching=caching::all, flag flags=flag::none) noexcept
static result< file_handle > file(const path_handle &base, path_view_type path, mode _mode=mode::read, creation _creation=creation::open_existing, caching _caching=caching::all, flag flags=flag::none) noexcept
Enumerated contents, and what parts of their stat_t is valid.
Definition contents.hpp:55
stat_t::want metadata
The metadata valid within all the stat_t in the contents traversed.
Definition contents.hpp:57
friend result< contents_type > contents(const path_handle &dirh, contents_visitor *visitor, size_t threads, bool force_slow_path) noexcept
Calculate the contents of everything within and under dirh. What is returned is unordered.
Definition contents.hpp:271

◆ pre_enumeration()

virtual result< bool > llfio_v2_xxx::algorithm::traverse_visitor::pre_enumeration ( void * data,
const directory_handle & dirh,
size_t depth )
inlinevirtualnoexceptinherited

Called to decide whether to enumerate a directory.

Note that it is more efficient to ignore the directory entry in post_enumeration() than to ignore it here, as a handle is opened for the directory before this callback. Equally, if you need that handle to inspect the directory e.g. to check if one is entering a different filesystem from the root, here is best.

The default returns true.

Note
May be called from multiple kernel threads concurrently.
75 {
76 (void) data;
77 (void) dirh;
78 (void) depth;
79 return true;
80 }

◆ stack_updated()

virtual result< void > llfio_v2_xxx::algorithm::traverse_visitor::stack_updated ( void * data,
size_t dirs_processed,
size_t known_dirs_remaining,
size_t depth_processed,
size_t known_depth_remaining )
inlinevirtualnoexceptinherited

Called whenever the traversed stack of directory hierarchy is updated. This can act as an estimated progress indicator, or to give an accurate progress indicator by matching it against a previous traversal.

Parameters
dataThe third party data pointer passed to traverse().
dirs_processedThe total number of directories traversed so far.
known_dirs_remainingThe currently known number of directories awaiting traversal.
depth_processedHow many levels deep we have already completely traversed.
known_depth_remainingThe currently known number of levels we shall traverse.
Note
May be called from multiple kernel threads concurrently.
116 {
117 (void) data;
118 (void) dirs_processed;
119 (void) known_dirs_remaining;
120 (void) depth_processed;
121 (void) known_depth_remaining;
122 return success();
123 }

◆ contents

result< contents_type > contents ( const path_handle & dirh,
contents_visitor * visitor = nullptr,
size_t threads = 0,
bool force_slow_path = false )
friend

Calculate the contents of everything within and under dirh. What is returned is unordered.

The paths returned are relative to dirh, never absolute.

This is a very thin veneer over traverse() which came out of the fact that I kept writing "get me the contents" traversal visitors again and again, so eventually I just wrote a library edition. Its only "clever" bit is that it stores the contents in thread local storage, and merges the contents afterwards.

It is race free to concurrent relocations of dirh. It is entirely implemented in header-only code, as it is very simple.

273 {
274 contents_visitor default_visitor;
275 if(visitor == nullptr)
276 {
277 visitor = &default_visitor;
278 }
279 contents_visitor::_state_type state(dirh);
280 OUTCOME_TRY(auto &&dirhpath, dirh.current_path());
281 state.rootdirpathlen.store(dirhpath.native().size(), std::memory_order_relaxed);
282 OUTCOME_TRY(traverse(dirh, visitor, threads, &state, force_slow_path));
283 return {std::move(state.contents)};
284 }
result< size_t > traverse(const path_handle &dirh, traverse_visitor *visitor, size_t threads=0, void *data=nullptr, bool force_slow_path=false) noexcept
Traverse everything within and under dirh.
contents_visitor()=default
Default construtor.

Member Data Documentation

◆ contents_include_directories

bool llfio_v2_xxx::algorithm::contents_visitor::contents_include_directories {true}

Whether to include directories in the contents.

46{true};

◆ contents_include_files

bool llfio_v2_xxx::algorithm::contents_visitor::contents_include_files {true}

Whether to include files in the contents.

44{true};

◆ contents_include_metadata

stat_t::want llfio_v2_xxx::algorithm::contents_visitor::contents_include_metadata {stat_t::want::none}

What stat_t::want to include, if enumeration doesn't provide these, they will be additionally fetched.

50{stat_t::want::none};

◆ contents_include_symlinks

bool llfio_v2_xxx::algorithm::contents_visitor::contents_include_symlinks {true}

Whether to include symlinks in the contents.

48{true};

The documentation for this struct was generated from the following file: