This is a more complicated functionality (implementation-wise) of the
`Entries` iterator, which allows a callee to transform it into a
`StoreIdIterator`.
It uses the crate internal `PathIterator` type and a function to extract
the internals of that iterator which is then turned into an iterator
over `Result<StoreId>`, which can be used to build a `StoreIdIterator`.
The implementation is ugly, but we need to be able to transform a
`Entries` iterator into a `StoreIdIterator` in the
`libimagentryannotation`, and possibly in more cases where we want to be
agnostic over iterators.
Maybe there is a cleaner solution to this, hence the comment about
whether this should be removed.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
With this change applied, libimagstore does not export backend
representing types anymore.
This change is necessar because when we want to switch the `StoreId`
implementation from "complex and stateful" to "Stateless and Cow<'_,
&str>", we need to be able to have a type which represents a "`StoreId`
plus the path of the Store itself".
This "the path of the Store itself" is passed around as reference, to
minimize runtime impact.
Because such a type should not be exported by the libimagstore crate, we
make it `pub(crate)` internally. But because the backend APIs also have
to use this type, we would export the (private) type in the APIs of the
backend.
Because of that we make the backend API also non-visible to crate users,
which also decreases the surface of the libimagstore API itself.
Besides:
Remove function `Link::with_base()` which is not needed anymore (was
used in tests only).
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This changes the implementation of the StoreId type to be less complex.
Before we always had to re-attach the path of the store itself each time
a StoreId object was passed to the store. Now we do not do this anymore,
hopefully we can move to a implementation of the Store API which takes
references of StoreId objects in the future.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
When printing the error chain, we print the error and then iterate over
the causes. Hence, we have to increase the printed number by one here,
because 0 (zero) is already printed.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This patch fixes a bug where the following code (here pseudocode) did
the wrong thing:
store.entries().in_collection("foo").for_each(||...)
because the `.in_collection()` call for the underlying PathIterator
object did not actually re-build the iterator. It only changed the
contained `PathIterBuilder`, but did not call `.build_iter()` on it to
rebuild the iterator object.
The test added with this patch checks whether the iterator does the
right thing.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This patch removes the magic constant we used when calling
`trace_unwrap_exit()` or `map_err_trace_exit_unwrap()`.
We used to call it with `1` as parameter, where the number was the exit
code to use. Now the implementation of the function does it
automatically (using 1 (one) as exit code).
All calls of these functions were fixed. Thanks to vim this was easy.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
We should be able, from the signature of the `generate_variants()`
helper function, to pass a `Vec<&T>` here, so cloning should not be
necessary.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
Because this API only errors when write!() errors occur, we can return
the exit code as an error here.
This way the user of the API can immediately exit if there was an IO
error, but the API automatically takes care of the right return value,
returning (exiting) with zero (0) if there was an "Broken pipe" error
and with one (1) otherwise, which is the expected behaviour here.
All calls to that API were changed accordingly.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This removes the feature to be able to override log destinations from
the commandline.
This feature is removed because the --override-config feature from the
runtime already contains this functionality. It is a little more complex
to use, though this is a feature hardly used at all, so I rather go for
less code (less complexity) here than feature bloat.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This feature gives the users of the Runtime API the possibility to check
whether the user wants the ID reporting to be ignored as well (and more
important) to override the default behaviour where the runtime assumes
that if STDOUT is a pipe, it is a pipe to an imag command.
Before, this would not have been possible:
imag contact find Jonas --json | jq 'map(.fullname)'
because the runtime automatically prints the output of the command to
stderr and the touched IDs to stdout.
But now with this change, a user can override this default behaviour and
do:
imag contact --ignore-ids find Jonas --json | jq 'map(.fullname)'
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
With this patch, IDs can be fetched from the CLI via libimagrt. This
gives us the possibility to automatically handle "stdin provides IDs" in
libimagrt with less boilerplate in the binaries codebases.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This patch adds the id reporting feature to libimagrt::runtime::Runtime,
where processed ("touched") ids can be reported to the Runtime and then
get printed to stdout if stdout is a pipe.
Other output is automatically redirected to stderr if stdout is a pipe
now.
The function already returns `Result<_>`, the only thing that had to be
done was refactoring the code for actually returning an error in that
case.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
With this patch, the log level aggregation is abstracted away by using
zero sized types and a macro to implement the whole thing.
This does not alter functionality, but makes the code more readable.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
::stf::fs::create_dir_all() takes any ref to `Path`, which is what we
have here, so we can leave out the allocation of a new PathBuf object
here.
Also remove the match by a `if let Some(_)`, which increases the
readability a bit.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
With this change, the cache is tested before accessing the filesystem,
which probably increases the speed if the cache has the entry, because
we avoid the slow IO operation.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This fixes a really ugly bug where the in-memory backend for the store
did not remove the entry from the in-memory hashmap on "move", but
simply copied it from the old location to the new one.
That caused tests to fail after the fixes introduced for the
Store::get() function which checked the filesystem and the internal
cache whether an entry exists before the actual "get" operation, because
an old entry would still exist after a move (only in the testcases).
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
Fixes: 09e8619cf5 ("libimagstore: Move from error-chain to failure")
This changes the implementation of Debug for the FileLockEntry to be
more explanatory of how the entry actually looks like. It does not only
print the path of the store anymore, but also the location of the Entry.
Printing header and content would be still too much, tho.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This actually caused an error.
When executing `imag ids --verbose trace` for example, the `imag` binary
got the default value (as in `--verbose info`) supplied from clap. So
far, so good. Problem is that the implementation then forwarded that
flag to `imag-ids`, which resulted in the `--verbose` flag to be passed
_two times_ to `imag-ids`.
Somehow this is really strange, but it does not really matter. First of
all: A default of "Info" is still too high IMO. Default should be
warnings and errors, but no information printed. We like silent tools,
don't we?
Second is that the commandline argument forwarding mechanism of `imag`
is broken and this was a fix which helped debugging of the brokenness,
so this is acutally a step forward in this regard as well.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This feature is required for the `imag` binary. It allows the binary to
use the imag internal logger for logging its own log output.
We need to be able to initialize the logger from an external module (in
all imag modules, the Runtime::new() implementation takes care of this,
but as we cannot use that in the `imag` binary itself, we allow this
method to be public behind a feature flag).
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
When specifying the `--debug` flag on the commandline, logging was not
enabled. That was because the config file parsing did not consider the
args.
Now, if `--debug` is passed on the CLI, logging is enabled for all
modules and level is set to `debug` for all modules.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
for safely creating a `String` object from `StoreId` which can be shown
to the user.
Mainly introduced because this is useful for error handling (when
putting a `StoreId` into an error kind, the compiler complains that
`StoreId` does not meet the required trait bounds. But `String` does and
we do not process the ID any further anyways).
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
The implementation of the in-memory filesystem for testing imag code did
not actually use `HashMap::remove()` when an entry was moved, but
`HashMap::get().cloned()`, which caused the original entry to exist
_after_ the move.
I'm not sure why this did not fail much earlier, but it was clearly
wrong. This commit adjust the test to check the "filesystem" before
checking the store and fixes the bug.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This actually caused tests to fail if there was indeed a file at
/tmp/store/test and the test tried to create a "test" entry in the store.
Do use backend instead to check whether entry actually exists.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
The issue was that the handlebars implementation logs as well and if we
use handlebars in the logger implementation that causes recursion which
crashes the program.
With handlebars 1.0.5, there is a feature[0] to disable logging in
handlebars (compiletime) which we use with this patch. The
exception-checking in the log implementation can be removed therefore.
[0]: https://github.com/sunng87/handlebars-rust/pull/236#issuecomment-427014611
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This reverts commit a1f0872486995b80216e8a08a2176debdef3752a.
As updating handlebars needs some more involvement, we roll back to the
version we use currently and schedule the update for later.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
The previous iterator was implemented to simply fetch _all_ pathes from
the filesystem, no matter what.
With this implementation, this changes. The iterator now has
functionality to optimize the iteration, if only a subdirectory of the
store is required, for example `$STORE/foo`.
This is done via functionality where the underlying iterator gets
altered.
First of all, the interface was changed to return a `Entries` object,
which itself only covers the libimagstore-internal `PathIterator` type.
This type was changed so that the backend implementation provides an
"PathIterBuilder`, which builds the actual iterator object for the
`PathIterator` type.
The intermediate `StoreIdConstructingIterator` was merged into
`PathIterator` for simplicity.
The `Entries` type got functionality similar to the
`StoreIdIteratorWithStore` type for easier transition to the new API.
This should probably be removed at a later point, though.
As the `walkdir::WalkDir` type is not as nice as it could be, iterators
for two collections in the store could be built like this (untested):
store
.entries()?
.in_collection("foo")
.chain(store.entries()?.in_collection("bar"))
Functionality to exclude subdirectories is not possible with the current
`walkdir::WalkDir` implementation and has to be done during iteration,
with filtering (as usual).
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This patch changes the filesystem-backend implementation of libimagstore
to open files on each read/write rather than holding the file handle in
memory at all times.
Whenever a lot of imag store entries are read into memory, the imag
process may ran out of file descriptors. With this patch applied, a
`Store::get()` call on an entry which is not yet in the store cache
would cause the file to be read, but the FD being dropped after that.
Likewise, a `Store::update()` (which is also called if the imag entry is
dropped) would re-open the file on the filesystem and write the contents
from the imag store cache back to the file.
With this patch, opening hundrets or thousands of imag entries should be
no problem anymore, only the available memory should be a limit then.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This patch fixes a bug we did not even hit (yet). It is: When deleting
an Entry from the store, this could potentially leave artifacts in the
cache.
Szenario: An Entry, which was loaded (via `Store::get()` for example),
gets `Store::delete()`ed twice. The first call would work as expected,
but leave the Entry in the Store cache. The second call would then fail,
as the Entry is already removed on the FS, but still in the cache. This
would fail - which is the right thing to do here - but with the wrong
error (with a FileError rather than a FileNotFound error).
This patch fixes this.
First of all, the appropriate `PathBuf` object is calculated in all
cases, as this object is needed to check whether the file is actually
there (which could be the case if the Entry is in the cache and if it is
not).
If the entry is in the cache and is borrowed: error. If not, remove the
entry from the cache. Afterwards the file is deleted on disk.
If the entry is not in the cache, but the file exists, the file is removed.
If the file does not exist: error.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
As of rustc 1.26, the `flatten()` method on iterators is preserved by
the rust standard library.
This could cause this code to hard-error some time in the future with
the `flatten()` function actually implemented by the standard library.
Hence we move to use the `Itertools::flatten()` function here
explicitely.
Because the iterators need to be able to check whether the file exists
_in the backend_ (not on disk, but in the backend, because of in-memory
test for example), we need to be able to pass the backend to the
iterator intermediate type.
This patch implements this. It does so by changing the internal backend
member of the store from `Box<FileAbstraction>` to
`Arc<FileAbstraction>`, which gives us the ability to clone the
reference to the backend easily without needing to rely on lifetimes
here, because of the Arc.
Also, less boxes are always good.
This patch reimplements the iterator extensions.
As we iterate (in StoreIdIterator) over Result<StoreId> now anyways, we
don't need the extensions for Result iterators anymore.
This patch rewrites the extensions to be more simple in every way and
generic over the error type in the iterator.
All the errors have to do is implement From<StoreError>, which is what
they do when linking the generated error types with error_chain to the
libimagstore error types.
This patch rewrites the Store::entries() function to not be collecting
the iterator.
It therefore introduces a new, internal, iterator type which creates the
StoreId objects from the pathes the PathIterator yields internally.
With this patch, the Store iterator interface changes, as the iterators
now yield `Result<StoreId, StoreError>` instead of `StoreId`.
This is necessary, as the internal conversion errors shouldn't be
hidden.
Of course, the iterator types (like the StoreGetIterator and so on)
should hold a Result<StoreId> internally as well, and also yield
appropritely. This was changed in this commit, too.
Before we had the problem that when iterating over _a lot_ (like 5k)
entries and also fetching them, at some point the OS would return with
"Too many files open".
That is because the store internally caches a lot.
With this change, the Store gets an API to query how big the cache is,
how much the cache can currently hold and (and that's the main thing in
this patch) to flush the cache to disk.
A function to simply ask the store whether its cache should be flushed
(which would us require to ask the OS how many files we can open...
which would be possible with `libc::getrlimit`) does not yet exist,
though, but could be added easily if desired.
