As of rustc 1.26, the `flatten()` method on iterators is preserved by
the rust standard library.
This could cause this code to hard-error some time in the future with
the `flatten()` function actually implemented by the standard library.
Hence we move to use the `Itertools::flatten()` function here
explicitely.
Because the iterators need to be able to check whether the file exists
_in the backend_ (not on disk, but in the backend, because of in-memory
test for example), we need to be able to pass the backend to the
iterator intermediate type.
This patch implements this. It does so by changing the internal backend
member of the store from `Box<FileAbstraction>` to
`Arc<FileAbstraction>`, which gives us the ability to clone the
reference to the backend easily without needing to rely on lifetimes
here, because of the Arc.
Also, less boxes are always good.
This patch reimplements the iterator extensions.
As we iterate (in StoreIdIterator) over Result<StoreId> now anyways, we
don't need the extensions for Result iterators anymore.
This patch rewrites the extensions to be more simple in every way and
generic over the error type in the iterator.
All the errors have to do is implement From<StoreError>, which is what
they do when linking the generated error types with error_chain to the
libimagstore error types.
This patch rewrites the Store::entries() function to not be collecting
the iterator.
It therefore introduces a new, internal, iterator type which creates the
StoreId objects from the pathes the PathIterator yields internally.
With this patch, the Store iterator interface changes, as the iterators
now yield `Result<StoreId, StoreError>` instead of `StoreId`.
This is necessary, as the internal conversion errors shouldn't be
hidden.
Of course, the iterator types (like the StoreGetIterator and so on)
should hold a Result<StoreId> internally as well, and also yield
appropritely. This was changed in this commit, too.
Before we had the problem that when iterating over _a lot_ (like 5k)
entries and also fetching them, at some point the OS would return with
"Too many files open".
That is because the store internally caches a lot.
With this change, the Store gets an API to query how big the cache is,
how much the cache can currently hold and (and that's the main thing in
this patch) to flush the cache to disk.
A function to simply ask the store whether its cache should be flushed
(which would us require to ask the OS how many files we can open...
which would be possible with `libc::getrlimit`) does not yet exist,
though, but could be added easily if desired.
The rust compiler does some fancy things for us: It automatically finds
the right fields if the name of the variable and the file is the same.
Lets use that to reduce boilerplate with this patch.
This reverts commit ce0bd9298a.
Pipe magic is removed with this patch.
We remove pipe magic because its implementation in libimagstore is too
complicated and the benefits are too small.
Having this functionality would be really nice, but the cost-benefit
ratio would still be too high.
The implementation in the store would require a rewrite of the internal
caching functionality in the store, plus some functionality to serialize
and deserialize the cache. This is theoretically possible, but as the
store only knows about "StoreEntry" objects, and only the backend knows
of "Entry" (which would be simply de/serializeable), the complexity
increases a _lot_.
Hence, we drop this feature-idea here.
Maybe, at some later point, this functionality will be in imag. The
history of development of this feature is in the history, we just don't
have it merged.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
