Before we had the problem that when iterating over _a lot_ (like 5k)
entries and also fetching them, at some point the OS would return with
"Too many files open".
That is because the store internally caches a lot.
With this change, the Store gets an API to query how big the cache is,
how much the cache can currently hold and (and that's the main thing in
this patch) to flush the cache to disk.
A function to simply ask the store whether its cache should be flushed
(which would us require to ask the OS how many files we can open...
which would be possible with `libc::getrlimit`) does not yet exist,
though, but could be added easily if desired.
The rust compiler does some fancy things for us: It automatically finds
the right fields if the name of the variable and the file is the same.
Lets use that to reduce boilerplate with this patch.
Unfortunately, our latest fix to file parsing did not solve all issues.
So we have to fix it _again_.
The problem was the `std::str::Lines` iterator, which apparently fails
this:
assert_eq!(1, "".lines().count());
as an empty line seems not to be a line.
Because of that, when reading a file with an empty line at its bottom
got stripped off that line.
This patch removes the use of the `lines()` iterator and uses
`split("\n")` instead. This only works on Unix operating systems, but as
we only target unix operating systems with imag, this is not considered
an issue right now.
This patch also adds extensive tests on multiple levels in the
`libimagstore` implementation:
* On the parsing level, for the function which implements the parsing
* On the filesystem abstraction levels
* On the `Store` levels
to make sure that everything is parsed correctly.
Rewrite without regex crate.
The regex approach was broken. If the following _content_ was provided
in the entry:
foo
---
bar
The regex approach parsed the header until the "---" in the content.
This is, of course, not the way to do that.
Now, the parsing is implemented by hand. Should be faster as well,
though I don't care about this.
This fixes a severe bug.
This patch adds API functions in the StoreIdIteratorWithStore iterator
type to transform it into a iterator which _does_ something (as in the
`libimagstore::iter` API).
It mimics the API which is offered by `libimagstore::iter`.
This test is not applicable anymore because it tests (and tested) the
wrong thing.
It was to check whether the function failed because the "imag" key
contained the wrong type, but this is not tested by that function. The
function only checks whether the "imag" key is present.
Before we extracted the store configuration from the configuration
toml::Value object and passed it to the store.
This is unecessary overhead.
Now we pass the whole configuration object and let the store extract the
required values.
Before the iterator did also yield storeids for directories, which was a
bug.
This change introduces a new if_file() function in the store-internal
backend, which is needed to check whether a path actually points to a
File, be it inmemory or on the real filesystem.
That's because tests might fail if they check via PathBuf::is_file() as
in tests, the entries only exist inmemory.
From the documentation of Walkdir::min_depth():
Set the minimum depth of entries yielded by the iterator.
The smallest depth is 0 and always corresponds to the path given to
the new function on this type. Its direct descendents have depth 1,
and their descendents have depth 2, and so on.
This means that when we started with "/tmp/store", we end up yielding
that exact path in the first iteration. This is exactly what we do _not_
want.
Setting the minimal depth to 1 fixes this bug.
This changes the internal GlobStoreIdIterator to return Result<StoreId>,
which gives us the possibility to aggregate errors in the
Store::retrieve_for_module() function and return them instead of tracing
them from the store.
The changes the internals to actually fetch the whole list of storeids,
which is unfortunate of course, but changing the interface is not an
option here, in my opinion.
At least we're only aggregating pathes, so the memory usage is pretty
low here.
This patch fixes a problem where the Store::delete() function only
checked the store-internal cache whether an entry exists, but not the
Filesystem. After this patch is applied, the Store::delete() function
also checks the filesystem whether the entry exists.
