This patch changes the filesystem-backend implementation of libimagstore
to open files on each read/write rather than holding the file handle in
memory at all times.
Whenever a lot of imag store entries are read into memory, the imag
process may ran out of file descriptors. With this patch applied, a
`Store::get()` call on an entry which is not yet in the store cache
would cause the file to be read, but the FD being dropped after that.
Likewise, a `Store::update()` (which is also called if the imag entry is
dropped) would re-open the file on the filesystem and write the contents
from the imag store cache back to the file.
With this patch, opening hundrets or thousands of imag entries should be
no problem anymore, only the available memory should be a limit then.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This patch fixes a bug we did not even hit (yet). It is: When deleting
an Entry from the store, this could potentially leave artifacts in the
cache.
Szenario: An Entry, which was loaded (via `Store::get()` for example),
gets `Store::delete()`ed twice. The first call would work as expected,
but leave the Entry in the Store cache. The second call would then fail,
as the Entry is already removed on the FS, but still in the cache. This
would fail - which is the right thing to do here - but with the wrong
error (with a FileError rather than a FileNotFound error).
This patch fixes this.
First of all, the appropriate `PathBuf` object is calculated in all
cases, as this object is needed to check whether the file is actually
there (which could be the case if the Entry is in the cache and if it is
not).
If the entry is in the cache and is borrowed: error. If not, remove the
entry from the cache. Afterwards the file is deleted on disk.
If the entry is not in the cache, but the file exists, the file is removed.
If the file does not exist: error.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
As of rustc 1.26, the `flatten()` method on iterators is preserved by
the rust standard library.
This could cause this code to hard-error some time in the future with
the `flatten()` function actually implemented by the standard library.
Hence we move to use the `Itertools::flatten()` function here
explicitely.
Because the iterators need to be able to check whether the file exists
_in the backend_ (not on disk, but in the backend, because of in-memory
test for example), we need to be able to pass the backend to the
iterator intermediate type.
This patch implements this. It does so by changing the internal backend
member of the store from `Box<FileAbstraction>` to
`Arc<FileAbstraction>`, which gives us the ability to clone the
reference to the backend easily without needing to rely on lifetimes
here, because of the Arc.
Also, less boxes are always good.
This patch reimplements the iterator extensions.
As we iterate (in StoreIdIterator) over Result<StoreId> now anyways, we
don't need the extensions for Result iterators anymore.
This patch rewrites the extensions to be more simple in every way and
generic over the error type in the iterator.
All the errors have to do is implement From<StoreError>, which is what
they do when linking the generated error types with error_chain to the
libimagstore error types.
This patch rewrites the Store::entries() function to not be collecting
the iterator.
It therefore introduces a new, internal, iterator type which creates the
StoreId objects from the pathes the PathIterator yields internally.
With this patch, the Store iterator interface changes, as the iterators
now yield `Result<StoreId, StoreError>` instead of `StoreId`.
This is necessary, as the internal conversion errors shouldn't be
hidden.
Of course, the iterator types (like the StoreGetIterator and so on)
should hold a Result<StoreId> internally as well, and also yield
appropritely. This was changed in this commit, too.
Before we had the problem that when iterating over _a lot_ (like 5k)
entries and also fetching them, at some point the OS would return with
"Too many files open".
That is because the store internally caches a lot.
With this change, the Store gets an API to query how big the cache is,
how much the cache can currently hold and (and that's the main thing in
this patch) to flush the cache to disk.
A function to simply ask the store whether its cache should be flushed
(which would us require to ask the OS how many files we can open...
which would be possible with `libc::getrlimit`) does not yet exist,
though, but could be added easily if desired.
The rust compiler does some fancy things for us: It automatically finds
the right fields if the name of the variable and the file is the same.
Lets use that to reduce boilerplate with this patch.
This reverts commit ce0bd9298a.
Pipe magic is removed with this patch.
We remove pipe magic because its implementation in libimagstore is too
complicated and the benefits are too small.
Having this functionality would be really nice, but the cost-benefit
ratio would still be too high.
The implementation in the store would require a rewrite of the internal
caching functionality in the store, plus some functionality to serialize
and deserialize the cache. This is theoretically possible, but as the
store only knows about "StoreEntry" objects, and only the backend knows
of "Entry" (which would be simply de/serializeable), the complexity
increases a _lot_.
Hence, we drop this feature-idea here.
Maybe, at some later point, this functionality will be in imag. The
history of development of this feature is in the history, we just don't
have it merged.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This fixes the following problem:
If the editor setting was "vim " instead of "vim", the editor was called
with `"vim" " "`, which resulted in unexpected behaviour.
The patch fixes this.
This fixes the issue that spawning the editor trashes the terminal.
The signature of the Runtime::editor() function changed, which has to be
fixed in using code.
do inherit stdin and stderr from parent process, to not break terminal
editors when editing stuff.
vim printed "Input not from terminal" warning messages. This was fixed
by this commit.
When we merged the changes in libimagrt so that it automatically detects
whether stdin/stdout is a TTY and provides the user with stderr in case
stdout is not a TTY, we forgot that things like
imag foo | grep bar
becomes impossible with that, because imag detects that stdout is not a
tty and automatically uses stderr for output.
But in this case, we don't want that. The output has to be stdout in
this case.
With this change, we have a flag in the runtime ("--pipe-magic" or "-P",
globally available) which turns on "pipe magic".
The expected behaviour is the following, if "-P" is passed:
* If stdout is a TTY, we print to stdout
* If stdout is not a TTY, we print to stderr
* If stdin is not a TTY, we do not provide it
If "-P" is not passed, we allow the user of libimagrt to use stdin for
interactive stuff (the interactive stuff is not yet implemented).
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
Unfortunately, our latest fix to file parsing did not solve all issues.
So we have to fix it _again_.
The problem was the `std::str::Lines` iterator, which apparently fails
this:
assert_eq!(1, "".lines().count());
as an empty line seems not to be a line.
Because of that, when reading a file with an empty line at its bottom
got stripped off that line.
This patch removes the use of the `lines()` iterator and uses
`split("\n")` instead. This only works on Unix operating systems, but as
we only target unix operating systems with imag, this is not considered
an issue right now.
This patch also adds extensive tests on multiple levels in the
`libimagstore` implementation:
* On the parsing level, for the function which implements the parsing
* On the filesystem abstraction levels
* On the `Store` levels
to make sure that everything is parsed correctly.
This is another approach for providing access to stdin/out/err via
libimagrt::runtime::Runtime.
The Runtime object does configure which output gets returned (stdout if
stdout is a tty, else stderr).
With this we can change libimagrt to read/write the store from/to
stdin/stdout without the user noticing that she does not write to stdout
but stderr.
Reading from stdin is not possible then, though.
This way we can control whether "out" output goes to stdout or stderr
without the user of the functionality knowing.
This is useful for later when we use libimagrt to automatically
read and write the store from and to stdout/in depending on whether we
are talking to a TTY or a pipe.
Adapt libimagrt interface to export the functions we need to do this.
This is not that nice, but the best approach without rewriting large
parts of libimagrt.
Rewrite without regex crate.
The regex approach was broken. If the following _content_ was provided
in the entry:
foo
---
bar
The regex approach parsed the header until the "---" in the content.
This is, of course, not the way to do that.
Now, the parsing is implemented by hand. Should be faster as well,
though I don't care about this.
This fixes a severe bug.
