Before we had the problem that when iterating over _a lot_ (like 5k)
entries and also fetching them, at some point the OS would return with
"Too many files open".
That is because the store internally caches a lot.
With this change, the Store gets an API to query how big the cache is,
how much the cache can currently hold and (and that's the main thing in
this patch) to flush the cache to disk.
A function to simply ask the store whether its cache should be flushed
(which would us require to ask the OS how many files we can open...
which would be possible with `libc::getrlimit`) does not yet exist,
though, but could be added easily if desired.
The rust compiler does some fancy things for us: It automatically finds
the right fields if the name of the variable and the file is the same.
Lets use that to reduce boilerplate with this patch.
This reverts commit ce0bd9298a.
Pipe magic is removed with this patch.
We remove pipe magic because its implementation in libimagstore is too
complicated and the benefits are too small.
Having this functionality would be really nice, but the cost-benefit
ratio would still be too high.
The implementation in the store would require a rewrite of the internal
caching functionality in the store, plus some functionality to serialize
and deserialize the cache. This is theoretically possible, but as the
store only knows about "StoreEntry" objects, and only the backend knows
of "Entry" (which would be simply de/serializeable), the complexity
increases a _lot_.
Hence, we drop this feature-idea here.
Maybe, at some later point, this functionality will be in imag. The
history of development of this feature is in the history, we just don't
have it merged.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This fixes the following problem:
If the editor setting was "vim " instead of "vim", the editor was called
with `"vim" " "`, which resulted in unexpected behaviour.
The patch fixes this.
This fixes the issue that spawning the editor trashes the terminal.
The signature of the Runtime::editor() function changed, which has to be
fixed in using code.
do inherit stdin and stderr from parent process, to not break terminal
editors when editing stuff.
vim printed "Input not from terminal" warning messages. This was fixed
by this commit.
When we merged the changes in libimagrt so that it automatically detects
whether stdin/stdout is a TTY and provides the user with stderr in case
stdout is not a TTY, we forgot that things like
imag foo | grep bar
becomes impossible with that, because imag detects that stdout is not a
tty and automatically uses stderr for output.
But in this case, we don't want that. The output has to be stdout in
this case.
With this change, we have a flag in the runtime ("--pipe-magic" or "-P",
globally available) which turns on "pipe magic".
The expected behaviour is the following, if "-P" is passed:
* If stdout is a TTY, we print to stdout
* If stdout is not a TTY, we print to stderr
* If stdin is not a TTY, we do not provide it
If "-P" is not passed, we allow the user of libimagrt to use stdin for
interactive stuff (the interactive stuff is not yet implemented).
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
Unfortunately, our latest fix to file parsing did not solve all issues.
So we have to fix it _again_.
The problem was the `std::str::Lines` iterator, which apparently fails
this:
assert_eq!(1, "".lines().count());
as an empty line seems not to be a line.
Because of that, when reading a file with an empty line at its bottom
got stripped off that line.
This patch removes the use of the `lines()` iterator and uses
`split("\n")` instead. This only works on Unix operating systems, but as
we only target unix operating systems with imag, this is not considered
an issue right now.
This patch also adds extensive tests on multiple levels in the
`libimagstore` implementation:
* On the parsing level, for the function which implements the parsing
* On the filesystem abstraction levels
* On the `Store` levels
to make sure that everything is parsed correctly.
This is another approach for providing access to stdin/out/err via
libimagrt::runtime::Runtime.
The Runtime object does configure which output gets returned (stdout if
stdout is a tty, else stderr).
With this we can change libimagrt to read/write the store from/to
stdin/stdout without the user noticing that she does not write to stdout
but stderr.
Reading from stdin is not possible then, though.
This way we can control whether "out" output goes to stdout or stderr
without the user of the functionality knowing.
This is useful for later when we use libimagrt to automatically
read and write the store from and to stdout/in depending on whether we
are talking to a TTY or a pipe.
Adapt libimagrt interface to export the functions we need to do this.
This is not that nice, but the best approach without rewriting large
parts of libimagrt.
Rewrite without regex crate.
The regex approach was broken. If the following _content_ was provided
in the entry:
foo
---
bar
The regex approach parsed the header until the "---" in the content.
This is, of course, not the way to do that.
Now, the parsing is implemented by hand. Should be faster as well,
though I don't care about this.
This fixes a severe bug.
This patch adds API functions in the StoreIdIteratorWithStore iterator
type to transform it into a iterator which _does_ something (as in the
`libimagstore::iter` API).
It mimics the API which is offered by `libimagstore::iter`.
This test is not applicable anymore because it tests (and tested) the
wrong thing.
It was to check whether the function failed because the "imag" key
contained the wrong type, but this is not tested by that function. The
function only checks whether the "imag" key is present.
Before we extracted the store configuration from the configuration
toml::Value object and passed it to the store.
This is unecessary overhead.
Now we pass the whole configuration object and let the store extract the
required values.
Before the configuration object (the raw TOML object) was provided via a
wrapper object `Configuration`. This was ugly and not very nice to use.
Now, we only have the `toml::Value` object we lend out from
`Runtime::config()`.
The changes included libimagrt internal rewrites, which are not visible
to the user. Anyways, this change changes the API for config-fetching
from the runtime, so fixes for all other crates may follow.
The changes also removed the support for reading the "editor" setting
from the configuration file, which was not used anyways (in the example
imagrc.toml file).
The CLI-reading and ENV-reading are still supported, though.
Formerly, the --debug flag was ignores. This change overrides the
logging with the CLI specified logging if it was provided.
If --debug was provided, the logging is set to debugging, if --verbose
was provided info logging is used.
Before the iterator did also yield storeids for directories, which was a
bug.
This change introduces a new if_file() function in the store-internal
backend, which is needed to check whether a path actually points to a
File, be it inmemory or on the real filesystem.
That's because tests might fail if they check via PathBuf::is_file() as
in tests, the entries only exist inmemory.
The logger was not able to handle multiple destinations before. Now it
is possible for the logger.
The file must be held behind an Arc<Mutex<_>> so we can use the logging
from multiple threads but also because we need to borrow mutably, so
that bit changes whith this commit.
From the documentation of Walkdir::min_depth():
Set the minimum depth of entries yielded by the iterator.
The smallest depth is 0 and always corresponds to the path given to
the new function on this type. Its direct descendents have depth 1,
and their descendents have depth 2, and so on.
This means that when we started with "/tmp/store", we end up yielding
that exact path in the first iteration. This is exactly what we do _not_
want.
Setting the minimal depth to 1 fixes this bug.
This changes the internal GlobStoreIdIterator to return Result<StoreId>,
which gives us the possibility to aggregate errors in the
Store::retrieve_for_module() function and return them instead of tracing
them from the store.
The changes the internals to actually fetch the whole list of storeids,
which is unfortunate of course, but changing the interface is not an
option here, in my opinion.
At least we're only aggregating pathes, so the memory usage is pretty
low here.
This patch fixes a problem where the Store::delete() function only
checked the store-internal cache whether an entry exists, but not the
Filesystem. After this patch is applied, the Store::delete() function
also checks the filesystem whether the entry exists.
As we rely on the filesystem in Store::entries(), which is a bug and
shouldn't be done, we rewrite this function and use the file_abstraction
framework.
This merge solved a _LOT_ of conflicts and was a rather complicated one,
as parts of the conflict-resolution involved rewriting of half the
stuff.
This merge commit fixes all the things so a `cargo check --all`
succeeds, but I did not yet check whether tests run without failure.
