Add functionality to flush the internal cache

Before we had the problem that when iterating over _a lot_ (like 5k)
entries and also fetching them, at some point the OS would return with
"Too many files open".

That is because the store internally caches a lot.

With this change, the Store gets an API to query how big the cache is,
how much the cache can currently hold and (and that's the main thing in
this patch) to flush the cache to disk.

A function to simply ask the store whether its cache should be flushed
(which would us require to ask the OS how many files we can open...
which would be possible with `libc::getrlimit`) does not yet exist,
though, but could be added easily if desired.
This commit is contained in:
Matthias Beyer 2018-04-30 18:23:20 +02:00
parent 76966bcd6c
commit 860e47a256
1 changed files with 43 additions and 0 deletions

View File

@ -477,6 +477,49 @@ impl Store {
Ok(())
}
/// Flush the store internal cache
///
/// This is helpful if a lot of entries are beeing read/written, because the store holds the
/// file handles internally. At some point, the OS simply errors with "Too many files open".
/// With this function, not-borrowed entries can be flushed back to disk and thus file handles
/// are dropped.
///
/// After the flushables are dropped, the internal cache is shrinked to fit the number of
/// elements still in the cache.
///
pub fn flush_cache(&self) -> Result<()> {
// We borrow this early so that between the aggregation of the flushables and the actual
// flush, there is no borrowing from the store.
let mut hsmap = self.entries.write().map_err(|_| SE::from_kind(SEK::LockPoisoned))?;
let mut to_flush = vec![];
for (storeid, se) in hsmap.deref() {
if !se.is_borrowed() {
to_flush.push(storeid.clone());
}
}
for id in to_flush {
let _ = hsmap.remove(&id);
}
hsmap.shrink_to_fit();
Ok(())
}
/// The number of elements in the internal cache
pub fn cache_size(&self) -> Result<usize> {
let hsmap = self.entries.read().map_err(|_| SE::from_kind(SEK::LockPoisoned))?;
Ok(hsmap.iter().count())
}
/// The size of the internal cache
pub fn cache_capacity(&self) -> Result<usize> {
let hsmap = self.entries.read().map_err(|_| SE::from_kind(SEK::LockPoisoned))?;
Ok(hsmap.capacity())
}
/// Get a copy of a given entry, this cannot be used to mutate the one on disk
///
/// # Return value