matthiasbeyer/imag
1
0
Fork 0
Code Releases Activity

Merge pull request #972 from matthiasbeyer/libimagstore/io-backend

Browse source 
Libimagstore/io backend
This commit is contained in:
Matthias Beyer 2017-06-18 12:31:38 +02:00 committed by GitHub
commit cd99873f17
13 changed files with 1075 additions and 422 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

167
doc/src/02000-store.md
View file

@ -146,3 +146,170 @@ moment. If the module "B" gets updated, it might update its entries in the store
as well. The link from the "a" should never get invalid in this case, though it
is not ensured by the core of imag itself.
## Backends {#sec:thestore:backends}
The store itself also has a backend. This backend is the "filesystem
abstraction" code.
Note: This is a very core thing. Casual users might want to skip this section.
### Problem {#sec:thestore:backends:problem}
First, we had a compiletime backend for the store.
This means that the actual filesystem operations were compiled into the stores
either as real filesystem operations (in a normal debug or release build) but as
a in-memory variant in the 'test' case.
So tests did not hit the filesystem when running.
This gave us us the possibility to run tests concurrently with multiple stores
that did not interfere with eachother.
This approach worked perfectly well until we started to test not the
store itself but crates that depend on the store implementation.
When running tests in a crate that depends on the store, the store
itself was compiled with the filesystem-hitting-backend.
This was problematic, as tests could not be implemented without hitting
the filesystem.
Hence we implemented this.
### Implementation {#sec:thestore:backends:implementation}
The filesystem is abstracted via a trait `FileAbstraction` which
contains the essential functions for working with the filesystem.
Two implementations are provided in the code:
* FSFileAbstraction
* InMemoryFileAbstraction
whereas the first actually works with the filesystem and the latter
works with an in-memory HashMap that is used as filesystem.
Further, the trait `FileAbstractionInstance` was introduced for
functions which are executed on actual instances of content from the
filesystem, which was previousely tied into the general abstraction
mechanism.
So, the `FileAbstraction` trait is for working with the filesystem, the
`FileAbstractionInstance` trait is for working with instances of content
from the filesystem (speak: actual Files).
In case of the `FSFileAbstractionInstance`, which is the implementation
of the `FileAbstractionInstance` for the actual filesystem-hitting code,
the underlying resource is managed like with the old code before.
The `InMemoryFileAbstractionInstance` implementation is corrosponding to
the `InMemoryFileAbstraction` implementation - for the in-memory
"filesystem".
The implementation of the `get_file_content()` function had to be
changed to return a `String` rather than a `&mut Read` because of
lifetime issues.
This change is store-internally and the API of the store itself was not
affected.
## The StdIo backend {#sec:thestore:backends:stdio}
Sidenote: The name is "StdIo" because its main purpose is Stdin/Stdio, but it
is abstracted over Read/Write actually, so it is also possible to use this
backend in other ways, too.
### Why? {#sec:thestore:backends:stdio:why}
This is a backend for the imag store which is created
from stdin, by piping contents into the store (via JSON or TOML) and piping the
store contents (as JSON or TOML) to stdout when the the backend is destructed.
This is one of some components which make command-chaining in imag possible.
With this, the application does not have to know whether the store actually
lives on the filesystem or just "in memory".
### Mappers {#sec:thestore:backends:stdio:mappers}
The backend contains a "Mapper" which defines how the contents get mapped into
the in-memory store representation: A JSON implementation or a TOML
implementation are possible.
The following section assumes a JSON mapper.
The backends themselves do not know "header" and "content" - they know only
blobs which live in pathes.
Indeed, this "backend" code does not serve "content" or "header" to the `Store`
implementation, but only a blob of bytes.
Anyways, the JSON-protocol for passing a store around _does_ know about content
and header (see @sec:thestore:backends:stdio:json for the JSON format).
So the mapper reads the JSON, parses it (thanks serde!) and translates it to
TOML, because TOML is the Store representation of a header.
But because the backend does not serve header and content, but only a blob,
this TOML is then translated (with the content of the respective file) to a
blob.
This is then made available to the store codebase.
This is complex and probably slow, we know.
To summarize what we do right now, lets have a look at the awesome ascii-art
below:
```
libimag*
|
v
IO Mapper FS Store FS Mapper IO
+--+-------------+---------+--------+---------+--------------+--+
| | | | | | | |
JSON -> TOML -> String -> Entry -> String -> TOML -> JSON
+ TOML
+ Content
```
This is what gets translated where for one imag call with a stdio store backend.
The rationale behind this implementation is that this is the best implementation
we can have in a relatively short amount of time.
### The JSON Mapper {#sec:thestore:backends:stdio:json}
The JSON mapper maps JSON which is read from a source into a HashMap which
represents the in-memory filesystem.
The strucure is as follows:
```json
{
"version": "0.3.0",
"store": {
"/example": {
"header": {
"imag": {
"version": "0.3.0",
},
},
"content": "hi there!",
},
},
}
```
### TODO {#sec:thestore:backends:todo}
Of course, the above is not optimal.
The TODO here is almost visible: Implement a proper backend where we do not need
to translate between types all the time.
The first goal would be to reduce the above figure to:
```
libimag*
|
v
IO Mapper Store Mapper IO
+--+------+--------+-------+--+
| | | | | |
JSON -> Entry -> JSON
+ TOML
+ Content
```
and the second step would be to abstract all the things away so the `libimag*`
crates handle the header without knowing whether it is JSON or TOML.

3
libimagstore/Cargo.toml
View file

@ -26,6 +26,9 @@ crossbeam = "0.2.*"
walkdir = "1.0.*"
itertools = "0.6.*"
is-match = "0.1"
serde = "1.0"
serde_json = "1.0"
serde_derive = "1.0"
[dependencies.libimagerror]
path = "../libimagerror"

2
libimagstore/src/error.rs
View file

@ -34,6 +34,8 @@ generate_custom_error_types!(StoreError, StoreErrorKind, CustomErrorData,
ConfigTypeError => "Store configuration type error",
ConfigKeyMissingError => "Configuration Key missing",
VersionError => "Incompatible store versions detected",
CreateStoreDirDenied => "Creating store directory implicitely denied",
FileError => "File Error",
IoError => "IO Error",

397
libimagstore/src/file_abstraction.rs
View file

@ -1,397 +0,0 @@
//
// imag - the personal information management suite for the commandline
// Copyright (C) 2015, 2016 Matthias Beyer <mail@beyermatthias.de> and contributors
//
// This library is free software; you can redistribute it and/or
// modify it under the terms of the GNU Lesser General Public
// License as published by the Free Software Foundation; version
// 2.1 of the License.
//
// This library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
// Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public
// License along with this library; if not, write to the Free Software
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
//
//! The filesystem abstraction code
//!
//! # Problem
//!
//! First, we had a compiletime backend for the store. This means that the actual filesystem
//! operations were compiled into the store either as real filesystem operations (in a normal debug
//! or release build) but as a in-memory variant in the 'test' case.
//! So tests did not hit the filesystem when running.
//! This gave us us the possibility to run tests concurrently with multiple
//! stores that did not interfere with eachother.
//!
//! This approach worked perfectly well until we started to test not the
//! store itself but crates that depend on the store implementation.
//! When running tests in a crate that depends on the store, the store
//! itself was compiled with the filesystem-hitting-backend.
//! This was problematic, as tests could not be implemented without hitting
//! the filesystem.
//!
//! Hence we implemented this.
//!
//! # Implementation
//!
//! The filesystem is abstracted via a trait `FileAbstraction` which
//! contains the essential functions for working with the filesystem.
//!
//! Two implementations are provided in the code:
//!
//! * FSFileAbstraction
//! * InMemoryFileAbstraction
//!
//! whereas the first actually works with the filesystem and the latter
//! works with an in-memory HashMap that is used as filesystem.
//!
//! Further, the trait `FileAbstractionInstance` was introduced for
//! functions which are executed on actual instances of content from the
//! filesystem, which was previousely tied into the general abstraction
//! mechanism.
//!
//! So, the `FileAbstraction` trait is for working with the filesystem, the
//! `FileAbstractionInstance` trait is for working with instances of content
//! from the filesystem (speak: actual Files).
//!
//! In case of the `FSFileAbstractionInstance`, which is the implementation
//! of the `FileAbstractionInstance` for the actual filesystem-hitting code,
//! the underlying resource is managed like with the old code before.
//! The `InMemoryFileAbstractionInstance` implementation is corrosponding to
//! the `InMemoryFileAbstraction` implementation - for the in-memory
//! "filesystem".
//!
//! The implementation of the `get_file_content()` function had to be
//! changed to return a `String` rather than a `&mut Read` because of
//! lifetime issues.
//! This change is store-internally and the API of the store itself was not
//! affected.
//!
use std::path::PathBuf;
use std::fmt::Debug;
use error::StoreError as SE;
pub use self::fs::FSFileAbstraction;
pub use self::fs::FSFileAbstractionInstance;
pub use self::inmemory::InMemoryFileAbstraction;
pub use self::inmemory::InMemoryFileAbstractionInstance;
/// An abstraction trait over filesystem actions
pub trait FileAbstraction : Debug {
fn remove_file(&self, path: &PathBuf) -> Result<(), SE>;
fn copy(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE>;
fn rename(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE>;
fn create_dir_all(&self, _: &PathBuf) -> Result<(), SE>;
fn new_instance(&self, p: PathBuf) -> Box<FileAbstractionInstance>;
}
/// An abstraction trait over actions on files
pub trait FileAbstractionInstance : Debug {
fn get_file_content(&mut self) -> Result<String, SE>;
fn write_file_content(&mut self, buf: &[u8]) -> Result<(), SE>;
}
mod fs {
use std::fs::{File, OpenOptions, create_dir_all, remove_file, copy, rename};
use std::io::{Seek, SeekFrom, Read};
use std::path::{Path, PathBuf};
use error::{MapErrInto, StoreError as SE, StoreErrorKind as SEK};
use super::FileAbstraction;
use super::FileAbstractionInstance;
#[derive(Debug)]
pub enum FSFileAbstractionInstance {
Absent(PathBuf),
File(File, PathBuf)
}
impl FileAbstractionInstance for FSFileAbstractionInstance {
/**
* Get the content behind this file
*/
fn get_file_content(&mut self) -> Result<String, SE> {
debug!("Getting lazy file: {:?}", self);
let (file, path) = match *self {
FSFileAbstractionInstance::File(ref mut f, _) => return {
// We seek to the beginning of the file since we expect each
// access to the file to be in a different context
try!(f.seek(SeekFrom::Start(0))
.map_err_into(SEK::FileNotSeeked));
let mut s = String::new();
f.read_to_string(&mut s)
.map_err_into(SEK::IoError)
.map(|_| s)
},
FSFileAbstractionInstance::Absent(ref p) =>
(try!(open_file(p).map_err_into(SEK::FileNotFound)), p.clone()),
};
*self = FSFileAbstractionInstance::File(file, path);
if let FSFileAbstractionInstance::File(ref mut f, _) = *self {
let mut s = String::new();
f.read_to_string(&mut s)
.map_err_into(SEK::IoError)
.map(|_| s)
} else {
unreachable!()
}
}
/**
* Write the content of this file
*/
fn write_file_content(&mut self, buf: &[u8]) -> Result<(), SE> {
use std::io::Write;
let (file, path) = match *self {
FSFileAbstractionInstance::File(ref mut f, _) => return {
// We seek to the beginning of the file since we expect each
// access to the file to be in a different context
try!(f.seek(SeekFrom::Start(0))
.map_err_into(SEK::FileNotCreated));
f.write_all(buf).map_err_into(SEK::FileNotWritten)
},
FSFileAbstractionInstance::Absent(ref p) =>
(try!(create_file(p).map_err_into(SEK::FileNotCreated)), p.clone()),
};
*self = FSFileAbstractionInstance::File(file, path);
if let FSFileAbstractionInstance::File(ref mut f, _) = *self {
return f.write_all(buf).map_err_into(SEK::FileNotWritten);
}
unreachable!();
}
}
/// `FSFileAbstraction` state type
///
/// A lazy file is either absent, but a path to it is available, or it is present.
#[derive(Debug)]
pub struct FSFileAbstraction {
}
impl FSFileAbstraction {
pub fn new() -> FSFileAbstraction {
FSFileAbstraction { }
}
}
impl FileAbstraction for FSFileAbstraction {
fn remove_file(&self, path: &PathBuf) -> Result<(), SE> {
remove_file(path).map_err_into(SEK::FileNotRemoved)
}
fn copy(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE> {
copy(from, to).map_err_into(SEK::FileNotCopied).map(|_| ())
}
fn rename(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE> {
rename(from, to).map_err_into(SEK::FileNotRenamed)
}
fn create_dir_all(&self, path: &PathBuf) -> Result<(), SE> {
create_dir_all(path).map_err_into(SEK::DirNotCreated)
}
fn new_instance(&self, p: PathBuf) -> Box<FileAbstractionInstance> {
Box::new(FSFileAbstractionInstance::Absent(p))
}
}
fn open_file<A: AsRef<Path>>(p: A) -> ::std::io::Result<File> {
OpenOptions::new().write(true).read(true).open(p)
}
fn create_file<A: AsRef<Path>>(p: A) -> ::std::io::Result<File> {
if let Some(parent) = p.as_ref().parent() {
debug!("Implicitely creating directory: {:?}", parent);
if let Err(e) = create_dir_all(parent) {
return Err(e);
}
}
OpenOptions::new().write(true).read(true).create(true).open(p)
}
}
mod inmemory {
use error::StoreError as SE;
use error::StoreErrorKind as SEK;
use std::io::Read;
use std::io::Cursor;
use std::path::PathBuf;
use std::collections::HashMap;
use std::sync::Mutex;
use std::cell::RefCell;
use std::sync::Arc;
use libimagerror::into::IntoError;
use super::FileAbstraction;
use super::FileAbstractionInstance;
use error::MapErrInto;
type Backend = Arc<Mutex<RefCell<HashMap<PathBuf, Cursor<Vec<u8>>>>>>;
/// `FileAbstraction` type, this is the Test version!
///
/// A lazy file is either absent, but a path to it is available, or it is present.
#[derive(Debug)]
pub struct InMemoryFileAbstractionInstance {
fs_abstraction: Backend,
absent_path: PathBuf,
}
impl InMemoryFileAbstractionInstance {
pub fn new(fs: Backend, pb: PathBuf) -> InMemoryFileAbstractionInstance {
InMemoryFileAbstractionInstance {
fs_abstraction: fs,
absent_path: pb
}
}
}
impl FileAbstractionInstance for InMemoryFileAbstractionInstance {
/**
* Get the mutable file behind a InMemoryFileAbstraction object
*/
fn get_file_content(&mut self) -> Result<String, SE> {
debug!("Getting lazy file: {:?}", self);
let p = self.absent_path.clone();
match self.fs_abstraction.lock() {
Ok(mut mtx) => {
mtx.get_mut()
.get_mut(&p)
.ok_or(SEK::FileNotFound.into_error())
.and_then(|t| {
let mut s = String::new();
t.read_to_string(&mut s)
.map_err_into(SEK::IoError)
.map(|_| s)
})
}
Err(_) => Err(SEK::LockError.into_error())
}
}
fn write_file_content(&mut self, buf: &[u8]) -> Result<(), SE> {
match *self {
InMemoryFileAbstractionInstance { ref absent_path, .. } => {
let mut mtx = self.fs_abstraction.lock().expect("Locking Mutex failed");
let mut backend = mtx.get_mut();
if let Some(ref mut cur) = backend.get_mut(absent_path) {
let mut vec = cur.get_mut();
vec.clear();
vec.extend_from_slice(buf);
return Ok(());
}
let vec = Vec::from(buf);
backend.insert(absent_path.clone(), Cursor::new(vec));
return Ok(());
},
};
}
}
#[derive(Debug)]
pub struct InMemoryFileAbstraction {
virtual_filesystem: Backend,
}
impl InMemoryFileAbstraction {
pub fn new() -> InMemoryFileAbstraction {
InMemoryFileAbstraction {
virtual_filesystem: Arc::new(Mutex::new(RefCell::new(HashMap::new()))),
}
}
pub fn backend(&self) -> &Backend {
&self.virtual_filesystem
}
}
impl FileAbstraction for InMemoryFileAbstraction {
fn remove_file(&self, path: &PathBuf) -> Result<(), SE> {
debug!("Removing: {:?}", path);
self.backend()
.lock()
.expect("Locking Mutex failed")
.get_mut()
.remove(path)
.map(|_| ())
.ok_or(SEK::FileNotFound.into_error())
}
fn copy(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE> {
debug!("Copying : {:?} -> {:?}", from, to);
let mut mtx = self.backend().lock().expect("Locking Mutex failed");
let mut backend = mtx.get_mut();
let a = try!(backend.get(from).cloned().ok_or(SEK::FileNotFound.into_error()));
backend.insert(to.clone(), a);
debug!("Copying: {:?} -> {:?} worked", from, to);
Ok(())
}
fn rename(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE> {
debug!("Renaming: {:?} -> {:?}", from, to);
let mut mtx = self.backend().lock().expect("Locking Mutex failed");
let mut backend = mtx.get_mut();
let a = try!(backend.get(from).cloned().ok_or(SEK::FileNotFound.into_error()));
backend.insert(to.clone(), a);
debug!("Renaming: {:?} -> {:?} worked", from, to);
Ok(())
}
fn create_dir_all(&self, _: &PathBuf) -> Result<(), SE> {
Ok(())
}
fn new_instance(&self, p: PathBuf) -> Box<FileAbstractionInstance> {
Box::new(InMemoryFileAbstractionInstance::new(self.backend().clone(), p))
}
}
}
#[cfg(test)]
mod test {
use super::FileAbstractionInstance;
use super::inmemory::InMemoryFileAbstraction;
use super::inmemory::InMemoryFileAbstractionInstance;
use std::path::PathBuf;
#[test]
fn lazy_file() {
let fs = InMemoryFileAbstraction::new();
let mut path = PathBuf::from("/tests");
path.set_file_name("test1");
let mut lf = InMemoryFileAbstractionInstance::new(fs.backend().clone(), path);
lf.write_file_content(b"Hello World").unwrap();
let bah = lf.get_file_content().unwrap();
assert_eq!(bah, "Hello World");
}
}

142
libimagstore/src/file_abstraction/fs.rs Normal file
View file

@ -0,0 +1,142 @@
//
// imag - the personal information management suite for the commandline
// Copyright (C) 2015, 2016 Matthias Beyer <mail@beyermatthias.de> and contributors
//
// This library is free software; you can redistribute it and/or
// modify it under the terms of the GNU Lesser General Public
// License as published by the Free Software Foundation; version
// 2.1 of the License.
//
// This library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
// Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public
// License along with this library; if not, write to the Free Software
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
//
use std::fs::{File, OpenOptions, create_dir_all, remove_file, copy, rename};
use std::io::{Seek, SeekFrom, Read};
use std::path::{Path, PathBuf};
use error::{MapErrInto, StoreError as SE, StoreErrorKind as SEK};
use super::FileAbstraction;
use super::FileAbstractionInstance;
#[derive(Debug)]
pub enum FSFileAbstractionInstance {
Absent(PathBuf),
File(File, PathBuf)
}
impl FileAbstractionInstance for FSFileAbstractionInstance {
/**
* Get the content behind this file
*/
fn get_file_content(&mut self) -> Result<String, SE> {
debug!("Getting lazy file: {:?}", self);
let (file, path) = match *self {
FSFileAbstractionInstance::File(ref mut f, _) => return {
// We seek to the beginning of the file since we expect each
// access to the file to be in a different context
try!(f.seek(SeekFrom::Start(0))
.map_err_into(SEK::FileNotSeeked));
let mut s = String::new();
f.read_to_string(&mut s)
.map_err_into(SEK::IoError)
.map(|_| s)
},
FSFileAbstractionInstance::Absent(ref p) =>
(try!(open_file(p).map_err_into(SEK::FileNotFound)), p.clone()),
};
*self = FSFileAbstractionInstance::File(file, path);
if let FSFileAbstractionInstance::File(ref mut f, _) = *self {
let mut s = String::new();
f.read_to_string(&mut s)
.map_err_into(SEK::IoError)
.map(|_| s)
} else {
unreachable!()
}
}
/**
* Write the content of this file
*/
fn write_file_content(&mut self, buf: &[u8]) -> Result<(), SE> {
use std::io::Write;
let (file, path) = match *self {
FSFileAbstractionInstance::File(ref mut f, _) => return {
// We seek to the beginning of the file since we expect each
// access to the file to be in a different context
try!(f.seek(SeekFrom::Start(0))
.map_err_into(SEK::FileNotCreated));
f.write_all(buf).map_err_into(SEK::FileNotWritten)
},
FSFileAbstractionInstance::Absent(ref p) =>
(try!(create_file(p).map_err_into(SEK::FileNotCreated)), p.clone()),
};
*self = FSFileAbstractionInstance::File(file, path);
if let FSFileAbstractionInstance::File(ref mut f, _) = *self {
return f.write_all(buf).map_err_into(SEK::FileNotWritten);
}
unreachable!();
}
}
/// `FSFileAbstraction` state type
///
/// A lazy file is either absent, but a path to it is available, or it is present.
#[derive(Debug)]
pub struct FSFileAbstraction {
}
impl FSFileAbstraction {
pub fn new() -> FSFileAbstraction {
FSFileAbstraction { }
}
}
impl FileAbstraction for FSFileAbstraction {
fn remove_file(&self, path: &PathBuf) -> Result<(), SE> {
remove_file(path).map_err_into(SEK::FileNotRemoved)
}
fn copy(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE> {
copy(from, to).map_err_into(SEK::FileNotCopied).map(|_| ())
}
fn rename(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE> {
rename(from, to).map_err_into(SEK::FileNotRenamed)
}
fn create_dir_all(&self, path: &PathBuf) -> Result<(), SE> {
create_dir_all(path).map_err_into(SEK::DirNotCreated)
}
fn new_instance(&self, p: PathBuf) -> Box<FileAbstractionInstance> {
Box::new(FSFileAbstractionInstance::Absent(p))
}
}
fn open_file<A: AsRef<Path>>(p: A) -> ::std::io::Result<File> {
OpenOptions::new().write(true).read(true).open(p)
}
fn create_file<A: AsRef<Path>>(p: A) -> ::std::io::Result<File> {
if let Some(parent) = p.as_ref().parent() {
debug!("Implicitely creating directory: {:?}", parent);
if let Err(e) = create_dir_all(parent) {
return Err(e);
}
}
OpenOptions::new().write(true).read(true).create(true).open(p)
}

166
libimagstore/src/file_abstraction/inmemory.rs Normal file
View file

@ -0,0 +1,166 @@
//
// imag - the personal information management suite for the commandline
// Copyright (C) 2015, 2016 Matthias Beyer <mail@beyermatthias.de> and contributors
//
// This library is free software; you can redistribute it and/or
// modify it under the terms of the GNU Lesser General Public
// License as published by the Free Software Foundation; version
// 2.1 of the License.
//
// This library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
// Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public
// License along with this library; if not, write to the Free Software
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
//
use error::StoreError as SE;
use error::StoreErrorKind as SEK;
use std::io::Read;
use std::io::Cursor;
use std::path::PathBuf;
use std::collections::HashMap;
use std::sync::Mutex;
use std::cell::RefCell;
use std::sync::Arc;
use libimagerror::into::IntoError;
use super::FileAbstraction;
use super::FileAbstractionInstance;
use error::MapErrInto;
type Backend = Arc<Mutex<RefCell<HashMap<PathBuf, Cursor<Vec<u8>>>>>>;
/// `FileAbstraction` type, this is the Test version!
///
/// A lazy file is either absent, but a path to it is available, or it is present.
#[derive(Debug)]
pub struct InMemoryFileAbstractionInstance {
fs_abstraction: Backend,
absent_path: PathBuf,
}
impl InMemoryFileAbstractionInstance {
pub fn new(fs: Backend, pb: PathBuf) -> InMemoryFileAbstractionInstance {
InMemoryFileAbstractionInstance {
fs_abstraction: fs,
absent_path: pb
}
}
}
impl FileAbstractionInstance for InMemoryFileAbstractionInstance {
/**
* Get the mutable file behind a InMemoryFileAbstraction object
*/
fn get_file_content(&mut self) -> Result<String, SE> {
debug!("Getting lazy file: {:?}", self);
let p = self.absent_path.clone();
match self.fs_abstraction.lock() {
Ok(mut mtx) => {
mtx.get_mut()
.get_mut(&p)
.ok_or(SEK::FileNotFound.into_error())
.and_then(|t| {
let mut s = String::new();
t.read_to_string(&mut s)
.map_err_into(SEK::IoError)
.map(|_| s)
})
}
Err(_) => Err(SEK::LockError.into_error())
}
}
fn write_file_content(&mut self, buf: &[u8]) -> Result<(), SE> {
match *self {
InMemoryFileAbstractionInstance { ref absent_path, .. } => {
let mut mtx = self.fs_abstraction.lock().expect("Locking Mutex failed");
let mut backend = mtx.get_mut();
if let Some(ref mut cur) = backend.get_mut(absent_path) {
let mut vec = cur.get_mut();
vec.clear();
vec.extend_from_slice(buf);
return Ok(());
}
let vec = Vec::from(buf);
backend.insert(absent_path.clone(), Cursor::new(vec));
return Ok(());
},
};
}
}
#[derive(Debug)]
pub struct InMemoryFileAbstraction {
virtual_filesystem: Backend,
}
impl InMemoryFileAbstraction {
pub fn new() -> InMemoryFileAbstraction {
InMemoryFileAbstraction {
virtual_filesystem: Arc::new(Mutex::new(RefCell::new(HashMap::new()))),
}
}
pub fn backend(&self) -> &Backend {
&self.virtual_filesystem
}
}
impl FileAbstraction for InMemoryFileAbstraction {
fn remove_file(&self, path: &PathBuf) -> Result<(), SE> {
debug!("Removing: {:?}", path);
self.backend()
.lock()
.expect("Locking Mutex failed")
.get_mut()
.remove(path)
.map(|_| ())
.ok_or(SEK::FileNotFound.into_error())
}
fn copy(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE> {
debug!("Copying : {:?} -> {:?}", from, to);
let mut mtx = self.backend().lock().expect("Locking Mutex failed");
let mut backend = mtx.get_mut();
let a = try!(backend.get(from).cloned().ok_or(SEK::FileNotFound.into_error()));
backend.insert(to.clone(), a);
debug!("Copying: {:?} -> {:?} worked", from, to);
Ok(())
}
fn rename(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE> {
debug!("Renaming: {:?} -> {:?}", from, to);
let mut mtx = self.backend().lock().expect("Locking Mutex failed");
let mut backend = mtx.get_mut();
let a = try!(backend.get(from).cloned().ok_or(SEK::FileNotFound.into_error()));
backend.insert(to.clone(), a);
debug!("Renaming: {:?} -> {:?} worked", from, to);
Ok(())
}
fn create_dir_all(&self, _: &PathBuf) -> Result<(), SE> {
Ok(())
}
fn new_instance(&self, p: PathBuf) -> Box<FileAbstractionInstance> {
Box::new(InMemoryFileAbstractionInstance::new(self.backend().clone(), p))
}
}

71
libimagstore/src/file_abstraction/mod.rs Normal file
View file

@ -0,0 +1,71 @@
//
// imag - the personal information management suite for the commandline
// Copyright (C) 2015, 2016 Matthias Beyer <mail@beyermatthias.de> and contributors
//
// This library is free software; you can redistribute it and/or
// modify it under the terms of the GNU Lesser General Public
// License as published by the Free Software Foundation; version
// 2.1 of the License.
//
// This library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
// Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public
// License along with this library; if not, write to the Free Software
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
//
use std::path::PathBuf;
use std::fmt::Debug;
use error::StoreError as SE;
mod fs;
mod inmemory;
pub mod stdio;
pub use self::fs::FSFileAbstraction;
pub use self::fs::FSFileAbstractionInstance;
pub use self::inmemory::InMemoryFileAbstraction;
pub use self::inmemory::InMemoryFileAbstractionInstance;
/// An abstraction trait over filesystem actions
pub trait FileAbstraction : Debug {
fn remove_file(&self, path: &PathBuf) -> Result<(), SE>;
fn copy(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE>;
fn rename(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE>;
fn create_dir_all(&self, _: &PathBuf) -> Result<(), SE>;
fn new_instance(&self, p: PathBuf) -> Box<FileAbstractionInstance>;
}
/// An abstraction trait over actions on files
pub trait FileAbstractionInstance : Debug {
fn get_file_content(&mut self) -> Result<String, SE>;
fn write_file_content(&mut self, buf: &[u8]) -> Result<(), SE>;
}
#[cfg(test)]
mod test {
use super::FileAbstractionInstance;
use super::inmemory::InMemoryFileAbstraction;
use super::inmemory::InMemoryFileAbstractionInstance;
use std::path::PathBuf;
#[test]
fn lazy_file() {
let fs = InMemoryFileAbstraction::new();
let mut path = PathBuf::from("/tests");
path.set_file_name("test1");
let mut lf = InMemoryFileAbstractionInstance::new(fs.backend().clone(), path);
lf.write_file_content(b"Hello World").unwrap();
let bah = lf.get_file_content().unwrap();
assert_eq!(bah, "Hello World");
}
}

225
libimagstore/src/file_abstraction/stdio/mapper/json.rs Normal file
View file

@ -0,0 +1,225 @@
//
// imag - the personal information management suite for the commandline
// Copyright (C) 2015, 2016 Matthias Beyer <mail@beyermatthias.de> and contributors
//
// This library is free software; you can redistribute it and/or
// modify it under the terms of the GNU Lesser General Public
// License as published by the Free Software Foundation; version
// 2.1 of the License.
//
// This library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
// Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public
// License along with this library; if not, write to the Free Software
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
//
use std::collections::HashMap;
use std::io::Cursor;
use std::io::{Read, Write};
use std::path::PathBuf;
use serde_json;
use toml;
use error::StoreErrorKind as SEK;
use error::MapErrInto;
use super::Mapper;
use store::Result;
use libimagerror::into::IntoError;
#[derive(Deserialize, Serialize)]
struct Entry {
header: serde_json::Value,
content: String,
}
impl Entry {
fn to_string(self) -> Result<String> {
toml::to_string(&self.header)
.map_err_into(SEK::IoError)
.map(|hdr| {
format!("---\n{header}---\n{content}",
header = hdr,
content = self.content)
})
}
}
#[derive(Deserialize, Serialize)]
struct Document {
version: String,
store: HashMap<PathBuf, Entry>,
}
pub struct JsonMapper;
impl JsonMapper {
pub fn new() -> JsonMapper {
JsonMapper
}
}
impl Mapper for JsonMapper {
fn read_to_fs<R: Read>(&self, r: &mut R, hm: &mut HashMap<PathBuf, Cursor<Vec<u8>>>) -> Result<()> {
let mut document = {
let mut s = String::new();
try!(r.read_to_string(&mut s).map_err_into(SEK::IoError));
let doc : Document = try!(serde_json::from_str(&s).map_err_into(SEK::IoError));
doc
};
let _ = try!(::semver::Version::parse(&document.version)
.map_err_into(SEK::VersionError)
.and_then(|doc_vers| {
// safe because cargo does not compile if crate version is not valid
let crate_version = ::semver::Version::parse(version!()).unwrap();
if doc_vers > crate_version {
Err(SEK::VersionError.into_error())
} else {
Ok(())
}
}));
for (key, val) in document.store.drain() {
let res = val
.to_string()
.map(|vals| hm.insert(key, Cursor::new(vals.into_bytes())))
.map(|_| ());
let _ = try!(res);
}
Ok(())
}
fn fs_to_write<W: Write>(&self, hm: &mut HashMap<PathBuf, Cursor<Vec<u8>>>, out: &mut W) -> Result<()> {
use util::entry_buffer_to_header_content;
#[derive(Serialize, Deserialize)]
struct Entry {
header: ::toml::Value,
content: String,
}
#[derive(Serialize)]
struct OutDocument {
version: String,
store: HashMap<PathBuf, Entry>,
}
let mut doc = OutDocument {
version: String::from(version!()),
store: HashMap::new(),
};
for (key, value) in hm.drain() {
let res = String::from_utf8(value.into_inner())
.map_err_into(SEK::IoError)
.and_then(|buf| entry_buffer_to_header_content(&buf))
.map(|(header, content)| {
let entry = Entry {
header: header,
content: content
};
doc.store.insert(key, entry);
})
.map(|_| ());
let _ = try!(res);
}
serde_json::to_string(&doc)
.map_err_into(SEK::IoError)
.and_then(|json| out.write(&json.into_bytes()).map_err_into(SEK::IoError))
.and_then(|_| out.flush().map_err_into(SEK::IoError))
.map(|_| ())
}
}
#[cfg(test)]
mod test {
use std::io::Cursor;
use super::*;
#[test]
fn test_json_to_fs() {
let json = r#"
{ "version": "0.3.0",
"store": {
"/example": {
"header": {
"imag": {
"version": "0.3.0"
}
},
"content": "test"
}
}
}
"#;
let mut json = Cursor::new(String::from(json).into_bytes());
let mapper = JsonMapper::new();
let mut hm = HashMap::new();
let io_res = mapper.read_to_fs(&mut json, &mut hm);
assert!(io_res.is_ok());
assert_eq!(1, hm.len()); // we should have exactly one entry
}
#[test]
fn test_fs_to_json() {
let mapper = JsonMapper::new();
let mut out : Cursor<Vec<u8>> = Cursor::new(vec![]);
let mut hm = {
let mut hm = HashMap::new();
let content = r#"---
[imag]
version = "0.3.0"
---
hi there!"#;
hm.insert(PathBuf::from("/example"), Cursor::new(String::from(content).into_bytes()));
hm
};
let io_res = mapper.fs_to_write(&mut hm, &mut out);
assert!(io_res.is_ok());
let example = r#"
{
"version": "0.3.0",
"store": {
"/example": {
"header": {
"imag": {
"version": "0.3.0"
}
},
"content": "hi there!"
}
}
}
"#;
let example_json : ::serde_json::Value = ::serde_json::from_str(example).unwrap();
let output_json = String::from_utf8(out.into_inner()).unwrap();
let output_json : ::serde_json::Value = ::serde_json::from_str(&output_json).unwrap();
assert_eq!(example_json, output_json);
}
}

32
libimagstore/src/file_abstraction/stdio/mapper/mod.rs Normal file
View file

@ -0,0 +1,32 @@
//
// imag - the personal information management suite for the commandline
// Copyright (C) 2015, 2016 Matthias Beyer <mail@beyermatthias.de> and contributors
//
// This library is free software; you can redistribute it and/or
// modify it under the terms of the GNU Lesser General Public
// License as published by the Free Software Foundation; version
// 2.1 of the License.
//
// This library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
// Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public
// License along with this library; if not, write to the Free Software
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
//
use std::collections::HashMap;
use std::io::Cursor;
use std::io::{Read, Write};
use std::path::PathBuf;
use store::Result;
pub trait Mapper {
fn read_to_fs<R: Read>(&self, &mut R, &mut HashMap<PathBuf, Cursor<Vec<u8>>>) -> Result<()>;
fn fs_to_write<W: Write>(&self, &mut HashMap<PathBuf, Cursor<Vec<u8>>>, &mut W) -> Result<()>;
}
pub mod json;

135
libimagstore/src/file_abstraction/stdio/mod.rs Normal file
View file

@ -0,0 +1,135 @@
//
// imag - the personal information management suite for the commandline
// Copyright (C) 2015, 2016 Matthias Beyer <mail@beyermatthias.de> and contributors
//
// This library is free software; you can redistribute it and/or
// modify it under the terms of the GNU Lesser General Public
// License as published by the Free Software Foundation; version
// 2.1 of the License.
//
// This library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
// Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public
// License along with this library; if not, write to the Free Software
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
//
use std::rc::Rc;
use std::cell::RefCell;
use std::collections::HashMap;
use std::fmt::Debug;
use std::fmt::Error as FmtError;
use std::fmt::Formatter;
use std::io::Cursor;
use std::io::{Read, Write};
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::Mutex;
use libimagerror::into::IntoError;
use libimagerror::trace::*;
use error::StoreErrorKind as SEK;
use error::StoreError as SE;
use super::FileAbstraction;
use super::FileAbstractionInstance;
use super::InMemoryFileAbstraction;
pub mod mapper;
use self::mapper::Mapper;
// Because this is not exported in super::inmemory;
type Backend = Arc<Mutex<RefCell<HashMap<PathBuf, Cursor<Vec<u8>>>>>>;
pub struct StdIoFileAbstraction<W: Write, M: Mapper> {
mapper: M,
mem: InMemoryFileAbstraction,
out: Rc<RefCell<W>>,
}
impl<W, M> Debug for StdIoFileAbstraction<W, M>
where M: Mapper,
W: Write
{
fn fmt(&self, f: &mut Formatter) -> Result<(), FmtError> {
write!(f, "StdIoFileAbstraction({:?}", self.mem)
}
}
impl<W, M> StdIoFileAbstraction<W, M>
where M: Mapper,
W: Write
{
pub fn new<R: Read>(in_stream: &mut R, out_stream: Rc<RefCell<W>>, mapper: M) -> Result<StdIoFileAbstraction<W, M>, SE> {
let mem = InMemoryFileAbstraction::new();
{
let fill_res = match mem.backend().lock() {
Err(_) => Err(SEK::LockError.into_error()),
Ok(mut mtx) => mapper.read_to_fs(in_stream, mtx.get_mut())
};
let _ = try!(fill_res);
}
Ok(StdIoFileAbstraction {
mapper: mapper,
mem: mem,
out: out_stream,
})
}
pub fn backend(&self) -> &Backend {
&self.mem.backend()
}
}
impl<W, M> Drop for StdIoFileAbstraction<W, M>
where M: Mapper,
W: Write
{
fn drop(&mut self) {
use std::ops::DerefMut;
let fill_res = match self.mem.backend().lock() {
Err(_) => Err(SEK::LockError.into_error()),
Ok(mut mtx) => {
self.mapper.fs_to_write(mtx.get_mut(), self.out.borrow_mut().deref_mut())
},
};
// We can do nothing but end this here with a trace.
// As this drop gets called when imag almost exits, there is no point in exit()ing here
// again.
let _ = fill_res.map_err_trace();
}
}
impl<W: Write, M: Mapper> FileAbstraction for StdIoFileAbstraction<W, M> {
fn remove_file(&self, path: &PathBuf) -> Result<(), SE> {
self.mem.remove_file(path)
}
fn copy(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE> {
self.mem.copy(from, to)
}
fn rename(&self, from: &PathBuf, to: &PathBuf) -> Result<(), SE> {
self.mem.rename(from, to)
}
fn create_dir_all(&self, pb: &PathBuf) -> Result<(), SE> {
self.mem.create_dir_all(pb)
}
fn new_instance(&self, p: PathBuf) -> Box<FileAbstractionInstance> {
self.mem.new_instance(p)
}
}

3
libimagstore/src/lib.rs
View file

@ -44,6 +44,9 @@ extern crate crossbeam;
extern crate walkdir;
extern crate itertools;
#[macro_use] extern crate is_match;
#[macro_use] extern crate serde;
#[macro_use] extern crate serde_json;
#[macro_use] extern crate serde_derive;
#[macro_use] extern crate libimagerror;
extern crate libimagutil;

119
libimagstore/src/store.rs
View file

@ -24,7 +24,6 @@ use std::result::Result as RResult;
use std::sync::Arc;
use std::sync::RwLock;
use std::io::Read;
use std::convert::From;
use std::convert::Into;
use std::ops::Deref;
use std::ops::DerefMut;
@ -33,7 +32,6 @@ use std::fmt::Debug;
use std::fmt::Error as FMTError;
use toml::Value;
use regex::Regex;
use glob::glob;
use walkdir::WalkDir;
use walkdir::Iter as WalkDirIter;
@ -919,33 +917,14 @@ impl Entry {
/// - Header cannot be parsed into a TOML object
///
pub fn from_str<S: IntoStoreId>(loc: S, s: &str) -> Result<Entry> {
debug!("Building entry from string");
lazy_static! {
static ref RE: Regex = Regex::new(r"(?smx)
^---$
(?P<header>.*) # Header
^---$\n
(?P<content>.*) # Content
").unwrap();
}
use util::entry_buffer_to_header_content;
let matches = match RE.captures(s) {
None => return Err(SE::new(SEK::MalformedEntry, None)),
Some(s) => s,
};
let (header, content) = try!(entry_buffer_to_header_content(s));
let header = match matches.name("header") {
None => return Err(SE::new(SEK::MalformedEntry, None)),
Some(s) => s
};
let content = matches.name("content").map(|r| r.as_str()).unwrap_or("");
debug!("Header and content found. Yay! Building Entry object now");
Ok(Entry {
location: try!(loc.into_storeid()),
header: try!(Value::parse(header.as_str())),
content: String::from(content),
header: header,
content: content,
})
}
@ -1262,6 +1241,96 @@ mod store_tests {
}
}
#[test]
fn test_store_create_with_io_backend() {
use std::io::Cursor;
use std::rc::Rc;
use std::cell::RefCell;
use serde_json::Value;
//let sink = vec![];
//let output : Cursor<&mut [u8]> = Cursor::new(&mut sink);
//let output = Rc::new(RefCell::new(output));
let output = Rc::new(RefCell::new(vec![]));
{
let store = {
use file_abstraction::stdio::StdIoFileAbstraction;
use file_abstraction::stdio::mapper::json::JsonMapper;
// Lets have an empty store as input
let mut input = Cursor::new(r#"
{ "version": "0.3.0",
"store": { }
}
"#);
let mapper = JsonMapper::new();
let backend = StdIoFileAbstraction::new(&mut input, output.clone(), mapper).unwrap();
let backend = Box::new(backend);
Store::new_with_backend(PathBuf::from("/"), None, backend).unwrap()
};
for n in 1..100 {
let s = format!("test-{}", n);
let entry = store.create(PathBuf::from(s.clone())).unwrap();
assert!(entry.verify().is_ok());
let loc = entry.get_location().clone().into_pathbuf().unwrap();
assert!(loc.starts_with("/"));
assert!(loc.ends_with(s));
}
}
let vec = Rc::try_unwrap(output).unwrap().into_inner();
let errstr = format!("Not UTF8: '{:?}'", vec);
let string = String::from_utf8(vec);
assert!(string.is_ok(), errstr);
let string = string.unwrap();
assert!(!string.is_empty(), format!("Expected not to be empty: '{}'", string));
let json : ::serde_json::Value = ::serde_json::from_str(&string).unwrap();
match json {
Value::Object(ref map) => {
assert!(map.get("version").is_some(), format!("No 'version' in JSON"));
match map.get("version").unwrap() {
&Value::String(ref s) => assert_eq!("0.3.0", s),
_ => panic!("Wrong type in JSON at 'version'"),
}
assert!(map.get("store").is_some(), format!("No 'store' in JSON"));
match map.get("store").unwrap() {
&Value::Object(ref objs) => {
for n in 1..100 {
let s = format!("/test-{}", n);
assert!(objs.get(&s).is_some(), format!("No entry: '{}'", s));
match objs.get(&s).unwrap() {
&Value::Object(ref entry) => {
match entry.get("header").unwrap() {
&Value::Object(_) => assert!(true),
_ => panic!("Wrong type in JSON at 'store.'{}'.header'", s),
}
match entry.get("content").unwrap() {
&Value::String(_) => assert!(true),
_ => panic!("Wrong type in JSON at 'store.'{}'.content'", s),
}
},
_ => panic!("Wrong type in JSON at 'store.'{}''", s),
}
}
},
_ => panic!("Wrong type in JSON at 'store'"),
}
},
_ => panic!("Wrong type in JSON at top level"),
}
}
#[test]
fn test_store_get_create_get_delete_get() {
let store = get_store();

35
libimagstore/src/util.rs
View file

@ -17,6 +17,15 @@
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
//
use regex::Regex;
use toml::Value;
use libimagerror::into::IntoError;
use store::Result;
use error::StoreErrorKind as SEK;
use toml_ext::Header;
#[cfg(feature = "early-panic")]
#[macro_export]
macro_rules! if_cfg_panic {
@ -33,3 +42,29 @@ macro_rules! if_cfg_panic {
($fmt:expr, $($arg:tt)+) => { };
}
pub fn entry_buffer_to_header_content(buf: &str) -> Result<(Value, String)> {
debug!("Building entry from string");
lazy_static! {
static ref RE: Regex = Regex::new(r"(?smx)
^---$
(?P<header>.*) # Header
^---$\n
(?P<content>.*) # Content
").unwrap();
}
let matches = match RE.captures(buf) {
None => return Err(SEK::MalformedEntry.into_error()),
Some(s) => s,
};
let header = match matches.name("header") {
None => return Err(SEK::MalformedEntry.into_error()),
Some(s) => s
};
let content = matches.name("content").map(|r| r.as_str()).unwrap_or("");
Ok((try!(Value::parse(header.as_str())), String::from(content)))
}