matthiasbeyer/imag
1
0
Fork 0
Code Releases Activity

Move the documentation to the docs

Browse source
This commit is contained in:
Matthias Beyer 2017-06-17 21:07:30 +02:00
parent 76dfe23f65
commit c4b2287876
3 changed files with 167 additions and 75 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

167
doc/src/02000-store.md
View file

@ -146,3 +146,170 @@ moment. If the module "B" gets updated, it might update its entries in the store
as well. The link from the "a" should never get invalid in this case, though it as well. The link from the "a" should never get invalid in this case, though it
is not ensured by the core of imag itself. is not ensured by the core of imag itself.
## Backends {#sec:thestore:backends}
The store itself also has a backend. This backend is the "filesystem
abstraction" code.
Note: This is a very core thing. Casual users might want to skip this section.
### Problem {#sec:thestore:backends:problem}
First, we had a compiletime backend for the store.
This means that the actual filesystem operations were compiled into the stores
either as real filesystem operations (in a normal debug or release build) but as
a in-memory variant in the 'test' case.
So tests did not hit the filesystem when running.
This gave us us the possibility to run tests concurrently with multiple stores
that did not interfere with eachother.
This approach worked perfectly well until we started to test not the
store itself but crates that depend on the store implementation.
When running tests in a crate that depends on the store, the store
itself was compiled with the filesystem-hitting-backend.
This was problematic, as tests could not be implemented without hitting
the filesystem.
Hence we implemented this.
### Implementation {#sec:thestore:backends:implementation}
The filesystem is abstracted via a trait `FileAbstraction` which
contains the essential functions for working with the filesystem.
Two implementations are provided in the code:
* FSFileAbstraction
* InMemoryFileAbstraction
whereas the first actually works with the filesystem and the latter
works with an in-memory HashMap that is used as filesystem.
Further, the trait `FileAbstractionInstance` was introduced for
functions which are executed on actual instances of content from the
filesystem, which was previousely tied into the general abstraction
mechanism.
So, the `FileAbstraction` trait is for working with the filesystem, the
`FileAbstractionInstance` trait is for working with instances of content
from the filesystem (speak: actual Files).
In case of the `FSFileAbstractionInstance`, which is the implementation
of the `FileAbstractionInstance` for the actual filesystem-hitting code,
the underlying resource is managed like with the old code before.
The `InMemoryFileAbstractionInstance` implementation is corrosponding to
the `InMemoryFileAbstraction` implementation - for the in-memory
"filesystem".
The implementation of the `get_file_content()` function had to be
changed to return a `String` rather than a `&mut Read` because of
lifetime issues.
This change is store-internally and the API of the store itself was not
affected.
## The StdIo backend {#sec:thestore:backends:stdio}
Sidenote: The name is "StdIo" because its main purpose is Stdin/Stdio, but it
is abstracted over Read/Write actually, so it is also possible to use this
backend in other ways, too.
### Why? {#sec:thestore:backends:stdio:why}
This is a backend for the imag store which is created
from stdin, by piping contents into the store (via JSON or TOML) and piping the
store contents (as JSON or TOML) to stdout when the the backend is destructed.
This is one of some components which make command-chaining in imag possible.
With this, the application does not have to know whether the store actually
lives on the filesystem or just "in memory".
### Mappers {#sec:thestore:backends:stdio:mappers}
The backend contains a "Mapper" which defines how the contents get mapped into
the in-memory store representation: A JSON implementation or a TOML
implementation are possible.
The following section assumes a JSON mapper.
The backends themselves do not know "header" and "content" - they know only
blobs which live in pathes.
Indeed, this "backend" code does not serve "content" or "header" to the `Store`
implementation, but only a blob of bytes.
Anyways, the JSON-protocol for passing a store around _does_ know about content
and header (see @sec:thestore:backends:stdio:json for the JSON format).
So the mapper reads the JSON, parses it (thanks serde!) and translates it to
TOML, because TOML is the Store representation of a header.
But because the backend does not serve header and content, but only a blob,
this TOML is then translated (with the content of the respective file) to a
blob.
This is then made available to the store codebase.
This is complex and probably slow, we know.
To summarize what we do right now, lets have a look at the awesome ascii-art
below:
```
libimag*
|
v
IO Mapper FS Store FS Mapper IO
+--+-------------+---------+--------+---------+--------------+--+
| | | | | | | |
JSON -> TOML -> String -> Entry -> String -> TOML -> JSON
+ TOML
+ Content
```
This is what gets translated where for one imag call with a stdio store backend.
The rationale behind this implementation is that this is the best implementation
we can have in a relatively short amount of time.
### The JSON Mapper {#sec:thestore:backends:stdio:json}
The JSON mapper maps JSON which is read from a source into a HashMap which
represents the in-memory filesystem.
The strucure is as follows:
```json
{
"version": "0.3.0",
"store": {
"/example": {
"header": {
"imag": {
"version": "0.3.0",
},
},
"content": "hi there!",
},
},
}
```
### TODO {#sec:thestore:backends:todo}
Of course, the above is not optimal.
The TODO here is almost visible: Implement a proper backend where we do not need
to translate between types all the time.
The first goal would be to reduce the above figure to:
```
libimag*
|
v
IO Mapper Store Mapper IO
+--+------+--------+-------+--+
| | | | | |
JSON -> Entry -> JSON
+ TOML
+ Content
```
and the second step would be to abstract all the things away so the `libimag*`
crates handle the header without knowing whether it is JSON or TOML.

56
libimagstore/src/file_abstraction/mod.rs
View file

@ -17,62 +17,6 @@
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA // Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
// //
//! The filesystem abstraction code
//!
//! # Problem
//!
//! First, we had a compiletime backend for the store. This means that the actual filesystem
//! operations were compiled into the store either as real filesystem operations (in a normal debug
//! or release build) but as a in-memory variant in the 'test' case.
//! So tests did not hit the filesystem when running.
//! This gave us us the possibility to run tests concurrently with multiple
//! stores that did not interfere with eachother.
//!
//! This approach worked perfectly well until we started to test not the
//! store itself but crates that depend on the store implementation.
//! When running tests in a crate that depends on the store, the store
//! itself was compiled with the filesystem-hitting-backend.
//! This was problematic, as tests could not be implemented without hitting
//! the filesystem.
//!
//! Hence we implemented this.
//!
//! # Implementation
//!
//! The filesystem is abstracted via a trait `FileAbstraction` which
//! contains the essential functions for working with the filesystem.
//!
//! Two implementations are provided in the code:
//!
//! * FSFileAbstraction
//! * InMemoryFileAbstraction
//!
//! whereas the first actually works with the filesystem and the latter
//! works with an in-memory HashMap that is used as filesystem.
//!
//! Further, the trait `FileAbstractionInstance` was introduced for
//! functions which are executed on actual instances of content from the
//! filesystem, which was previousely tied into the general abstraction
//! mechanism.
//!
//! So, the `FileAbstraction` trait is for working with the filesystem, the
//! `FileAbstractionInstance` trait is for working with instances of content
//! from the filesystem (speak: actual Files).
//!
//! In case of the `FSFileAbstractionInstance`, which is the implementation
//! of the `FileAbstractionInstance` for the actual filesystem-hitting code,
//! the underlying resource is managed like with the old code before.
//! The `InMemoryFileAbstractionInstance` implementation is corrosponding to
//! the `InMemoryFileAbstraction` implementation - for the in-memory
//! "filesystem".
//!
//! The implementation of the `get_file_content()` function had to be
//! changed to return a `String` rather than a `&mut Read` because of
//! lifetime issues.
//! This change is store-internally and the API of the store itself was not
//! affected.
//!
use std::path::PathBuf; use std::path::PathBuf;
use std::fmt::Debug; use std::fmt::Debug;

19
libimagstore/src/file_abstraction/stdio/mod.rs
View file

@ -17,25 +17,6 @@
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA // Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
// //
//! The StdIo backend
//!
//! Sidenote: The name is "StdIo" because its main purpose is Stdin/Stdio, but it is abstracted over
//! Read/Write actually, so it is also possible to use this backend in other ways, too.
//!
//! So what is this about? This is a backend for the imag store which is created from stdin, by
//! piping contents into the store (via JSON or TOML) and piping the store contents (as JSON or
//! TOML) to stdout when the the backend is destructed.
//!
//! This is one of some components which make command-chaining in imag possible. With this, the
//! application does not have to know whether the store actually lives on the filesystem or just "in
//! memory".
//!
//! The backend contains a "Mapper" which defines how the contents get mapped into the in-memory
//! store representation: A JSON implementation or a TOML implementation are possible.
//!
//! In fact, a JSON implementation exists in the "json" submodule of this module.
//!
use std::cell::RefCell; use std::cell::RefCell;
use std::collections::HashMap; use std::collections::HashMap;
use std::fmt::Debug; use std::fmt::Debug;