Revisit the documentation for the store

This commit is contained in:
Matthias Beyer 2017-10-14 21:52:16 +02:00
parent 6068f2b52d
commit 16ebf52385

View file

@ -4,13 +4,19 @@ The store is where all the good things happen.
The store is basically just a directory on the filesystem imag manages and keeps The store is basically just a directory on the filesystem imag manages and keeps
its state in. its state in.
One could say that the store is simply a databases, and it really is. We opted One could say that the store is simply a database, and it really is. We opted
to go for plain text, though, as we believe that plain text is the only sane way to go for plain text, though, as we believe that plain text is the only sane way
to do such a thing. to do such a thing, especially because the amount of data which is to be
expected in this domain is in the lower Megabytes range and even if it is
_really_ much won't exceed the Gigabytes ever.
Having a storage format which is plain-text based is the superior approach, as
text editors will always be there.
A user should always be able to read her data without great effort and putting A user should always be able to read her data without great effort and putting
everything in a _real_ database like sqlite or even postgresql would need a user everything in a _real_ database like sqlite or even postgresql would need a user
to install additional software just to read his own data. We don't want that. to install additional software just to read his own data. We don't want that.
Text is readable until the worlds end and we think it is therefor better to Text is readable until the worlds end and we think it is therefore better to
store the data in plain text. store the data in plain text.
The following sections describe the store and the file format we use to store The following sections describe the store and the file format we use to store
@ -19,10 +25,10 @@ to dig into the store with their editors.
## File Format {#sec:thestore:fileformat} ## File Format {#sec:thestore:fileformat}
The contents of the store are encoded in either UTF-8 or ASCII. Either way, a The contents of the store are encoded in UTF-8.
normal text editor (like `vim` or the other one) will always be sufficient to A normal text editor (like `vim` or the other one) will always be sufficient to
dog into the store and modify files. For simple viewing even a pager (like dig into the store and modify files.
`less`) is sufficient. For simple viewing even a pager (like `less`) is sufficient.
Each entry in the store consists of two parts: Each entry in the store consists of two parts:
@ -44,24 +50,24 @@ restricted in their way of altering the data.
So normally there are several sections in the header. One section (`[imag]`) is So normally there are several sections in the header. One section (`[imag]`) is
always present. It contains a `version` field, which tells imag which version always present. It contains a `version` field, which tells imag which version
this file was created with (the version information is _also_ encoded in the this file was created with.
filename, just in case things change in the future). It also contains a `links`
field which is an Array of values. This `links` field is for linking (see
@sec:thestore:linking) to other entries in the store.
Other sections are named like the modules which created them. Every module is Other sections are named like the modules which created them. Every module is
allowed to store arbitrary data under its own section and a module may never allowed to store arbitrary data under its own section and a module may never
read other sections than its own. This is not enforced by imag itself, though. read other sections than its own.
These conventions are not enforced by imag itself, though.
### Content Format {#sec:thestore:fileformat:content} ### Content Format {#sec:thestore:fileformat:content}
The content is the part of the file where the user is free to enter any textual The content is the part of the file where the user is free to enter any textual
content. The content may be rendered as Markdown or other markup format for the content. The content may be rendered as Markdown or other markup format for the
users convenience. The store does never expect and specific markup and actually users convenience. The store does never expect and specific markup and actually
the markup implementation is not inside the very code of imag. the markup implementation is not inside the very core of imag.
Technically it would be possible that the content part of a file is used to Technically it would be possible that the content part of a file is used to
store binary data. We don't want this, though. store binary data.
We don't want this, though, as it is contrary to the goals of imag.
### Example {#sec:thestore:fileformat:example} ### Example {#sec:thestore:fileformat:example}
@ -101,7 +107,7 @@ So if the store exists in `/home/user/store/`, a file with the storepath
`/home/user/store/example.file`. `/home/user/store/example.file`.
By convention, each `libimagentry<name>` and `libimag<name>` module stores its By convention, each `libimagentry<name>` and `libimag<name>` module stores its
entries in in `/<name/`. entries in in `/<name>/`.
So, the pattern for the storepath is So, the pattern for the storepath is
@ -110,8 +116,8 @@ So, the pattern for the storepath is
``` ```
Any number of subdirectories may be used, so creating folder hierarchies is Any number of subdirectories may be used, so creating folder hierarchies is
possible and valid. A file "example" for a module "module" possible and valid.
would be stored in sub-folders like this: A file "example" for a module "module" could be stored in sub-folders like this:
``` ```
/module/some/sub/folder/example /module/some/sub/folder/example
@ -127,21 +133,21 @@ Note: This is a very core thing. Casual users might want to skip this section.
### Problem {#sec:thestore:backends:problem} ### Problem {#sec:thestore:backends:problem}
First, we had a compiletime backend for the store. First, we had a compiletime backend for the store.
This means that the actual filesystem operations were compiled into the stores This means that the actual filesystem operations were compiled into the store
either as real filesystem operations (in a normal debug or release build) but as either as real filesystem operations (in a normal debug or release build) but as
a in-memory variant in the 'test' case. a in-memory variant in the 'test' case.
So tests did not hit the filesystem when running. So tests did not hit the filesystem when running.
This gave us us the possibility to run tests concurrently with multiple stores This gave us us the possibility to run tests concurrently with multiple stores
that did not interfere with eachother. that did not interfere with each other.
This approach worked perfectly well until we started to test not the This approach worked perfectly well until we started to test not the
store itself but crates that depend on the store implementation. store itself but crates that depend on the store implementation.
When running tests in a crate that depends on the store, the store When running tests in a crate that depends on the store, the store
itself was compiled with the filesystem-hitting-backend. itself was compiled with the filesystem-hitting-backend.
This was problematic, as tests could not be implemented without hitting This was problematic, as tests could not be implemented without hitting
the filesystem. the filesystem and mess up other currently-running tests.
Hence we implemented this. Hence we implemented store backends.
### Implementation {#sec:thestore:backends:implementation} ### Implementation {#sec:thestore:backends:implementation}
@ -182,7 +188,7 @@ backend in other ways, too.
This is a backend for the imag store which is created This is a backend for the imag store which is created
from stdin, by piping contents into the store (via JSON or TOML) and piping the from stdin, by piping contents into the store (via JSON or TOML) and piping the
store contents (as JSON or TOML) to stdout when the the backend is destructed. store contents (as JSON or TOML) to stdout when the backend is destructed.
This is one of some components which make command-chaining in imag possible. This is one of some components which make command-chaining in imag possible.
With this, the application does not have to know whether the store actually With this, the application does not have to know whether the store actually
@ -196,12 +202,8 @@ implementation are possible.
The following section assumes a JSON mapper. The following section assumes a JSON mapper.
The mapper reads the JSON, parses it (thanks serde!) and translates it to The mapper reads the JSON, parses it and translates it to a `Entry`.
a `Entry`, which is the in-memory representation of the files. Then, the entry is made available to the store codebase.
The `Entry` contains a `Header` part and a `Content` part.
This is then made available to the store codebase.
To summarize what we do right now, lets have a look at the awesome ascii-art To summarize what we do right now, lets have a look at the awesome ascii-art
below: below:
@ -213,8 +215,6 @@ below:
+--+---------+----------------+--------+--+ +--+---------+----------------+--------+--+
| | | | | | | | | | | |
JSON -> Entry -> JSON JSON -> Entry -> JSON
+ Header
+ Content
``` ```
This is what gets translated where for one imag call with a stdio store backend. This is what gets translated where for one imag call with a stdio store backend.