Revisit the documentation for the store
This commit is contained in:
parent
6068f2b52d
commit
16ebf52385
1 changed files with 31 additions and 31 deletions
|
@ -4,13 +4,19 @@ The store is where all the good things happen.
|
||||||
The store is basically just a directory on the filesystem imag manages and keeps
|
The store is basically just a directory on the filesystem imag manages and keeps
|
||||||
its state in.
|
its state in.
|
||||||
|
|
||||||
One could say that the store is simply a databases, and it really is. We opted
|
One could say that the store is simply a database, and it really is. We opted
|
||||||
to go for plain text, though, as we believe that plain text is the only sane way
|
to go for plain text, though, as we believe that plain text is the only sane way
|
||||||
to do such a thing.
|
to do such a thing, especially because the amount of data which is to be
|
||||||
|
expected in this domain is in the lower Megabytes range and even if it is
|
||||||
|
_really_ much won't exceed the Gigabytes ever.
|
||||||
|
|
||||||
|
Having a storage format which is plain-text based is the superior approach, as
|
||||||
|
text editors will always be there.
|
||||||
|
|
||||||
A user should always be able to read her data without great effort and putting
|
A user should always be able to read her data without great effort and putting
|
||||||
everything in a _real_ database like sqlite or even postgresql would need a user
|
everything in a _real_ database like sqlite or even postgresql would need a user
|
||||||
to install additional software just to read his own data. We don't want that.
|
to install additional software just to read his own data. We don't want that.
|
||||||
Text is readable until the worlds end and we think it is therefor better to
|
Text is readable until the worlds end and we think it is therefore better to
|
||||||
store the data in plain text.
|
store the data in plain text.
|
||||||
|
|
||||||
The following sections describe the store and the file format we use to store
|
The following sections describe the store and the file format we use to store
|
||||||
|
@ -19,10 +25,10 @@ to dig into the store with their editors.
|
||||||
|
|
||||||
## File Format {#sec:thestore:fileformat}
|
## File Format {#sec:thestore:fileformat}
|
||||||
|
|
||||||
The contents of the store are encoded in either UTF-8 or ASCII. Either way, a
|
The contents of the store are encoded in UTF-8.
|
||||||
normal text editor (like `vim` or the other one) will always be sufficient to
|
A normal text editor (like `vim` or the other one) will always be sufficient to
|
||||||
dog into the store and modify files. For simple viewing even a pager (like
|
dig into the store and modify files.
|
||||||
`less`) is sufficient.
|
For simple viewing even a pager (like `less`) is sufficient.
|
||||||
|
|
||||||
Each entry in the store consists of two parts:
|
Each entry in the store consists of two parts:
|
||||||
|
|
||||||
|
@ -44,24 +50,24 @@ restricted in their way of altering the data.
|
||||||
|
|
||||||
So normally there are several sections in the header. One section (`[imag]`) is
|
So normally there are several sections in the header. One section (`[imag]`) is
|
||||||
always present. It contains a `version` field, which tells imag which version
|
always present. It contains a `version` field, which tells imag which version
|
||||||
this file was created with (the version information is _also_ encoded in the
|
this file was created with.
|
||||||
filename, just in case things change in the future). It also contains a `links`
|
|
||||||
field which is an Array of values. This `links` field is for linking (see
|
|
||||||
@sec:thestore:linking) to other entries in the store.
|
|
||||||
|
|
||||||
Other sections are named like the modules which created them. Every module is
|
Other sections are named like the modules which created them. Every module is
|
||||||
allowed to store arbitrary data under its own section and a module may never
|
allowed to store arbitrary data under its own section and a module may never
|
||||||
read other sections than its own. This is not enforced by imag itself, though.
|
read other sections than its own.
|
||||||
|
|
||||||
|
These conventions are not enforced by imag itself, though.
|
||||||
|
|
||||||
### Content Format {#sec:thestore:fileformat:content}
|
### Content Format {#sec:thestore:fileformat:content}
|
||||||
|
|
||||||
The content is the part of the file where the user is free to enter any textual
|
The content is the part of the file where the user is free to enter any textual
|
||||||
content. The content may be rendered as Markdown or other markup format for the
|
content. The content may be rendered as Markdown or other markup format for the
|
||||||
users convenience. The store does never expect and specific markup and actually
|
users convenience. The store does never expect and specific markup and actually
|
||||||
the markup implementation is not inside the very code of imag.
|
the markup implementation is not inside the very core of imag.
|
||||||
|
|
||||||
Technically it would be possible that the content part of a file is used to
|
Technically it would be possible that the content part of a file is used to
|
||||||
store binary data. We don't want this, though.
|
store binary data.
|
||||||
|
We don't want this, though, as it is contrary to the goals of imag.
|
||||||
|
|
||||||
### Example {#sec:thestore:fileformat:example}
|
### Example {#sec:thestore:fileformat:example}
|
||||||
|
|
||||||
|
@ -101,7 +107,7 @@ So if the store exists in `/home/user/store/`, a file with the storepath
|
||||||
`/home/user/store/example.file`.
|
`/home/user/store/example.file`.
|
||||||
|
|
||||||
By convention, each `libimagentry<name>` and `libimag<name>` module stores its
|
By convention, each `libimagentry<name>` and `libimag<name>` module stores its
|
||||||
entries in in `/<name/`.
|
entries in in `/<name>/`.
|
||||||
|
|
||||||
So, the pattern for the storepath is
|
So, the pattern for the storepath is
|
||||||
|
|
||||||
|
@ -110,8 +116,8 @@ So, the pattern for the storepath is
|
||||||
```
|
```
|
||||||
|
|
||||||
Any number of subdirectories may be used, so creating folder hierarchies is
|
Any number of subdirectories may be used, so creating folder hierarchies is
|
||||||
possible and valid. A file "example" for a module "module"
|
possible and valid.
|
||||||
would be stored in sub-folders like this:
|
A file "example" for a module "module" could be stored in sub-folders like this:
|
||||||
|
|
||||||
```
|
```
|
||||||
/module/some/sub/folder/example
|
/module/some/sub/folder/example
|
||||||
|
@ -127,21 +133,21 @@ Note: This is a very core thing. Casual users might want to skip this section.
|
||||||
### Problem {#sec:thestore:backends:problem}
|
### Problem {#sec:thestore:backends:problem}
|
||||||
|
|
||||||
First, we had a compiletime backend for the store.
|
First, we had a compiletime backend for the store.
|
||||||
This means that the actual filesystem operations were compiled into the stores
|
This means that the actual filesystem operations were compiled into the store
|
||||||
either as real filesystem operations (in a normal debug or release build) but as
|
either as real filesystem operations (in a normal debug or release build) but as
|
||||||
a in-memory variant in the 'test' case.
|
a in-memory variant in the 'test' case.
|
||||||
So tests did not hit the filesystem when running.
|
So tests did not hit the filesystem when running.
|
||||||
This gave us us the possibility to run tests concurrently with multiple stores
|
This gave us us the possibility to run tests concurrently with multiple stores
|
||||||
that did not interfere with eachother.
|
that did not interfere with each other.
|
||||||
|
|
||||||
This approach worked perfectly well until we started to test not the
|
This approach worked perfectly well until we started to test not the
|
||||||
store itself but crates that depend on the store implementation.
|
store itself but crates that depend on the store implementation.
|
||||||
When running tests in a crate that depends on the store, the store
|
When running tests in a crate that depends on the store, the store
|
||||||
itself was compiled with the filesystem-hitting-backend.
|
itself was compiled with the filesystem-hitting-backend.
|
||||||
This was problematic, as tests could not be implemented without hitting
|
This was problematic, as tests could not be implemented without hitting
|
||||||
the filesystem.
|
the filesystem and mess up other currently-running tests.
|
||||||
|
|
||||||
Hence we implemented this.
|
Hence we implemented store backends.
|
||||||
|
|
||||||
### Implementation {#sec:thestore:backends:implementation}
|
### Implementation {#sec:thestore:backends:implementation}
|
||||||
|
|
||||||
|
@ -182,7 +188,7 @@ backend in other ways, too.
|
||||||
|
|
||||||
This is a backend for the imag store which is created
|
This is a backend for the imag store which is created
|
||||||
from stdin, by piping contents into the store (via JSON or TOML) and piping the
|
from stdin, by piping contents into the store (via JSON or TOML) and piping the
|
||||||
store contents (as JSON or TOML) to stdout when the the backend is destructed.
|
store contents (as JSON or TOML) to stdout when the backend is destructed.
|
||||||
|
|
||||||
This is one of some components which make command-chaining in imag possible.
|
This is one of some components which make command-chaining in imag possible.
|
||||||
With this, the application does not have to know whether the store actually
|
With this, the application does not have to know whether the store actually
|
||||||
|
@ -196,12 +202,8 @@ implementation are possible.
|
||||||
|
|
||||||
The following section assumes a JSON mapper.
|
The following section assumes a JSON mapper.
|
||||||
|
|
||||||
The mapper reads the JSON, parses it (thanks serde!) and translates it to
|
The mapper reads the JSON, parses it and translates it to a `Entry`.
|
||||||
a `Entry`, which is the in-memory representation of the files.
|
Then, the entry is made available to the store codebase.
|
||||||
The `Entry` contains a `Header` part and a `Content` part.
|
|
||||||
|
|
||||||
This is then made available to the store codebase.
|
|
||||||
|
|
||||||
To summarize what we do right now, lets have a look at the awesome ascii-art
|
To summarize what we do right now, lets have a look at the awesome ascii-art
|
||||||
below:
|
below:
|
||||||
|
|
||||||
|
@ -212,9 +214,7 @@ below:
|
||||||
IO Mapper Store Mapper IO
|
IO Mapper Store Mapper IO
|
||||||
+--+---------+----------------+--------+--+
|
+--+---------+----------------+--------+--+
|
||||||
| | | | | |
|
| | | | | |
|
||||||
JSON -> Entry -> JSON
|
JSON -> Entry -> JSON
|
||||||
+ Header
|
|
||||||
+ Content
|
|
||||||
```
|
```
|
||||||
|
|
||||||
This is what gets translated where for one imag call with a stdio store backend.
|
This is what gets translated where for one imag call with a stdio store backend.
|
||||||
|
|
Loading…
Reference in a new issue