Linkspace

This introduction uses the `lk` cli with bash. You can follow along by downloading the latest release. If you're more comfortable with Python or want a more detailed explanation check out the guide.

lk --version
linkspace-cli linkspace-cli - 0.5.1 - main - c770b73 - 1.75.0-nightly

Point

A single unit|event|message in linkspace is called a `point`. A point can hold just shy of 64kb.

echo "Hello, Sol!" | lk point > mylog

Points have a few of optional properties besides the data field.

a spacename:

echo -n some data | lk point ::/my/example/spacename --data-stdin >> mylog

a timestamp in microseconds since epoch using `now` by default:

# 12 seconds in the past - we'll get back to the [..] syntax 
lk point ::/my/example/other_space --create [now:-12s] >> mylog

and a signature:

# We don't have a linkspace instance yet - we can still sign packets by creating/providing an Argon2 encrypted key
KEY=$(lk key --no-pubkey --no-lk --password 'my secret') 
lk point ::/my/example/spacename/subspace --sign --enckey "$KEY" --password 'my secret' >> mylog

All points are hashed using Blake3.

Point's refer to the hashed fields/data. The point, hash, and a header are packed into the packet format. Functions/APIs deal exclusively in the packet format.

`pktf` formats a stream of packets.

cat mylog | lk pktf "Hash: [hash:str] is a point at '[spacename:str]' with data '[data]'"
Hash: Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk is a point at '' with data 'Hello, Sol!
'
Hash: Rvv4O1jSAWlLr5GaDao-YaJvCC_jJdJ3WvFtmR-6lH8 is a point at '/my/example/spacename' with data 'some data'
Hash: MlCys7eu_ozmoZalfqOWDiCvIn8b8mG8qX_sXOhXvtI is a point at '/my/example/other_space' with data ''
Hash: yHgbC6zQvVFp5w_buiwpTisbRnH1WMiPKwSNAM5SlQM is a point at '/my/example/spacename/subspace' with data ''

Or output only data: `cat mylog | lk pktf "[data]"`.

The other fields are readable by general purpose tools to index, address, and filter the packets. For example, lk filter. Here we only take those with a spacename starting with `/my/example` and 1 additional component

cat mylog | lk filter ::/my/example:* | lk pktf "[hash:str] [spacename:str]"
Rvv4O1jSAWlLr5GaDao-YaJvCC_jJdJ3WvFtmR-6lH8 /my/example/spacename
MlCys7eu_ozmoZalfqOWDiCvIn8b8mG8qX_sXOhXvtI /my/example/other_space

Points are addressable by their hash. To reference one point from another you add a link. Each link is a 16 byte tag and 32 byte hash value. Tags can be anything. If less than 16 bytes are supplied it is left-padded with 0's.

You can get fancy with `pktf` and `xargs`.

When building a log you should start with only using the data field, in a readable format like JSON.
Only when required should you expand to use the more advanced features like spacenames, signatures, and links.

You can ask if cryptographic properties, like signatures or distributed event ordering by linking, are worth to add upfront if you want to focus on the data.
The answer is yes.
Cryptographic signatures and hash addresses are difficult to get right, and infeasible to add retroactively.

A tool like `lk collect` has a few additional options for building points linking to other points. However, the `lk` binary is meant to do simple things. Use the library and a better programming language when doing non-trivial stuff.

Click here to see a graphical representation of mylog

Creating packets with `lk point` detects what kind you're trying to build. But it is better to be explicit. The 3 types of points are: `datapoint`, `linkpoint`, or `keypoint` (a signed linkpoint).

NOTE: linkpoint and keypoint do not read data from stdin by default.

echo somedata | lk linkpoint ::/my/other_spacename --data-stdin | lk pktf "[hash:str] = [data]"
bg9ADn-zLYHCqYWKIKGN3ddbEoWdtNGgrKFTsO9L8Vk = somedata

Database

Linkspace is primarily its packet format. Using the database is optional.

The database and other tools exists to make it easier to build complex systems and packet flows.

export LK_DIR=/tmp/linkspace ; 
lk init ; 
cat mylog | lk save > /dev/null ; 
LkInfo { kind: "lmdb", dir: "/tmp/linkspace" }

Instead of using `save` you can set a write destinations directly.

echo hello world | lk point --write db --write file:mylog --write stdout | lk pktf [data]
hello world

The database has three indices. A 'log', 'hash', and the 'tree' index. The log-index is ordered by receive order, the hash-index by the point hash, and the tree-index primarily by a point's [spacename, create stamp] (see the guide for full details).

The database is always accessed through the runtime. The runtime lets multiple processes/threads can read, write, and watch for new points.

The library API uses callbacks and a user-driven eventloop (guide). `lk` is focused on piping packets. Commands are `watch-log`, `watch-tree`, `watch-hash`. These are shorthand for `watch –mode ..`.

lk watch-tree ::/my:** | lk pktf "[spacename:str]" > ./watching &
[1] 10412

cat ./watching
/my/example/link
/my/example/link
/my/example/spacename
/my/example/other_space
/my/example/spacename/subspace

Adding a new point

lk linkpoint ::/my/my/my --write db

Wakes the watching threads to output the new point.

cat ./watching
/my/example/link
/my/example/link
/my/example/spacename
/my/example/other_space
/my/example/spacename/subspace
/my/my/my

Applications

There are two optional fields included in the hash not yet shown. The domain and group.

The domain is analogous to a IP port. An application pick a domain name (max 16 bytes). For example `imageboard`.

`pktf` is common enough to have the alias `lk p`.
lk linkpoint imageboard:: | lk p "[domain:str]"
imageboard

Groups indicate the set of intended recipients. If a group exchange process is running, an application doesn't have to deal with sockets, (HTTP) endpoints, or other IO except for the user interface. The application can read, write, request from the group, and process packets using just the linkspace library.

Building an application is done by mapping an application state to and from linkspace packets (in the database). For example, a drawing application where multiple people can paint to a shared image board. A simple mapping could be:

Adding an image might look something like:

X=30 ; Y=200 ; IMG="https://upload.wikimedia.org/wikipedia/commons/3/35/Tux.svg" ; 
curl -s $IMG | lk datapoint > tux.pkt
IMG_HASH=$(cat tux.pkt | lk p "[hash:str]")
lk linkpoint imageboard:: -- $(printf "%08d%08d" "$X" "$Y"):$IMG_HASH >> tux.pkt
lk save --pkts ./tux.pkt # Instead of `cat` we can provide a file

Building an image requires the program to watch for new packets in `imageboard::`, and on every (new) point draw over the image.

lk watch-tree "imageboard::" --max 1 \
   | lk p "[hash:str] has the links:\n [links]" 
b-tKZMVvXtyC24Y4TPlsgb7yizRyb25Kb4sNKpu_5zo has the links:
 0000003000000200:Sz0ZZDWxKht-jbM7Tfkn0nis4tNoKNPH_kfI7JYUnY4
You should always quote your arguments.
Otherwise, the previous example would not have worked.
Without the quotes the characters `[lin` in "[links]" would be interpreted by the default bash shell.

A link might reference a point that is not (yet) available on the device. An application has to decide how to handle the situation. In this example we'll just wait. Waiting can be done manually. e.g.

got point Sz0ZZDWxKht-jbM7Tfkn0nis4tNoKNPH_kfI7JYUnY4 which has 49983 bytes

Or use `lk get-links`. It has a few common strategies.

Sz0ZZDWxKht-jbM7Tfkn0nis4tNoKNPH_kfI7JYUnY4
b-tKZMVvXtyC24Y4TPlsgb7yizRyb25Kb4sNKpu_5zo

To complete the imageboard application we'll have to add a few more steps to merge the data into a single picture. See the tutorial for an example on doing this and more.

The final piece of the puzzle is the group field. A group is 32 bytes to signal the intended set of recipients. It is orthogonal to the domain field, as the application should not care which group its running in.

PUB=$(echo "Hello, Sol!" | lk data | lk pktf "[hash:str]")
lk linkpoint :$PUB:/example | lk p "[group:str]"
Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk

If no group is specified (like we've been doing) the public group is used.

lk linkpoint :[#:pub]:/example  | lk p "[group:str]" 
Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk
See ABE for more on the `[..]` syntax.
Its a small byte templating language included in the library for convenience with the syntax being the same for all programming language.
ABE is also heavily used for CLI arguments, e.g. `lk linkpoint :: –stamp [now]` or `[now:+2h]`

The other special group is `[0;32]`, also called the private group. You can refer to it with the expression `[#:0]`. Functions/subcommands that read/write existing points skip and/or warn whenever a point from the private group is seen unless enabled with `–private`.

lk linkpoint domain:[#:0] | lk save 2>&1 # creating a packet is ok - but receving is not accepted by `lk save` without --private
error: Args { inner: ["/home/rs/Projects/linkspace/target/debug/lk", "save"] }
Pkt(
    PrivateGroup,
)

A system to exchange points in a group can be made from scratch. Linkspace does not prescribe a way to do so. Each group / network is different, and no single solution can cover every situation.

For example, use `lk watch imageboard:$MYGROUP | …` and forward the entire stream to another device using netcat/socat, ssh, email, http, a USB stick, or other way to exchange bytes.

Linkspace is designed to only ever be a streams of packets, without additional overhead of a (custom) serialization formats. As evident by the 'mylog' file we have used thus far. This keeps streams compatible with all tools that process streams.

To that end, each packet has a mutable header excluded from the hash.

Filters work on these mutable bytes as well. This let you quickly build specific network topologies.

See the guide for the mutable field names.
netcat 10.0.0.1 -p 6000 | lk route ubits0:=:0000 | lk save & # get packets from a host and set their ubits0 to 0000
netcat 10.0.2.0 -p 6000 | lk route ubits0:=:0001 | lk save & # get packets from another host and set their ubits0 to 0001
lk linkpoint example::/hello | lk route ubits0:=:0002 | lk save # save my packets with ubits 0002
lk watch-log --asc example::/hello -- "ubits0:>:0000" | nc 10.0.0.1 -p 6000  & # forward all packets with ubits0 higher than 0000 back to host.

A single linkspace instance can be used by multiple applications on device, and connect to others. To that end there are some conventions. These are functions that create/watch for point with some predefined spacename, links, and data format. Conventions enable interoperability between multiple applications and background processes.

One such convention is the `pull` convention. This writes a query as a specific point.

lk pull imageboard:: --write stdout | lk p "[spacename:str]\n\n[data]"
/pull/[b:Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk]/[a:imageboard]/default

:qid:default
type:1:[b2:00000010]
domain:=:[a:imageboard]
group:=:[b:Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk]
depth:=:[u8:0]

The goal of `pull` is to allow one process, e.g. an application like imageboard (bash) or mineweeper (python) to signal another process, e.g. a group exchange process like bash.exchange, that it wants packets matching a query from the group.

Queries define a 'set of points' in linkspace. The `filter` and `watch` commands are syntax sugar over queries. You can add `–print-query` to those commands to see the query used.

Queries are designed such that joining two query strings the result is the common subset of both or an error if the union is empty.

lk print-query example::/ok 
:mode:tree-desc

type:1:[b2:00000010]
domain:=:[a:example]
group:=:[b:Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk]
prefix:=:/ok
depth:=:[u8:1]

lk print-query example::/ok -- "spacename:=:/not_ok"
error: Args { inner: ["/home/rs/Projects/linkspace/target/debug/lk", "print-query", "example::/ok", "--", "spacename:=:/not_ok"] }
Error {
    context: "Error adding rule \'spacename\'",
    source: Error {
        context: "spacename:=:/not_ok",
        source: "space prefix conflict",
    },
}

That's it for this quick introduction. Some notes on high level algorithm design are worth a read. For a more in-depth technical guide or the library API see the Guide.