ytree

Winter called for more children. This week's is named ytree! His heritage traces back through some earlier projects, and this week I felt a compulsion to give him new shape.

./foo type=dir
./foo/bar type=file content=foo/bar.yaml

Arch packaging uses MTREE as a programmable archiving format. I built on the concept by switching from the bespoke line based format to a YAML encoding with an archive entry per document. A complete archive is serialized from a stream of independent documents.

--- !ytree/tar/dir/v1
name: ./foo

--- !ytree/tar/file/v1
name: ./foo/bar
contents: !ytree/res/file/v1
  path: files/ytree/example.yaml

--- !ytree/tar/file/v1
name: ./hello/world
contents: !ytree/res/buffer/v1
  contents: !!binary aGVsbG8sIHdvcmxk

--- !ytree/tar/file/v1
name: ./hello/world
contents: !ytree/res/string/v1
  value: hello, world

The file resource type produces the most familiar behavior, populating archive contents from local files. I also added content providers that allow for specifying object data inline, rendered from a template, or fetched remotely.

def entries(version):
    return [
        ytree.tar.File(
            "version",
            ytree.res.String(version),
        ),
    ]

All of the mapped types have public python APIs so they can be generated programmatically. A function like this could encapsulate a version number on the fly. That could be transformed directly into a tarchive or serialized to YAML and served elsewhere.

# version.yaml
--- !ytree/tar/file/v1
name: version
contents: !ytree/res/string/v1
  value: decorating

I could build a sort of tree of archives with indirection. The first file could represent some set of metrics each named by a path and this second one a catalog with the particular "database" name given by the path for dispatching.

# status.yaml
--- !ytree/tar/file/v1
name: version.yaml
contents: !ytree/res/url/v1
  url: https://yieldsfalsehood.com/version.yaml

That evaluates to a tarchive with one entry populated by a snapshot of that URL. I can access a specific configuration item in there by deferring to tar to "extract" the database (version.yaml) first then the item stored as a file (version) inside it.

#!/bin/bash

set -euo pipefail

domain="${DOMAIN:-https://yieldsfalsehood.com}"
url="$domain"/status.yaml

curl -s "$url" \
    | python -m ytree.tar -c \
    | tar -Oxf - version.yaml \
    | python -m ytree.tar -c \
    | tar -Oxf - version

There's currently unoccupied space to configure SSL trust for those types of resources. So far I do have runtime configurable support for most jinja template loaders. Template resources themselves bind context data to a template name, which are resolved at runtime. There's no telling what this might look like when rendered!

--- !ytree/tar/file/v1
name: foo
contents: !ytree/res/template/v1
  template: template
  context:
    name: world

There is effectively no loader by default, which is fine if you never reference a template. If you do then you need glue, like this ad hoc loader that embeds a template body into the command line so you can see the secret message.

#!/bin/bash

set -euo pipefail

domain="${DOMAIN:-https://yieldsfalsehood.com}"

curl -s "${domain}"/ytree/template.yaml \
    | LOADER='--- !jinja2/loader/dict/v1 {mapping: {template: "hello, {{ name }}"}}' \
            python -m ytree.tar -c \
    | tar -Oxf - foo