Skip to content

Persistent Data

Direktiv supports storing and retrieving data that is persisted beyond the scope of a single state or flow instance. This article shows how to store and retrieve these variables.

Demo

direktiv_api: workflow/v1
states:
- id: a
  type: getter
  variables:
  - key: x
    scope: workflow
  transform: 'jq(.var.x += 1)'
  transition: b
- id: b
  type: setter
  variables:
  - key: x
    scope: workflow
    value: 'jq(.var.x)'

This demo increments a counter each time the flow is executed. It gets the variable x from workflow scope and increments it vi jq. The secons state stores the data in the same variable on the same scope.

Scopes

There are three scopes for storing persistent data: instance, workflow, and namespace.

Data stored in the instance scope only exists for the duration of the running flow instance.

Data stored in the workflow scope exists until the flow definition is deleted, and is accessible to all instances of that flow.

Data stored in the namespace scope exists until the namespace itself is deleted, and is accessible to all instances of all flows originating on that namespace.

Setter State

The Setter State can be used to store any number of variables. Each variable must be explicitly scoped, and the value stored for a variable is generated by the output of a jq query.

direktiv_api: workflow/v1
states:
- id: a
  type: setter
  variables:
  - key: MyVar
    scope: namespace
    value: 'Hello'

The only way to delete a stored value is to set it to null.

direktiv_api: workflow/v1
states:
- id: a
  type: setter
  variables:
  - key: MyVar
    scope: namespace
    value:

Getter State

The Getter State is used to retrieve any number of variables in persistent storage. Each variable must be explicitly scoped, and the value retrieved will be stored under .var.KEY where KEY is the variable's name.

- id: a
  type: getter
  variables:
  - key: x
    scope: namespace

A key doesn't need to exist in storage to return successfully, but the value returned will be null if it doesn't exist.

Concurrency

Direktiv makes no effort to guarantee any thread-safety on persistent data. Multiple instances that interact with the same variable may have inconsistent results.

Getting & Setting from Functions

Getting

Accessing persistent data from within a function is a fairly straightforward process. The request that the custom function receives from Direktiv contains a header 'Direktiv-TempDir', which contains all of the variables specified in the function definition. The as, key, scope, and type fields can all play a role in the placement and naming of files within this directory:

  • key
  • The key used to select a variable from within the flow definition. If no as field is provided, the file on a custom function will correspond to the value of key.
  • scope
  • Which scope to get the variable from: instance, workflow, or namespace. Defaults to instance if omitted.
  • as
  • An optional field used to set the name of the file as it appears on the isolate.
  • type
  • plain
    • The variable data inside of the file will be written 'as-is'.
  • base64
    • If the variable is stored as base64-encoded data, it will be decoded before being written to the file system.
  • tar
    • If the variable is a valid tar archive, a directory will be created instead of a file, with the contents of the tar archive populating it.
  • tar.gz
    • Similar to tar, this will result in a populated directory being created from a valid .tar.gz file.

For example, given the following state definition, a directory named 'myFiles' should exist within the directory specified by the Direktiv-TempDir header. Assuming that this header has a value of /mnt/shared/example, the following structure would be expected:

  - id: get
    image: localhost:5000/iv-getter:v1
    files:
    - key: "myFiles"
      scope: instance
      type: tar

/mnt/shared/example/
└── myFiles
    └── file-1
    └── file-2
    └── file-3

Setting

From within a function running on Direktiv, variables can be set by sending a POST request:

POST http://localhost:8889/var?aid=<EXAMPLE>&scope=instance&key=myFiles

Body: <VARIABLE DATA>
  • query parameters
  • aid
    • The action ID, found from the Direktiv-ActionID header of the request being served by the isolate.
  • scope
    • The scope for which the variable is set (namespace, workflow, or instance)
  • key
    • The key used by subsequent actions to access the variable.

An alternative approach is to write files into certain directories. The direktiv sidecar will store those files as variables. There are three different folders for the three different scopes. For the above example they would be:

/mnt/shared/example/out/instance
/mnt/shared/example/out/workflow
/mnt/shared/example/out/namespace

Files under these folders will be stored with their names under the scope of the folder. Diretories will be stored as tar.gz files.