Interface

Command line

usage: beaver [-h] [--cache CACHE] [--file FILE]
              [--log_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
              {build,list,reset} ...

Named Arguments

--cache, -c

file containing cached information, including composite digests

Default: β€œ.beavercache”

--file, -f

file containing artifact and transform definitions

Default: β€œbeaver.py”

--log_level, -l

Possible choices: DEBUG, INFO, WARNING, ERROR, CRITICAL

level of log messages to emit

Default: info

Sub-commands:

build

Build artifacts.

beaver build [-h] [--num_concurrent NUM_CONCURRENT] [--dry-run] [--all]
             [patterns ...]
Positional Arguments
patterns

patterns to match artifacts against

Named Arguments
--num_concurrent, -c

number of concurrent transforms

Default: 1

--dry-run, -n

print transforms without executing them

Default: False

--all, -a

match all artifacts

Default: False

list

List artifacts.

beaver list [-h] [--stale] [--raw] [--all] [patterns ...]
Positional Arguments
patterns

patterns to match artifacts against

Named Arguments
--stale, -s

list only stale artifacts

Default: False

--raw, -r

list names only without status indicators

Default: False

--all, -a

match all artifacts

Default: False

reset

Reset the composite digest of artifacts

beaver reset [-h] [--all] [patterns ...]
Positional Arguments
patterns

patterns to match artifacts against

Named Arguments
--all, -a

match all artifacts

Default: False

Transforms

class Download(output: beaver_build.artifacts.File, url: str)

Download a file.

Parameters

  • output – Output artifact for the downloaded data.

  • url – Url to download from.

Example

>>> data = bb.File("20news.tar.gz", expected_digest="af604312")
>>> bb.Download(data, url="https://ndownloader.figshare.com/files/5975967")
Download([] -> [File(20news.tar.gz)])
async apply() → None

Apply the transform.

class Functional(outputs: Iterable[beaver_build.artifacts.Artifact], inputs: Iterable[beaver_build.artifacts.Artifact], func: Callable, *args, **kwargs)

Apply a python function.

Parameters

  • outputs – Artifacts to generate.

  • inputs – Artifacts consumed by the transform.

  • func – Function to execute.

  • *args – Positional arguments passed to func.

  • *kwargs – Keyword arguments passed to func.

apply() → None

Apply the transform.

class Shell(outputs: Iterable[beaver_build.artifacts.Artifact], inputs: Iterable[beaver_build.artifacts.Artifact], cmd: str, *, env: Optional[dict[str, str]] = None, **kwargs)

Execute a command in the shell. See Subprocess for details.

Example: >>> Shell(β€œoutput.txt”, None, β€œecho hello > output.txt”) Shell([] -> [File(output.txt)])

class Subprocess(outputs: Iterable[beaver_build.artifacts.Artifact], inputs: Iterable[beaver_build.artifacts.Artifact], cmd: Union[str, Iterable[str]], *, env: Optional[dict[str, str]] = None, shell: bool = False, **kwargs)

Execute a subprocess.

The transform supports variable substitution using f-strings and Makefile syntax:

  • $@ represents the first output.

  • $< represents the first input.

  • $^ represents all inputs.

  • $$@ represents the literal $@, i.e. double dollars are properly escaped.

  • {outputs[0].name} represents the name of the first output. The available f-string variables are outputs and inputs, each a list of Artifacts.

  • $! represents the current python interpreter.

Environment variables are inherited by default, but global environment variables for all Subprocess transforms can be specified by set_global_env() or modifying get_global_env(), and specific environment variables can be specified using the env argument. Environment variables that are None are removed from the environment of the transform.

Parameters

  • outputs – Artifacts to generate.

  • inputs – Artifacts consumed by the transform.

  • cmd – Command to execute. Must be a single string if shell is truthy and a sequence of strings otherwise.

  • env – Mapping of environment variables.

  • shell – Whether to execute the command through the shell, e.g. to expand the home directory ~ or pipe information between processes or files using |, <, or >. See subprocess.Popen for details, including security considerations.

  • **kwargs – Keyword arguments passed to asyncio.subprocess.create_subprocess_shell() (if shell == True) or asyncio.subprocess.create_subprocess_exec() (if shell == False).

Raises

ValueError – If shell == True and cmd is not a string, or shell == False and cmd is not a list of strings.

ENV

Default mapping of environment variables for all Subprocess transforms.

Example

>>> bb.Subprocess("copy.txt", "input.txt", ["cp", "$<", "$@"])
Subprocess([File(input.txt)] -> [File(copy.txt)])
async apply() → None

Apply the transform.

classmethod get_global_env() → dict

Get the global environment variables used by all Subprocess transforms.

classmethod set_global_env(value: dict) → None

Set the global environment variables used by all Subprocess transforms.

class Transform(outputs: Iterable[beaver_build.artifacts.Artifact], inputs: Iterable[beaver_build.artifacts.Artifact])

Base class for transforms that generates outputs given inputs. Inheriting classes should implement execute().

Parameters

  • outputs – Artifacts to generate.

  • inputs – Artifacts consumed by the transform.

stale_outputs

Sequence of outputs that are stale and need to be updated.

async apply() → None

Apply the transform.

cancel_all_transforms() → None

Cancel all running transforms.

evaluate_composite_digests(outputs: Iterable[beaver_build.artifacts.Artifact], inputs: Iterable[beaver_build.artifacts.Artifact]) → dict['artifacts.Artifact', bytes]

Evaluate composite digests for all outputs.

A composite digest is defined as the digest of the digests of inputs and the digest of the output, i.e. it is a joint, concise summary of both inputs and outputs. A composite digest is None if the corresponding output digest is None or any input digests are None.

Parameters

  • inputs – Sequence of input artifacts.

  • outputs – Sequence of output artifacts.

Returns

digests – Mapping from outputs to composite digests.

Artifacts

class Artifact(name, *args, **kwargs)

Artifact generated by a transform.

Parameters

  • name – Unique name of the artifact.

  • expected_digest – Digest expected when the artifact is available.

  • ignore_groups – Ignore any groups and create a root-level artifact.

Raises

ValueError – If the given name is already in use by another artifact.

children

Transforms that consume this artifact; empty if the artifact is a leaf of the directed acyclic graph.

Type

Iterable[beaver_build.transforms.Transform]

parent

Transform that generates this artifact; None if the artifact is a root of the directed acyclic graph.

digest

Concise summary of the artifact; None if the artifact does not exist, cannot be summarized, or should always be generated using its parent transform.

class ArtifactFactory(name, bases, members)

Metaclass that creates Artifact instances or returns existing instances.

REGISTRY

Mapping of artifact names to Artifact instances. The registry is used to keep track of all artifacts and ensure artifact names are unique.

class File(name, *args, **kwargs)

Artifact representing a file.

Parameters

  • name – Unique name of the artifact.

  • expected_digest – Digest expected when the artifact is available.

  • ignore_groups – Ignore any groups and create a root-level artifact.

property exists: bool

Return whether the file exists.

classmethod glob(pattern: str) → Iterable[beaver_build.artifacts.File]

Create a list of file artifacts based on a glob pattern.

Parameters

pattern – Pattern passed to glob.glob().

Returns

artifacts – File artifacts.

read() → str

Read the file contents.

class Group(name, *args, **kwargs)

Artifact representing a group of other artifacts.

Parameters

  • name – Name of the group. The group name is added as a prefix for all artifacts created within the context manager of the group.

  • ignore_groups – Ignore any groups and create a root-level artifact.

members

Members of the group.

classmethod append(artifact: beaver_build.artifacts.Artifact) → None

Append an artifact to the current group if available.

Parameters

artifact – Artifact to append to the current group.

classmethod evaluate_qualified_name(name: str) → str

Evaluate the fully qualified name of an artifact given its context.

Parameters

name – Unqualified name.

Returns

name – Qualified name.

async gather_artifacts(*artifacts_or_transforms, num_concurrent: Optional[int] = None) → Coroutine

Gather one or more artifacts and wrap them in a coroutine.

Parameters

  • *artifacts_or_transforms – Artifacts to gather and/or transforms whose artifacts to gather.

  • num_concurrent – Maximum number of concurrent transforms. Defaults to unrestricted.

Returns

gathered – Coroutine that awaits all gathered artifacts.

group_artifacts(*names: str, squeeze=True) → list[beaver_build.artifacts.Group]

Group artifacts in a context.

Parameters

  • names – Name of nested groups.

  • squeeze – Whether to return a single element if there is only one group.

Returns

groups – Artifacts representing the nested groups.

normalize_artifacts(artifacts: Union[str, beaver_build.artifacts.Artifact, Iterable[Union[str, beaver_build.artifacts.Artifact]]]) → Iterable[beaver_build.artifacts.Artifact]

Normalize one or more artifacts. Strings are implicitly converted to Files.

Parameters

artifacts – One or more artifacts or transforms.

Returns

normalized – Normalized artifacts.