Protocol Documentation#

flyteidl/core/catalog.proto#

CatalogArtifactTag#

CatalogArtifactTag type fields#

Field

Type

Label

Description

artifact_id

string

Artifact ID is generated name

name

string

Flyte computes the tag automatically, as the hash of the values

CatalogMetadata#

Catalog artifact information with specific metadata

CatalogMetadata type fields#

Field

Type

Label

Description

dataset_id

Identifier

Dataset ID in the catalog

artifact_tag

CatalogArtifactTag

Artifact tag in the catalog

source_task_execution

TaskExecutionIdentifier

Today we only support TaskExecutionIdentifier as a source, as catalog caching only works for task executions

CatalogReservation#

CatalogCacheStatus#

Indicates the status of CatalogCaching. The reason why this is not embedded in TaskNodeMetadata is, that we may use for other types of nodes as well in the future

Enum CatalogCacheStatus values#

Name

Number

Description

CACHE_DISABLED

0

Used to indicate that caching was disabled

CACHE_MISS

1

Used to indicate that the cache lookup resulted in no matches

CACHE_HIT

2

used to indicate that the associated artifact was a result of a previous execution

CACHE_POPULATED

3

used to indicate that the resultant artifact was added to the cache

CACHE_LOOKUP_FAILURE

4

Used to indicate that cache lookup failed because of an error

CACHE_PUT_FAILURE

5

Used to indicate that cache lookup failed because of an error

CACHE_SKIPPED

6

Used to indicate the cache lookup was skipped

CatalogReservation.Status#

Indicates the status of a catalog reservation operation.

Enum CatalogReservation.Status values#

Name

Number

Description

RESERVATION_DISABLED

0

Used to indicate that reservations are disabled

RESERVATION_ACQUIRED

1

Used to indicate that a reservation was successfully acquired or extended

RESERVATION_EXISTS

2

Used to indicate that an active reservation currently exists

RESERVATION_RELEASED

3

Used to indicate that the reservation has been successfully released

RESERVATION_FAILURE

4

Used to indicate that a reservation operation resulted in failure

flyteidl/core/compiler.proto#

CompiledTask#

Output of the Compilation step. This object represent one Task. We store more metadata at this layer

CompiledTask type fields#

Field

Type

Label

Description

template

TaskTemplate

Completely contained TaskTemplate

CompiledWorkflow#

Output of the compilation Step. This object represents one workflow. We store more metadata at this layer

CompiledWorkflow type fields#

Field

Type

Label

Description

template

WorkflowTemplate

Completely contained Workflow Template

connections

ConnectionSet

For internal use only! This field is used by the system and must not be filled in. Any values set will be ignored.

CompiledWorkflowClosure#

A Compiled Workflow Closure contains all the information required to start a new execution, or to visualize a workflow and its details. The CompiledWorkflowClosure should always contain a primary workflow, that is the main workflow that will being the execution. All subworkflows are denormalized. WorkflowNodes refer to the workflow identifiers of compiled subworkflows.

CompiledWorkflowClosure type fields#

Field

Type

Label

Description

primary

CompiledWorkflow

+required

sub_workflows

CompiledWorkflow

repeated

Guaranteed that there will only exist one and only one workflow with a given id, i.e., every sub workflow has a unique identifier. Also every enclosed subworkflow is used either by a primary workflow or by a subworkflow as an inlined workflow +optional

tasks

CompiledTask

repeated

Guaranteed that there will only exist one and only one task with a given id, i.e., every task has a unique id +required (at least 1)

ConnectionSet#

Adjacency list for the workflow. This is created as part of the compilation process. Every process after the compilation step uses this created ConnectionSet

ConnectionSet type fields#

Field

Type

Label

Description

downstream

ConnectionSet.DownstreamEntry

repeated

A list of all the node ids that are downstream from a given node id

upstream

ConnectionSet.UpstreamEntry

repeated

A list of all the node ids, that are upstream of this node id

ConnectionSet.DownstreamEntry#

ConnectionSet.DownstreamEntry type fields#

Field

Type

Label

Description

key

string

value

ConnectionSet.IdList

ConnectionSet.IdList#

ConnectionSet.IdList type fields#

Field

Type

Label

Description

ids

string

repeated

ConnectionSet.UpstreamEntry#

ConnectionSet.UpstreamEntry type fields#

Field

Type

Label

Description

key

string

value

ConnectionSet.IdList

flyteidl/core/condition.proto#

BooleanExpression#

Defines a boolean expression tree. It can be a simple or a conjunction expression. Multiple expressions can be combined using a conjunction or a disjunction to result in a final boolean result.

BooleanExpression type fields#

Field

Type

Label

Description

conjunction

ConjunctionExpression

comparison

ComparisonExpression

ComparisonExpression#

Defines a 2-level tree where the root is a comparison operator and Operands are primitives or known variables. Each expression results in a boolean result.

ComparisonExpression type fields#

Field

Type

Label

Description

operator

ComparisonExpression.Operator

left_value

Operand

right_value

Operand

ConjunctionExpression#

Defines a conjunction expression of two boolean expressions.

ConjunctionExpression type fields#

Field

Type

Label

Description

operator

ConjunctionExpression.LogicalOperator

left_expression

BooleanExpression

right_expression

BooleanExpression

Operand#

Defines an operand to a comparison expression.

Operand type fields#

Field

Type

Label

Description

primitive

Primitive

Can be a constant

var

string

Or one of this node’s input variables

ComparisonExpression.Operator#

Binary Operator for each expression

Enum ComparisonExpression.Operator values#

Name

Number

Description

EQ

0

NEQ

1

GT

2

Greater Than

GTE

3

LT

4

Less Than

LTE

5

ConjunctionExpression.LogicalOperator#

Nested conditions. They can be conjoined using AND / OR Order of evaluation is not important as the operators are Commutative

Enum ConjunctionExpression.LogicalOperator values#

Name

Number

Description

AND

0

Conjunction

OR

1

flyteidl/core/dynamic_job.proto#

DynamicJobSpec#

Describes a set of tasks to execute and how the final outputs are produced.

DynamicJobSpec type fields#

Field

Type

Label

Description

nodes

Node

repeated

A collection of nodes to execute.

min_successes

int64

An absolute number of successful completions of nodes required to mark this job as succeeded. As soon as this criteria is met, the dynamic job will be marked as successful and outputs will be computed. If this number becomes impossible to reach (e.g. number of currently running tasks + number of already succeeded tasks < min_successes) the task will be aborted immediately and marked as failed. The default value of this field, if not specified, is the count of nodes repeated field.

outputs

Binding

repeated

Describes how to bind the final output of the dynamic job from the outputs of executed nodes. The referenced ids in bindings should have the generated id for the subtask.

tasks

TaskTemplate

repeated

[Optional] A complete list of task specs referenced in nodes.

subworkflows

WorkflowTemplate

repeated

[Optional] A complete list of task specs referenced in nodes.

flyteidl/core/errors.proto#

ContainerError#

Error message to propagate detailed errors from container executions to the execution engine.

ContainerError type fields#

Field

Type

Label

Description

code

string

A simplified code for errors, so that we can provide a glossary of all possible errors.

message

string

A detailed error message.

kind

ContainerError.Kind

An abstract error kind for this error. Defaults to Non_Recoverable if not specified.

origin

ExecutionError.ErrorKind

Defines the origin of the error (system, user, unknown).

ErrorDocument#

Defines the errors.pb file format the container can produce to communicate failure reasons to the execution engine.

ErrorDocument type fields#

Field

Type

Label

Description

error

ContainerError

The error raised during execution.

ContainerError.Kind#

Defines a generic error type that dictates the behavior of the retry strategy.

Enum ContainerError.Kind values#

Name

Number

Description

NON_RECOVERABLE

0

RECOVERABLE

1

flyteidl/core/execution.proto#

ExecutionError#

Represents the error message from the execution.

ExecutionError type fields#

Field

Type

Label

Description

code

string

Error code indicates a grouping of a type of error. More Info: <Link>

message

string

Detailed description of the error - including stack trace.

error_uri

string

Full error contents accessible via a URI

kind

ExecutionError.ErrorKind

NodeExecution#

Indicates various phases of Node Execution that only include the time spent to run the nodes/workflows

QualityOfService#

Indicates the priority of an execution.

QualityOfService type fields#

Field

Type

Label

Description

tier

QualityOfService.Tier

spec

QualityOfServiceSpec

QualityOfServiceSpec#

Represents customized execution run-time attributes.

QualityOfServiceSpec type fields#

Field

Type

Label

Description

queueing_budget

Duration

Indicates how much queueing delay an execution can tolerate.

TaskExecution#

Phases that task plugins can go through. Not all phases may be applicable to a specific plugin task, but this is the cumulative list that customers may want to know about for their task.

TaskLog#

Log information for the task that is specific to a log sink When our log story is flushed out, we may have more metadata here like log link expiry

TaskLog type fields#

Field

Type

Label

Description

uri

string

name

string

message_format

TaskLog.MessageFormat

ttl

Duration

WorkflowExecution#

Indicates various phases of Workflow Execution

ExecutionError.ErrorKind#

Error type: System or User

Enum ExecutionError.ErrorKind values#

Name

Number

Description

UNKNOWN

0

USER

1

SYSTEM

2

NodeExecution.Phase#

Enum NodeExecution.Phase values#

Name

Number

Description

UNDEFINED

0

QUEUED

1

RUNNING

2

SUCCEEDED

3

FAILING

4

FAILED

5

ABORTED

6

SKIPPED

7

TIMED_OUT

8

DYNAMIC_RUNNING

9

RECOVERED

10

QualityOfService.Tier#

Enum QualityOfService.Tier values#

Name

Number

Description

UNDEFINED

0

Default: no quality of service specified.

HIGH

1

MEDIUM

2

LOW

3

TaskExecution.Phase#

Enum TaskExecution.Phase values#

Name

Number

Description

UNDEFINED

0

QUEUED

1

RUNNING

2

SUCCEEDED

3

ABORTED

4

FAILED

5

INITIALIZING

6

To indicate cases where task is initializing, like: ErrImagePull, ContainerCreating, PodInitializing

WAITING_FOR_RESOURCES

7

To address cases, where underlying resource is not available: Backoff error, Resource quota exceeded

TaskLog.MessageFormat#

Enum TaskLog.MessageFormat values#

Name

Number

Description

UNKNOWN

0

CSV

1

JSON

2

WorkflowExecution.Phase#

Enum WorkflowExecution.Phase values#

Name

Number

Description

UNDEFINED

0

QUEUED

1

RUNNING

2

SUCCEEDING

3

SUCCEEDED

4

FAILING

5

FAILED

6

ABORTED

7

TIMED_OUT

8

ABORTING

9

flyteidl/core/identifier.proto#

Identifier#

Encapsulation of fields that uniquely identifies a Flyte resource.

Identifier type fields#

Field

Type

Label

Description

resource_type

ResourceType

Identifies the specific type of resource that this identifier corresponds to.

project

string

Name of the project the resource belongs to.

domain

string

Name of the domain the resource belongs to. A domain can be considered as a subset within a specific project.

name

string

User provided value for the resource.

version

string

Specific version of the resource.

NodeExecutionIdentifier#

Encapsulation of fields that identify a Flyte node execution entity.

NodeExecutionIdentifier type fields#

Field

Type

Label

Description

node_id

string

execution_id

WorkflowExecutionIdentifier

SignalIdentifier#

Encapsulation of fields the uniquely identify a signal.

SignalIdentifier type fields#

Field

Type

Label

Description

signal_id

string

Unique identifier for a signal.

execution_id

WorkflowExecutionIdentifier

Identifies the Flyte workflow execution this signal belongs to.

TaskExecutionIdentifier#

Encapsulation of fields that identify a Flyte task execution entity.

TaskExecutionIdentifier type fields#

Field

Type

Label

Description

task_id

Identifier

node_execution_id

NodeExecutionIdentifier

retry_attempt

uint32

WorkflowExecutionIdentifier#

Encapsulation of fields that uniquely identifies a Flyte workflow execution

WorkflowExecutionIdentifier type fields#

Field

Type

Label

Description

project

string

Name of the project the resource belongs to.

domain

string

Name of the domain the resource belongs to. A domain can be considered as a subset within a specific project.

name

string

User or system provided value for the resource.

ResourceType#

Indicates a resource type within Flyte.

Enum ResourceType values#

Name

Number

Description

UNSPECIFIED

0

TASK

1

WORKFLOW

2

LAUNCH_PLAN

3

DATASET

4

A dataset represents an entity modeled in Flyte DataCatalog. A Dataset is also a versioned entity and can be a compilation of multiple individual objects. Eventually all Catalog objects should be modeled similar to Flyte Objects. The Dataset entities makes it possible for the UI and CLI to act on the objects in a similar manner to other Flyte objects

flyteidl/core/interface.proto#

Parameter#

A parameter is used as input to a launch plan and has the special ability to have a default value or mark itself as required.

Parameter type fields#

Field

Type

Label

Description

var

Variable

+required Variable. Defines the type of the variable backing this parameter.

default

Literal

Defines a default value that has to match the variable type defined.

required

bool

+optional, is this value required to be filled.

ParameterMap#

A map of Parameters.

ParameterMap type fields#

Field

Type

Label

Description

parameters

ParameterMap.ParametersEntry

repeated

Defines a map of parameter names to parameters.

ParameterMap.ParametersEntry#

ParameterMap.ParametersEntry type fields#

Field

Type

Label

Description

key

string

value

Parameter

TypedInterface#

Defines strongly typed inputs and outputs.

TypedInterface type fields#

Field

Type

Label

Description

inputs

VariableMap

outputs

VariableMap

Variable#

Defines a strongly typed variable.

Variable type fields#

Field

Type

Label

Description

type

LiteralType

Variable literal type.

description

string

+optional string describing input variable

VariableMap#

A map of Variables

VariableMap type fields#

Field

Type

Label

Description

variables

VariableMap.VariablesEntry

repeated

Defines a map of variable names to variables.

VariableMap.VariablesEntry#

VariableMap.VariablesEntry type fields#

Field

Type

Label

Description

key

string

value

Variable

flyteidl/core/literals.proto#

Binary#

A simple byte array with a tag to help different parts of the system communicate about what is in the byte array. It’s strongly advisable that consumers of this type define a unique tag and validate the tag before parsing the data.

Binary type fields#

Field

Type

Label

Description

value

bytes

tag

string

Binding#

An input/output binding of a variable to either static value or a node output.

Binding type fields#

Field

Type

Label

Description

var

string

Variable name must match an input/output variable of the node.

binding

BindingData

Data to use to bind this variable.

BindingData#

Specifies either a simple value or a reference to another output.

BindingData type fields#

Field

Type

Label

Description

scalar

Scalar

A simple scalar value.

collection

BindingDataCollection

A collection of binding data. This allows nesting of binding data to any number of levels.

promise

OutputReference

References an output promised by another node.

map

BindingDataMap

A map of bindings. The key is always a string.

union

UnionInfo

BindingDataCollection#

A collection of BindingData items.

BindingDataCollection type fields#

Field

Type

Label

Description

bindings

BindingData

repeated

BindingDataMap#

A map of BindingData items.

BindingDataMap type fields#

Field

Type

Label

Description

bindings

BindingDataMap.BindingsEntry

repeated

BindingDataMap.BindingsEntry#

BindingDataMap.BindingsEntry type fields#

Field

Type

Label

Description

key

string

value

BindingData

Blob#

Refers to an offloaded set of files. It encapsulates the type of the store and a unique uri for where the data is. There are no restrictions on how the uri is formatted since it will depend on how to interact with the store.

Blob type fields#

Field

Type

Label

Description

metadata

BlobMetadata

uri

string

BlobMetadata#

BlobMetadata type fields#

Field

Type

Label

Description

type

BlobType

KeyValuePair#

A generic key value pair.

KeyValuePair type fields#

Field

Type

Label

Description

key

string

required.

value

string

+optional.

Literal#

A simple value. This supports any level of nesting (e.g. array of array of array of Blobs) as well as simple primitives.

Literal type fields#

Field

Type

Label

Description

scalar

Scalar

A simple value.

collection

LiteralCollection

A collection of literals to allow nesting.

map

LiteralMap

A map of strings to literals.

hash

string

A hash representing this literal. This is used for caching purposes. For more details refer to RFC 1893 (flyteorg/flyte)

LiteralCollection#

A collection of literals. This is a workaround since oneofs in proto messages cannot contain a repeated field.

LiteralCollection type fields#

Field

Type

Label

Description

literals

Literal

repeated

LiteralMap#

A map of literals. This is a workaround since oneofs in proto messages cannot contain a repeated field.

LiteralMap type fields#

Field

Type

Label

Description

literals

LiteralMap.LiteralsEntry

repeated

LiteralMap.LiteralsEntry#

LiteralMap.LiteralsEntry type fields#

Field

Type

Label

Description

key

string

value

Literal

Primitive#

Primitive Types

Primitive type fields#

Field

Type

Label

Description

integer

int64

float_value

double

string_value

string

boolean

bool

datetime

Timestamp

duration

Duration

RetryStrategy#

Retry strategy associated with an executable unit.

RetryStrategy type fields#

Field

Type

Label

Description

retries

uint32

Number of retries. Retries will be consumed when the job fails with a recoverable error. The number of retries must be less than or equals to 10.

Scalar#

Scalar type fields#

Field

Type

Label

Description

primitive

Primitive

blob

Blob

binary

Binary

schema

Schema

none_type

Void

error

Error

generic

Struct

structured_dataset

StructuredDataset

union

Union

Schema#

A strongly typed schema that defines the interface of data retrieved from the underlying storage medium.

Schema type fields#

Field

Type

Label

Description

uri

string

type

SchemaType

StructuredDataset#

StructuredDataset type fields#

Field

Type

Label

Description

uri

string

String location uniquely identifying where the data is. Should start with the storage location (e.g. s3://, gs://, bq://, etc.)

metadata

StructuredDatasetMetadata

StructuredDatasetMetadata#

StructuredDatasetMetadata type fields#

Field

Type

Label

Description

structured_dataset_type

StructuredDatasetType

Bundle the type information along with the literal. This is here because StructuredDatasets can often be more defined at run time than at compile time. That is, at compile time you might only declare a task to return a pandas dataframe or a StructuredDataset, without any column information, but at run time, you might have that column information. flytekit python will copy this type information into the literal, from the type information, if not provided by the various plugins (encoders). Since this field is run time generated, it’s not used for any type checking.

Union#

The runtime representation of a tagged union value. See UnionType for more details.

Union type fields#

Field

Type

Label

Description

value

Literal

type

LiteralType

UnionInfo#

UnionInfo type fields#

Field

Type

Label

Description

targetType

LiteralType

Void#

Used to denote a nil/null/None assignment to a scalar value. The underlying LiteralType for Void is intentionally undefined since it can be assigned to a scalar of any LiteralType.

flyteidl/core/security.proto#

Identity#

Identity encapsulates the various security identities a task can run as. It’s up to the underlying plugin to pick the right identity for the execution environment.

Identity type fields#

Field

Type

Label

Description

iam_role

string

iam_role references the fully qualified name of Identity & Access Management role to impersonate.

k8s_service_account

string

k8s_service_account references a kubernetes service account to impersonate.

oauth2_client

OAuth2Client

oauth2_client references an oauth2 client. Backend plugins can use this information to impersonate the client when making external calls.

OAuth2Client#

OAuth2Client encapsulates OAuth2 Client Credentials to be used when making calls on behalf of that task.

OAuth2Client type fields#

Field

Type

Label

Description

client_id

string

client_id is the public id for the client to use. The system will not perform any pre-auth validation that the secret requested matches the client_id indicated here. +required

client_secret

Secret

client_secret is a reference to the secret used to authenticate the OAuth2 client. +required

OAuth2TokenRequest#

OAuth2TokenRequest encapsulates information needed to request an OAuth2 token. FLYTE_TOKENS_ENV_PREFIX will be passed to indicate the prefix of the environment variables that will be present if tokens are passed through environment variables. FLYTE_TOKENS_PATH_PREFIX will be passed to indicate the prefix of the path where secrets will be mounted if tokens are passed through file mounts.

OAuth2TokenRequest type fields#

Field

Type

Label

Description

name

string

name indicates a unique id for the token request within this task token requests. It’ll be used as a suffix for environment variables and as a filename for mounting tokens as files. +required

type

OAuth2TokenRequest.Type

type indicates the type of the request to make. Defaults to CLIENT_CREDENTIALS. +required

client

OAuth2Client

client references the client_id/secret to use to request the OAuth2 token. +required

idp_discovery_endpoint

string

idp_discovery_endpoint references the discovery endpoint used to retrieve token endpoint and other related information. +optional

token_endpoint

string

token_endpoint references the token issuance endpoint. If idp_discovery_endpoint is not provided, this parameter is mandatory. +optional

Secret#

Secret encapsulates information about the secret a task needs to proceed. An environment variable FLYTE_SECRETS_ENV_PREFIX will be passed to indicate the prefix of the environment variables that will be present if secrets are passed through environment variables. FLYTE_SECRETS_DEFAULT_DIR will be passed to indicate the prefix of the path where secrets will be mounted if secrets are passed through file mounts.

Secret type fields#

Field

Type

Label

Description

group

string

The name of the secret group where to find the key referenced below. For K8s secrets, this should be the name of the v1/secret object. For Confidant, this should be the Credential name. For Vault, this should be the secret name. For AWS Secret Manager, this should be the name of the secret. +required

group_version

string

The group version to fetch. This is not supported in all secret management systems. It’ll be ignored for the ones that do not support it. +optional

key

string

The name of the secret to mount. This has to match an existing secret in the system. It’s up to the implementation of the secret management system to require case sensitivity. For K8s secrets, Confidant and Vault, this should match one of the keys inside the secret. For AWS Secret Manager, it’s ignored. +optional

mount_requirement

Secret.MountType

mount_requirement is optional. Indicates where the secret has to be mounted. If provided, the execution will fail if the underlying key management system cannot satisfy that requirement. If not provided, the default location will depend on the key management system. +optional

SecurityContext#

SecurityContext holds security attributes that apply to tasks.

SecurityContext type fields#

Field

Type

Label

Description

run_as

Identity

run_as encapsulates the identity a pod should run as. If the task fills in multiple fields here, it’ll be up to the backend plugin to choose the appropriate identity for the execution engine the task will run on.

secrets

Secret

repeated

secrets indicate the list of secrets the task needs in order to proceed. Secrets will be mounted/passed to the pod as it starts. If the plugin responsible for kicking of the task will not run it on a flyte cluster (e.g. AWS Batch), it’s the responsibility of the plugin to fetch the secret (which means propeller identity will need access to the secret) and to pass it to the remote execution engine.

tokens

OAuth2TokenRequest

repeated

tokens indicate the list of token requests the task needs in order to proceed. Tokens will be mounted/passed to the pod as it starts. If the plugin responsible for kicking of the task will not run it on a flyte cluster (e.g. AWS Batch), it’s the responsibility of the plugin to fetch the secret (which means propeller identity will need access to the secret) and to pass it to the remote execution engine.

OAuth2TokenRequest.Type#

Type of the token requested.

Enum OAuth2TokenRequest.Type values#

Name

Number

Description

CLIENT_CREDENTIALS

0

CLIENT_CREDENTIALS indicates a 2-legged OAuth token requested using client credentials.

Secret.MountType#

Enum Secret.MountType values#

Name

Number

Description

ANY

0

Default case, indicates the client can tolerate either mounting options.

ENV_VAR

1

ENV_VAR indicates the secret needs to be mounted as an environment variable.

FILE

2

FILE indicates the secret needs to be mounted as a file.

flyteidl/core/tasks.proto#

Container#

Container type fields#

Field

Type

Label

Description

image

string

Container image url. Eg: docker/redis:latest

command

string

repeated

Command to be executed, if not provided, the default entrypoint in the container image will be used.

args

string

repeated

These will default to Flyte given paths. If provided, the system will not append known paths. If the task still needs flyte’s inputs and outputs path, add $(FLYTE_INPUT_FILE), $(FLYTE_OUTPUT_FILE) wherever makes sense and the system will populate these before executing the container.

resources

Resources

Container resources requirement as specified by the container engine.

env

KeyValuePair

repeated

Environment variables will be set as the container is starting up.

config

KeyValuePair

repeated

Deprecated. Allows extra configs to be available for the container. TODO: elaborate on how configs will become available. Deprecated, please use TaskTemplate.config instead.

ports

ContainerPort

repeated

Ports to open in the container. This feature is not supported by all execution engines. (e.g. supported on K8s but not supported on AWS Batch) Only K8s

data_config

DataLoadingConfig

BETA: Optional configuration for DataLoading. If not specified, then default values are used. This makes it possible to to run a completely portable container, that uses inputs and outputs only from the local file-system and without having any reference to flyteidl. This is supported only on K8s at the moment. If data loading is enabled, then data will be mounted in accompanying directories specified in the DataLoadingConfig. If the directories are not specified, inputs will be mounted onto and outputs will be uploaded from a pre-determined file-system path. Refer to the documentation to understand the default paths. Only K8s

architecture

Container.Architecture

ContainerPort#

Defines port properties for a container.

ContainerPort type fields#

Field

Type

Label

Description

container_port

uint32

Number of port to expose on the pod’s IP address. This must be a valid port number, 0 < x < 65536.

DataLoadingConfig#

This configuration allows executing raw containers in Flyte using the Flyte CoPilot system. Flyte CoPilot, eliminates the needs of flytekit or sdk inside the container. Any inputs required by the users container are side-loaded in the input_path Any outputs generated by the user container - within output_path are automatically uploaded.

DataLoadingConfig type fields#

Field

Type

Label

Description

enabled

bool

Flag enables DataLoading Config. If this is not set, data loading will not be used!

input_path

string

File system path (start at root). This folder will contain all the inputs exploded to a separate file. Example, if the input interface needs (x: int, y: blob, z: multipart_blob) and the input path is ‘/var/flyte/inputs’, then the file system will look like /var/flyte/inputs/inputs.<metadata format dependent -> .pb .json .yaml> -> Format as defined previously. The Blob and Multipart blob will reference local filesystem instead of remote locations /var/flyte/inputs/x -> X is a file that contains the value of x (integer) in string format /var/flyte/inputs/y -> Y is a file in Binary format /var/flyte/inputs/z/… -> Note Z itself is a directory More information about the protocol - refer to docs #TODO reference docs here

output_path

string

File system path (start at root). This folder should contain all the outputs for the task as individual files and/or an error text file

format

DataLoadingConfig.LiteralMapFormat

In the inputs folder, there will be an additional summary/metadata file that contains references to all files or inlined primitive values. This format decides the actual encoding for the data. Refer to the encoding to understand the specifics of the contents and the encoding

io_strategy

IOStrategy

IOStrategy#

Strategy to use when dealing with Blob, Schema, or multipart blob data (large datasets)

IOStrategy type fields#

Field

Type

Label

Description

download_mode

IOStrategy.DownloadMode

Mode to use to manage downloads

upload_mode

IOStrategy.UploadMode

Mode to use to manage uploads

K8sObjectMetadata#

Metadata for building a kubernetes object when a task is executed.

K8sObjectMetadata type fields#

Field

Type

Label

Description

labels

K8sObjectMetadata.LabelsEntry

repeated

Optional labels to add to the pod definition.

annotations

K8sObjectMetadata.AnnotationsEntry

repeated

Optional annotations to add to the pod definition.

K8sObjectMetadata.AnnotationsEntry#

K8sObjectMetadata.AnnotationsEntry type fields#

Field

Type

Label

Description

key

string

value

string

K8sObjectMetadata.LabelsEntry#

K8sObjectMetadata.LabelsEntry type fields#

Field

Type

Label

Description

key

string

value

string

K8sPod#

Defines a pod spec and additional pod metadata that is created when a task is executed.

K8sPod type fields#

Field

Type

Label

Description

metadata

K8sObjectMetadata

Contains additional metadata for building a kubernetes pod.

pod_spec

Struct

Defines the primary pod spec created when a task is executed. This should be a JSON-marshalled pod spec, which can be defined in - go, using: kubernetes/api - python: using kubernetes-client/python

Resources#

A customizable interface to convey resources requested for a container. This can be interpreted differently for different container engines.

Resources type fields#

Field

Type

Label

Description

requests

Resources.ResourceEntry

repeated

The desired set of resources requested. ResourceNames must be unique within the list.

limits

Resources.ResourceEntry

repeated

Defines a set of bounds (e.g. min/max) within which the task can reliably run. ResourceNames must be unique within the list.

Resources.ResourceEntry#

Encapsulates a resource name and value.

Resources.ResourceEntry type fields#

Field

Type

Label

Description

name

Resources.ResourceName

Resource name.

value

string

Value must be a valid k8s quantity. See kubernetes/apimachinery

RuntimeMetadata#

Runtime information. This is loosely defined to allow for extensibility.

RuntimeMetadata type fields#

Field

Type

Label

Description

type

RuntimeMetadata.RuntimeType

Type of runtime.

version

string

Version of the runtime. All versions should be backward compatible. However, certain cases call for version checks to ensure tighter validation or setting expectations.

flavor

string

+optional It can be used to provide extra information about the runtime (e.g. python, golang… etc.).

Sql#

Sql represents a generic sql workload with a statement and dialect.

Sql type fields#

Field

Type

Label

Description

statement

string

The actual query to run, the query can have templated parameters. We use Flyte’s Golang templating format for Query templating. For example, insert overwrite directory ‘{{ .rawOutputDataPrefix }}’ stored as parquet select * from my_table where ds = ‘{{ .Inputs.ds }}’

dialect

Sql.Dialect

TaskMetadata#

Task Metadata

TaskMetadata type fields#

Field

Type

Label

Description

discoverable

bool

Indicates whether the system should attempt to lookup this task’s output to avoid duplication of work.

runtime

RuntimeMetadata

Runtime information about the task.

timeout

Duration

The overall timeout of a task including user-triggered retries.

retries

RetryStrategy

Number of retries per task.

discovery_version

string

Indicates a logical version to apply to this task for the purpose of discovery.

deprecated_error_message

string

If set, this indicates that this task is deprecated. This will enable owners of tasks to notify consumers of the ending of support for a given task.

interruptible

bool

cache_serializable

bool

Indicates whether the system should attempt to execute discoverable instances in serial to avoid duplicate work

generates_deck

bool

Indicates whether the task will generate a Deck URI when it finishes executing.

tags

TaskMetadata.TagsEntry

repeated

Arbitrary tags that allow users and the platform to store small but arbitrary labels

TaskMetadata.TagsEntry#

TaskMetadata.TagsEntry type fields#

Field

Type

Label

Description

key

string

value

string

TaskTemplate#

A Task structure that uniquely identifies a task in the system Tasks are registered as a first step in the system.

TaskTemplate type fields#

Field

Type

Label

Description

id

Identifier

Auto generated taskId by the system. Task Id uniquely identifies this task globally.

type

string

A predefined yet extensible Task type identifier. This can be used to customize any of the components. If no extensions are provided in the system, Flyte will resolve the this task to its TaskCategory and default the implementation registered for the TaskCategory.

metadata

TaskMetadata

Extra metadata about the task.

interface

TypedInterface

A strongly typed interface for the task. This enables others to use this task within a workflow and guarantees compile-time validation of the workflow to avoid costly runtime failures.

custom

Struct

Custom data about the task. This is extensible to allow various plugins in the system.

container

Container

k8s_pod

K8sPod

sql

Sql

task_type_version

int32

This can be used to customize task handling at execution time for the same task type.

security_context

SecurityContext

security_context encapsulates security attributes requested to run this task.

config

TaskTemplate.ConfigEntry

repeated

Metadata about the custom defined for this task. This is extensible to allow various plugins in the system to use as required. reserve the field numbers 1 through 15 for very frequently occurring message elements

TaskTemplate.ConfigEntry#

TaskTemplate.ConfigEntry type fields#

Field

Type

Label

Description

key

string

value

string

Container.Architecture#

Architecture-type the container image supports.

Enum Container.Architecture values#

Name

Number

Description

UNKNOWN

0

AMD64

1

ARM64

2

ARM_V6

3

ARM_V7

4

DataLoadingConfig.LiteralMapFormat#

LiteralMapFormat decides the encoding format in which the input metadata should be made available to the containers. If the user has access to the protocol buffer definitions, it is recommended to use the PROTO format. JSON and YAML do not need any protobuf definitions to read it All remote references in core.LiteralMap are replaced with local filesystem references (the data is downloaded to local filesystem)

Enum DataLoadingConfig.LiteralMapFormat values#

Name

Number

Description

JSON

0

JSON / YAML for the metadata (which contains inlined primitive values). The representation is inline with the standard json specification as specified - https://www.json.org/json-en.html

YAML

1

PROTO

2

Proto is a serialized binary of core.LiteralMap defined in flyteidl/core

IOStrategy.DownloadMode#

Mode to use for downloading

Enum IOStrategy.DownloadMode values#

Name

Number

Description

DOWNLOAD_EAGER

0

All data will be downloaded before the main container is executed

DOWNLOAD_STREAM

1

Data will be downloaded as a stream and an End-Of-Stream marker will be written to indicate all data has been downloaded. Refer to protocol for details

DO_NOT_DOWNLOAD

2

Large objects (offloaded) will not be downloaded

IOStrategy.UploadMode#

Mode to use for uploading

Enum IOStrategy.UploadMode values#

Name

Number

Description

UPLOAD_ON_EXIT

0

All data will be uploaded after the main container exits

UPLOAD_EAGER

1

Data will be uploaded as it appears. Refer to protocol specification for details

DO_NOT_UPLOAD

2

Data will not be uploaded, only references will be written

Resources.ResourceName#

Known resource names.

Enum Resources.ResourceName values#

Name

Number

Description

UNKNOWN

0

CPU

1

GPU

2

MEMORY

3

STORAGE

4

EPHEMERAL_STORAGE

5

For Kubernetes-based deployments, pods use ephemeral local storage for scratch space, caching, and for logs.

RuntimeMetadata.RuntimeType#

Enum RuntimeMetadata.RuntimeType values#

Name

Number

Description

OTHER

0

FLYTE_SDK

1

Sql.Dialect#

The dialect of the SQL statement. This is used to validate and parse SQL statements at compilation time to avoid expensive runtime operations. If set to an unsupported dialect, no validation will be done on the statement. We support the following dialect: ansi, hive.

Enum Sql.Dialect values#

Name

Number

Description

UNDEFINED

0

ANSI

1

HIVE

2

OTHER

3

flyteidl/core/types.proto#

BlobType#

Defines type behavior for blob objects

BlobType type fields#

Field

Type

Label

Description

format

string

Format can be a free form string understood by SDK/UI etc like csv, parquet etc

dimensionality

BlobType.BlobDimensionality

EnumType#

Enables declaring enum types, with predefined string values For len(values) > 0, the first value in the ordered list is regarded as the default value. If you wish To provide no defaults, make the first value as undefined.

EnumType type fields#

Field

Type

Label

Description

values

string

repeated

Predefined set of enum values.

Error#

Represents an error thrown from a node.

Error type fields#

Field

Type

Label

Description

failed_node_id

string

The node id that threw the error.

message

string

Error message thrown.

LiteralType#

Defines a strong type to allow type checking between interfaces.

LiteralType type fields#

Field

Type

Label

Description

simple

SimpleType

A simple type that can be compared one-to-one with another.

schema

SchemaType

A complex type that requires matching of inner fields.

collection_type

LiteralType

Defines the type of the value of a collection. Only homogeneous collections are allowed.

map_value_type

LiteralType

Defines the type of the value of a map type. The type of the key is always a string.

blob

BlobType

A blob might have specialized implementation details depending on associated metadata.

enum_type

EnumType

Defines an enum with pre-defined string values.

structured_dataset_type

StructuredDatasetType

Generalized schema support

union_type

UnionType

Defines an union type with pre-defined LiteralTypes.

metadata

Struct

This field contains type metadata that is descriptive of the type, but is NOT considered in type-checking. This might be used by consumers to identify special behavior or display extended information for the type.

annotation

TypeAnnotation

This field contains arbitrary data that might have special semantic meaning for the client but does not effect internal flyte behavior.

structure

TypeStructure

Hints to improve type matching.

OutputReference#

A reference to an output produced by a node. The type can be retrieved -and validated- from the underlying interface of the node.

OutputReference type fields#

Field

Type

Label

Description

node_id

string

Node id must exist at the graph layer.

var

string

Variable name must refer to an output variable for the node.

SchemaType#

Defines schema columns and types to strongly type-validate schemas interoperability.

SchemaType type fields#

Field

Type

Label

Description

columns

SchemaType.SchemaColumn

repeated

A list of ordered columns this schema comprises of.

SchemaType.SchemaColumn#

SchemaType.SchemaColumn type fields#

Field

Type

Label

Description

name

string

A unique name -within the schema type- for the column

type

SchemaType.SchemaColumn.SchemaColumnType

The column type. This allows a limited set of types currently.

StructuredDatasetType#

StructuredDatasetType type fields#

Field

Type

Label

Description

columns

StructuredDatasetType.DatasetColumn

repeated

A list of ordered columns this schema comprises of.

format

string

This is the storage format, the format of the bits at rest parquet, feather, csv, etc. For two types to be compatible, the format will need to be an exact match.

external_schema_type

string

This is a string representing the type that the bytes in external_schema_bytes are formatted in. This is an optional field that will not be used for type checking.

external_schema_bytes

bytes

The serialized bytes of a third-party schema library like Arrow. This is an optional field that will not be used for type checking.

StructuredDatasetType.DatasetColumn#

StructuredDatasetType.DatasetColumn type fields#

Field

Type

Label

Description

name

string

A unique name within the schema type for the column.

literal_type

LiteralType

The column type.

TypeAnnotation#

TypeAnnotation encapsulates registration time information about a type. This can be used for various control-plane operations. TypeAnnotation will not be available at runtime when a task runs.

TypeAnnotation type fields#

Field

Type

Label

Description

annotations

Struct

A arbitrary JSON payload to describe a type.

TypeStructure#

Hints to improve type matching e.g. allows distinguishing output from custom type transformers even if the underlying IDL serialization matches.

TypeStructure type fields#

Field

Type

Label

Description

tag

string

Must exactly match for types to be castable

UnionType#

Defines a tagged union type, also known as a variant (and formally as the sum type).

A sum type S is defined by a sequence of types (A, B, C, …), each tagged by a string tag A value of type S is constructed from a value of any of the variant types. The specific choice of type is recorded by storing the varaint’s tag with the literal value and can be examined in runtime.

Type S is typically written as S := Apple A | Banana B | Cantaloupe C | …

Notably, a nullable (optional) type is a sum type between some type X and the singleton type representing a null-value: Optional X := X | Null

See also: https://en.wikipedia.org/wiki/Tagged_union

UnionType type fields#

Field

Type

Label

Description

variants

LiteralType

repeated

Predefined set of variants in union.

BlobType.BlobDimensionality#

Enum BlobType.BlobDimensionality values#

Name

Number

Description

SINGLE

0

MULTIPART

1

SchemaType.SchemaColumn.SchemaColumnType#

Enum SchemaType.SchemaColumn.SchemaColumnType values#

Name

Number

Description

INTEGER

0

FLOAT

1

STRING

2

BOOLEAN

3

DATETIME

4

DURATION

5

SimpleType#

Define a set of simple types.

Enum SimpleType values#

Name

Number

Description

NONE

0

INTEGER

1

FLOAT

2

STRING

3

BOOLEAN

4

DATETIME

5

DURATION

6

BINARY

7

ERROR

8

STRUCT

9

flyteidl/core/workflow.proto#

Alias#

Links a variable to an alias.

Alias type fields#

Field

Type

Label

Description

var

string

Must match one of the output variable names on a node.

alias

string

A workflow-level unique alias that downstream nodes can refer to in their input.

ApproveCondition#

ApproveCondition represents a dependency on an external approval. During execution, this will manifest as a boolean signal with the provided signal_id.

ApproveCondition type fields#

Field

Type

Label

Description

signal_id

string

A unique identifier for the requested boolean signal.

BranchNode#

BranchNode is a special node that alter the flow of the workflow graph. It allows the control flow to branch at runtime based on a series of conditions that get evaluated on various parameters (e.g. inputs, primitives).

BranchNode type fields#

Field

Type

Label

Description

if_else

IfElseBlock

+required

GateNode#

GateNode refers to the condition that is required for the gate to successfully complete.

GateNode type fields#

Field

Type

Label

Description

approve

ApproveCondition

ApproveCondition represents a dependency on an external approval provided by a boolean signal.

signal

SignalCondition

SignalCondition represents a dependency on an signal.

sleep

SleepCondition

SleepCondition represents a dependency on waiting for the specified duration.

IfBlock#

Defines a condition and the execution unit that should be executed if the condition is satisfied.

IfBlock type fields#

Field

Type

Label

Description

condition

BooleanExpression

then_node

Node

IfElseBlock#

Defines a series of if/else blocks. The first branch whose condition evaluates to true is the one to execute. If no conditions were satisfied, the else_node or the error will execute.

IfElseBlock type fields#

Field

Type

Label

Description

case

IfBlock

+required. First condition to evaluate.

other

IfBlock

repeated

+optional. Additional branches to evaluate.

else_node

Node

The node to execute in case none of the branches were taken.

error

Error

An error to throw in case none of the branches were taken.

Node#

A Workflow graph Node. One unit of execution in the graph. Each node can be linked to a Task, a Workflow or a branch node.

Node type fields#

Field

Type

Label

Description

id

string

A workflow-level unique identifier that identifies this node in the workflow. ‘inputs’ and ‘outputs’ are reserved node ids that cannot be used by other nodes.

metadata

NodeMetadata

Extra metadata about the node.

inputs

Binding

repeated

Specifies how to bind the underlying interface’s inputs. All required inputs specified in the underlying interface must be fulfilled.

upstream_node_ids

string

repeated

+optional Specifies execution dependency for this node ensuring it will only get scheduled to run after all its upstream nodes have completed. This node will have an implicit dependency on any node that appears in inputs field.

output_aliases

Alias

repeated

+optional. A node can define aliases for a subset of its outputs. This is particularly useful if different nodes need to conform to the same interface (e.g. all branches in a branch node). Downstream nodes must refer to this nodes outputs using the alias if one’s specified.

task_node

TaskNode

Information about the Task to execute in this node.

workflow_node

WorkflowNode

Information about the Workflow to execute in this mode.

branch_node

BranchNode

Information about the branch node to evaluate in this node.

gate_node

GateNode

Information about the condition to evaluate in this node.

NodeMetadata#

Defines extra information about the Node.

NodeMetadata type fields#

Field

Type

Label

Description

name

string

A friendly name for the Node

timeout

Duration

The overall timeout of a task.

retries

RetryStrategy

Number of retries per task.

interruptible

bool

SignalCondition#

SignalCondition represents a dependency on an signal.

SignalCondition type fields#

Field

Type

Label

Description

signal_id

string

A unique identifier for the requested signal.

type

LiteralType

A type denoting the required value type for this signal.

output_variable_name

string

The variable name for the signal value in this nodes outputs.

SleepCondition#

SleepCondition represents a dependency on waiting for the specified duration.

SleepCondition type fields#

Field

Type

Label

Description

duration

Duration

The overall duration for this sleep.

TaskNode#

Refers to the task that the Node is to execute.

TaskNode type fields#

Field

Type

Label

Description

reference_id

Identifier

A globally unique identifier for the task.

overrides

TaskNodeOverrides

Optional overrides applied at task execution time.

TaskNodeOverrides#

Optional task node overrides that will be applied at task execution time.

TaskNodeOverrides type fields#

Field

Type

Label

Description

resources

Resources

A customizable interface to convey resources requested for a task container.

WorkflowMetadata#

This is workflow layer metadata. These settings are only applicable to the workflow as a whole, and do not percolate down to child entities (like tasks) launched by the workflow.

WorkflowMetadata type fields#

Field

Type

Label

Description

quality_of_service

QualityOfService

Indicates the runtime priority of workflow executions.

on_failure

WorkflowMetadata.OnFailurePolicy

Defines how the system should behave when a failure is detected in the workflow execution.

tags

WorkflowMetadata.TagsEntry

repeated

Arbitrary tags that allow users and the platform to store small but arbitrary labels

WorkflowMetadata.TagsEntry#

WorkflowMetadata.TagsEntry type fields#

Field

Type

Label

Description

key

string

value

string

WorkflowMetadataDefaults#

The difference between these settings and the WorkflowMetadata ones is that these are meant to be passed down to a workflow’s underlying entities (like tasks). For instance, ‘interruptible’ has no meaning at the workflow layer, it is only relevant when a task executes. The settings here are the defaults that are passed to all nodes unless explicitly overridden at the node layer. If you are adding a setting that applies to both the Workflow itself, and everything underneath it, it should be added to both this object and the WorkflowMetadata object above.

WorkflowMetadataDefaults type fields#

Field

Type

Label

Description

interruptible

bool

Whether child nodes of the workflow are interruptible.

WorkflowNode#

Refers to a the workflow the node is to execute.

WorkflowNode type fields#

Field

Type

Label

Description

launchplan_ref

Identifier

A globally unique identifier for the launch plan.

sub_workflow_ref

Identifier

Reference to a subworkflow, that should be defined with the compiler context

WorkflowTemplate#

Flyte Workflow Structure that encapsulates task, branch and subworkflow nodes to form a statically analyzable, directed acyclic graph.

WorkflowTemplate type fields#

Field

Type

Label

Description

id

Identifier

A globally unique identifier for the workflow.

metadata

WorkflowMetadata

Extra metadata about the workflow.

interface

TypedInterface

Defines a strongly typed interface for the Workflow. This can include some optional parameters.

nodes

Node

repeated

A list of nodes. In addition, ‘globals’ is a special reserved node id that can be used to consume workflow inputs.

outputs

Binding

repeated

A list of output bindings that specify how to construct workflow outputs. Bindings can pull node outputs or specify literals. All workflow outputs specified in the interface field must be bound in order for the workflow to be validated. A workflow has an implicit dependency on all of its nodes to execute successfully in order to bind final outputs. Most of these outputs will be Binding’s with a BindingData of type OutputReference. That is, your workflow can just have an output of some constant (Output(5)), but usually, the workflow will be pulling outputs from the output of a task.

failure_node

Node

+optional A catch-all node. This node is executed whenever the execution engine determines the workflow has failed. The interface of this node must match the Workflow interface with an additional input named ‘error’ of type pb.lyft.flyte.core.Error.

metadata_defaults

WorkflowMetadataDefaults

workflow defaults

WorkflowMetadata.OnFailurePolicy#

Failure Handling Strategy

Enum WorkflowMetadata.OnFailurePolicy values#

Name

Number

Description

FAIL_IMMEDIATELY

0

FAIL_IMMEDIATELY instructs the system to fail as soon as a node fails in the workflow. It’ll automatically abort all currently running nodes and clean up resources before finally marking the workflow executions as failed.

FAIL_AFTER_EXECUTABLE_NODES_COMPLETE

1

FAIL_AFTER_EXECUTABLE_NODES_COMPLETE instructs the system to make as much progress as it can. The system will not alter the dependencies of the execution graph so any node that depend on the failed node will not be run. Other nodes that will be executed to completion before cleaning up resources and marking the workflow execution as failed.

flyteidl/core/workflow_closure.proto#

WorkflowClosure#

Defines an enclosed package of workflow and tasks it references.

WorkflowClosure type fields#

Field

Type

Label

Description

workflow

WorkflowTemplate

required. Workflow template.

tasks

TaskTemplate

repeated

optional. A collection of tasks referenced by the workflow. Only needed if the workflow references tasks.

google/protobuf/timestamp.proto#

Timestamp#

A Timestamp represents a point in time independent of any time zone or local calendar, encoded as a count of seconds and fractions of seconds at nanosecond resolution. The count is relative to an epoch at UTC midnight on January 1, 1970, in the proleptic Gregorian calendar which extends the Gregorian calendar backwards to year one.

All minutes are 60 seconds long. Leap seconds are “smeared” so that no leap second table is needed for interpretation, using a [24-hour linear smear](https://developers.google.com/time/smear).

The range is from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59.999999999Z. By restricting to that range, we ensure that we can convert to and from [RFC 3339](https://www.ietf.org/rfc/rfc3339.txt) date strings.

# Examples

Example 1: Compute Timestamp from POSIX time().

Timestamp timestamp; timestamp.set_seconds(time(NULL)); timestamp.set_nanos(0);

Example 2: Compute Timestamp from POSIX gettimeofday().

struct timeval tv; gettimeofday(&tv, NULL);

Timestamp timestamp; timestamp.set_seconds(tv.tv_sec); timestamp.set_nanos(tv.tv_usec * 1000);

Example 3: Compute Timestamp from Win32 GetSystemTimeAsFileTime().

FILETIME ft; GetSystemTimeAsFileTime(&ft); UINT64 ticks = (((UINT64)ft.dwHighDateTime) << 32) | ft.dwLowDateTime;

// A Windows tick is 100 nanoseconds. Windows epoch 1601-01-01T00:00:00Z // is 11644473600 seconds before Unix epoch 1970-01-01T00:00:00Z. Timestamp timestamp; timestamp.set_seconds((INT64) ((ticks / 10000000) - 11644473600LL)); timestamp.set_nanos((INT32) ((ticks % 10000000) * 100));

Example 4: Compute Timestamp from Java System.currentTimeMillis().

long millis = System.currentTimeMillis();

Timestamp timestamp = Timestamp.newBuilder().setSeconds(millis / 1000)

.setNanos((int) ((millis % 1000) * 1000000)).build();

Example 5: Compute Timestamp from Java Instant.now().

Instant now = Instant.now();

Timestamp timestamp =
Timestamp.newBuilder().setSeconds(now.getEpochSecond())

.setNanos(now.getNano()).build();

Example 6: Compute Timestamp from current time in Python.

timestamp = Timestamp() timestamp.GetCurrentTime()

# JSON Mapping

In JSON format, the Timestamp type is encoded as a string in the [RFC 3339](https://www.ietf.org/rfc/rfc3339.txt) format. That is, the format is “{year}-{month}-{day}T{hour}:{min}:{sec}[.{frac_sec}]Z” where {year} is always expressed using four digits while {month}, {day}, {hour}, {min}, and {sec} are zero-padded to two digits each. The fractional seconds, which can go up to 9 digits (i.e. up to 1 nanosecond resolution), are optional. The “Z” suffix indicates the timezone (“UTC”); the timezone is required. A proto3 JSON serializer should always use UTC (as indicated by “Z”) when printing the Timestamp type and a proto3 JSON parser should be able to accept both UTC and other timezones (as indicated by an offset).

For example, “2017-01-15T01:30:15.01Z” encodes 15.01 seconds past 01:30 UTC on January 15, 2017.

In JavaScript, one can convert a Date object to this format using the standard [toISOString()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toISOString) method. In Python, a standard datetime.datetime object can be converted to this format using [strftime](https://docs.python.org/2/library/time.html#time.strftime) with the time format spec ‘%Y-%m-%dT%H:%M:%S.%fZ’. Likewise, in Java, one can use the Joda Time’s [ISODateTimeFormat.dateTime()]( http://www.joda.org/joda-time/apidocs/org/joda/time/format/ISODateTimeFormat.html#dateTime%2D%2D ) to obtain a formatter capable of generating timestamps in this format.

Timestamp type fields#

Field

Type

Label

Description

seconds

int64

Represents seconds of UTC time since Unix epoch 1970-01-01T00:00:00Z. Must be from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59Z inclusive.

nanos

int32

Non-negative fractions of a second at nanosecond resolution. Negative second values with fractions must still have non-negative nanos values that count forward in time. Must be from 0 to 999,999,999 inclusive.

google/protobuf/duration.proto#

Duration#

A Duration represents a signed, fixed-length span of time represented as a count of seconds and fractions of seconds at nanosecond resolution. It is independent of any calendar and concepts like “day” or “month”. It is related to Timestamp in that the difference between two Timestamp values is a Duration and it can be added or subtracted from a Timestamp. Range is approximately +-10,000 years.

# Examples

Example 1: Compute Duration from two Timestamps in pseudo code.

Timestamp start = …; Timestamp end = …; Duration duration = …;

duration.seconds = end.seconds - start.seconds; duration.nanos = end.nanos - start.nanos;

if (duration.seconds < 0 && duration.nanos > 0) {

duration.seconds += 1; duration.nanos -= 1000000000;

} else if (duration.seconds > 0 && duration.nanos < 0) {

duration.seconds -= 1; duration.nanos += 1000000000;

}

Example 2: Compute Timestamp from Timestamp + Duration in pseudo code.

Timestamp start = …; Duration duration = …; Timestamp end = …;

end.seconds = start.seconds + duration.seconds; end.nanos = start.nanos + duration.nanos;

if (end.nanos < 0) {

end.seconds -= 1; end.nanos += 1000000000;

} else if (end.nanos >= 1000000000) {

end.seconds += 1; end.nanos -= 1000000000;

}

Example 3: Compute Duration from datetime.timedelta in Python.

td = datetime.timedelta(days=3, minutes=10) duration = Duration() duration.FromTimedelta(td)

# JSON Mapping

In JSON format, the Duration type is encoded as a string rather than an object, where the string ends in the suffix “s” (indicating seconds) and is preceded by the number of seconds, with nanoseconds expressed as fractional seconds. For example, 3 seconds with 0 nanoseconds should be encoded in JSON format as “3s”, while 3 seconds and 1 nanosecond should be expressed in JSON format as “3.000000001s”, and 3 seconds and 1 microsecond should be expressed in JSON format as “3.000001s”.

Duration type fields#

Field

Type

Label

Description

seconds

int64

Signed seconds of the span of time. Must be from -315,576,000,000 to +315,576,000,000 inclusive. Note: these bounds are computed from: 60 sec/min * 60 min/hr * 24 hr/day * 365.25 days/year * 10000 years

nanos

int32

Signed fractions of a second at nanosecond resolution of the span of time. Durations less than one second are represented with a 0 seconds field and a positive or negative nanos field. For durations of one second or more, a non-zero value for the nanos field must be of the same sign as the seconds field. Must be from -999,999,999 to +999,999,999 inclusive.

google/protobuf/struct.proto#

ListValue#

ListValue is a wrapper around a repeated field of values.

The JSON representation for ListValue is JSON array.

ListValue type fields#

Field

Type

Label

Description

values

Value

repeated

Repeated field of dynamically typed values.

Struct#

Struct represents a structured data value, consisting of fields which map to dynamically typed values. In some languages, Struct might be supported by a native representation. For example, in scripting languages like JS a struct is represented as an object. The details of that representation are described together with the proto support for the language.

The JSON representation for Struct is JSON object.

Struct type fields#

Field

Type

Label

Description

fields

Struct.FieldsEntry

repeated

Unordered map of dynamically typed values.

Struct.FieldsEntry#

Struct.FieldsEntry type fields#

Field

Type

Label

Description

key

string

value

Value

Value#

Value represents a dynamically typed value which can be either null, a number, a string, a boolean, a recursive struct value, or a list of values. A producer of value is expected to set one of these variants. Absence of any variant indicates an error.

The JSON representation for Value is JSON value.

Value type fields#

Field

Type

Label

Description

null_value

NullValue

Represents a null value.

number_value

double

Represents a double value.

string_value

string

Represents a string value.

bool_value

bool

Represents a boolean value.

struct_value

Struct

Represents a structured value.

list_value

ListValue

Represents a repeated Value.

NullValue#

NullValue is a singleton enumeration to represent the null value for the Value type union.

The JSON representation for NullValue is JSON null.

Enum NullValue values#

Name

Number

Description

NULL_VALUE

0

Null value.

Scalar Value Types#

double#

double language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

double

double

double

float

float64

double

float

Float

float#

float language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

float

float

float

float

float32

float

float

Float

int32#

Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead.

int32 language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

int32

int32

int

int

int32

int

integer

Bignum or Fixnum (as required)

int64#

Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead.

int64 language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

int64

int64

long

int/long

int64

long

integer/string

Bignum

uint32#

Uses variable-length encoding.

uint32 language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

uint32

uint32

int

int/long

uint32

uint

integer

Bignum or Fixnum (as required)

uint64#

Uses variable-length encoding.

uint64 language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

uint64

uint64

long

int/long

uint64

ulong

integer/string

Bignum or Fixnum (as required)

sint32#

Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s.

sint32 language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

sint32

int32

int

int

int32

int

integer

Bignum or Fixnum (as required)

sint64#

Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s.

sint64 language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

sint64

int64

long

int/long

int64

long

integer/string

Bignum

fixed32#

Always four bytes. More efficient than uint32 if values are often greater than 2^28.

fixed32 language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

fixed32

uint32

int

int

uint32

uint

integer

Bignum or Fixnum (as required)

fixed64#

Always eight bytes. More efficient than uint64 if values are often greater than 2^56.

fixed64 language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

fixed64

uint64

long

int/long

uint64

ulong

integer/string

Bignum

sfixed32#

Always four bytes.

sfixed32 language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

sfixed32

int32

int

int

int32

int

integer

Bignum or Fixnum (as required)

sfixed64#

Always eight bytes.

sfixed64 language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

sfixed64

int64

long

int/long

int64

long

integer/string

Bignum

bool#

bool language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

bool

bool

boolean

boolean

bool

bool

boolean

TrueClass/FalseClass

string#

A string must always contain UTF-8 encoded or 7-bit ASCII text.

string language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

string

string

String

str/unicode

string

string

string

String (UTF-8)

bytes#

May contain any arbitrary sequence of bytes.

bytes language representation#

.proto Type

C++

Java

Python

Go

C#

PHP

Ruby

bytes

string

ByteString

str

[]byte

ByteString

string

String (ASCII-8BIT)