######################
Protocol Documentation
######################




.. _ref_flyteidl/event/event.proto:

flyteidl/event/event.proto
==================================================================





.. _ref_flyteidl.event.DynamicWorkflowNodeMetadata:

DynamicWorkflowNodeMetadata
------------------------------------------------------------------

For dynamic workflow nodes we send information about the dynamic workflow definition that gets generated.



.. csv-table:: DynamicWorkflowNodeMetadata type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "id", ":ref:`ref_flyteidl.core.Identifier`", "", "id represents the unique identifier of the workflow."
   "compiled_workflow", ":ref:`ref_flyteidl.core.CompiledWorkflowClosure`", "", "Represents the compiled representation of the embedded dynamic workflow."







.. _ref_flyteidl.event.ExternalResourceInfo:

ExternalResourceInfo
------------------------------------------------------------------

This message contains metadata about external resources produced or used by a specific task execution.



.. csv-table:: ExternalResourceInfo type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "external_id", ":ref:`ref_string`", "", "Identifier for an external resource created by this task execution, for example Qubole query ID or presto query ids."
   "index", ":ref:`ref_uint32`", "", "A unique index for the external resource with respect to all external resources for this task. Although the identifier may change between task reporting events or retries, this will remain the same to enable aggregating information from multiple reports."
   "retry_attempt", ":ref:`ref_uint32`", "", "Retry attempt number for this external resource, ie., 2 for the second attempt"
   "phase", ":ref:`ref_flyteidl.core.TaskExecution.Phase`", "", "Phase associated with the external resource"
   "cache_status", ":ref:`ref_flyteidl.core.CatalogCacheStatus`", "", "Captures the status of caching for this external resource execution."
   "logs", ":ref:`ref_flyteidl.core.TaskLog`", "repeated", "log information for the external resource execution"







.. _ref_flyteidl.event.NodeExecutionEvent:

NodeExecutionEvent
------------------------------------------------------------------





.. csv-table:: NodeExecutionEvent type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "id", ":ref:`ref_flyteidl.core.NodeExecutionIdentifier`", "", "Unique identifier for this node execution"
   "producer_id", ":ref:`ref_string`", "", "the id of the originator (Propeller) of the event"
   "phase", ":ref:`ref_flyteidl.core.NodeExecution.Phase`", "", ""
   "occurred_at", ":ref:`ref_google.protobuf.Timestamp`", "", "This timestamp represents when the original event occurred, it is generated by the executor of the node."
   "input_uri", ":ref:`ref_string`", "", ""
   "output_uri", ":ref:`ref_string`", "", "URL to the output of the execution, it encodes all the information including Cloud source provider. ie., s3://..."
   "error", ":ref:`ref_flyteidl.core.ExecutionError`", "", "Error information for the execution"
   "output_data", ":ref:`ref_flyteidl.core.LiteralMap`", "", "Raw output data produced by this node execution."
   "workflow_node_metadata", ":ref:`ref_flyteidl.event.WorkflowNodeMetadata`", "", ""
   "task_node_metadata", ":ref:`ref_flyteidl.event.TaskNodeMetadata`", "", ""
   "parent_task_metadata", ":ref:`ref_flyteidl.event.ParentTaskExecutionMetadata`", "", "[To be deprecated] Specifies which task (if any) launched this node."
   "parent_node_metadata", ":ref:`ref_flyteidl.event.ParentNodeExecutionMetadata`", "", "Specifies the parent node of the current node execution. Node executions at level zero will not have a parent node."
   "retry_group", ":ref:`ref_string`", "", "Retry group to indicate grouping of nodes by retries"
   "spec_node_id", ":ref:`ref_string`", "", "Identifier of the node in the original workflow/graph This maps to value of WorkflowTemplate.nodes[X].id"
   "node_name", ":ref:`ref_string`", "", "Friendly readable name for the node"
   "event_version", ":ref:`ref_int32`", "", ""
   "is_parent", ":ref:`ref_bool`", "", "Whether this node launched a subworkflow."
   "is_dynamic", ":ref:`ref_bool`", "", "Whether this node yielded a dynamic workflow."
   "deck_uri", ":ref:`ref_string`", "", "String location uniquely identifying where the deck HTML file is NativeUrl specifies the url in the format of the configured storage provider (e.g. s3://my-bucket/randomstring/suffix.tar)"







.. _ref_flyteidl.event.ParentNodeExecutionMetadata:

ParentNodeExecutionMetadata
------------------------------------------------------------------





.. csv-table:: ParentNodeExecutionMetadata type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "node_id", ":ref:`ref_string`", "", "Unique identifier of the parent node id within the execution This is value of core.NodeExecutionIdentifier.node_id of the parent node"







.. _ref_flyteidl.event.ParentTaskExecutionMetadata:

ParentTaskExecutionMetadata
------------------------------------------------------------------





.. csv-table:: ParentTaskExecutionMetadata type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "id", ":ref:`ref_flyteidl.core.TaskExecutionIdentifier`", "", ""







.. _ref_flyteidl.event.ResourcePoolInfo:

ResourcePoolInfo
------------------------------------------------------------------

This message holds task execution metadata specific to resource allocation used to manage concurrent
executions for a project namespace.



.. csv-table:: ResourcePoolInfo type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "allocation_token", ":ref:`ref_string`", "", "Unique resource ID used to identify this execution when allocating a token."
   "namespace", ":ref:`ref_string`", "", "Namespace under which this task execution requested an allocation token."







.. _ref_flyteidl.event.TaskExecutionEvent:

TaskExecutionEvent
------------------------------------------------------------------

Plugin specific execution event information. For tasks like Python, Hive, Spark, DynamicJob.



.. csv-table:: TaskExecutionEvent type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "task_id", ":ref:`ref_flyteidl.core.Identifier`", "", "ID of the task. In combination with the retryAttempt this will indicate the task execution uniquely for a given parent node execution."
   "parent_node_execution_id", ":ref:`ref_flyteidl.core.NodeExecutionIdentifier`", "", "A task execution is always kicked off by a node execution, the event consumer will use the parent_id to relate the task to it's parent node execution"
   "retry_attempt", ":ref:`ref_uint32`", "", "retry attempt number for this task, ie., 2 for the second attempt"
   "phase", ":ref:`ref_flyteidl.core.TaskExecution.Phase`", "", "Phase associated with the event"
   "producer_id", ":ref:`ref_string`", "", "id of the process that sent this event, mainly for trace debugging"
   "logs", ":ref:`ref_flyteidl.core.TaskLog`", "repeated", "log information for the task execution"
   "occurred_at", ":ref:`ref_google.protobuf.Timestamp`", "", "This timestamp represents when the original event occurred, it is generated by the executor of the task."
   "input_uri", ":ref:`ref_string`", "", "URI of the input file, it encodes all the information including Cloud source provider. ie., s3://..."
   "output_uri", ":ref:`ref_string`", "", "URI to the output of the execution, it will be in a format that encodes all the information including Cloud source provider. ie., s3://..."
   "error", ":ref:`ref_flyteidl.core.ExecutionError`", "", "Error information for the execution"
   "output_data", ":ref:`ref_flyteidl.core.LiteralMap`", "", "Raw output data produced by this task execution."
   "custom_info", ":ref:`ref_google.protobuf.Struct`", "", "Custom data that the task plugin sends back. This is extensible to allow various plugins in the system."
   "phase_version", ":ref:`ref_uint32`", "", "Some phases, like RUNNING, can send multiple events with changed metadata (new logs, additional custom_info, etc) that should be recorded regardless of the lack of phase change. The version field should be incremented when metadata changes across the duration of an individual phase."
   "reason", ":ref:`ref_string`", "", "An optional explanation for the phase transition."
   "task_type", ":ref:`ref_string`", "", "A predefined yet extensible Task type identifier. If the task definition is already registered in flyte admin this type will be identical, but not all task executions necessarily use pre-registered definitions and this type is useful to render the task in the UI, filter task executions, etc."
   "metadata", ":ref:`ref_flyteidl.event.TaskExecutionMetadata`", "", "Metadata around how a task was executed."
   "event_version", ":ref:`ref_int32`", "", "The event version is used to indicate versioned changes in how data is reported using this proto message. For example, event_verison > 0 means that maps tasks report logs using the TaskExecutionMetadata ExternalResourceInfo fields for each subtask rather than the TaskLog in this message."







.. _ref_flyteidl.event.TaskExecutionMetadata:

TaskExecutionMetadata
------------------------------------------------------------------

Holds metadata around how a task was executed.
As a task transitions across event phases during execution some attributes, such its generated name, generated external resources,
and more may grow in size but not change necessarily based on the phase transition that sparked the event update.
Metadata is a container for these attributes across the task execution lifecycle.



.. csv-table:: TaskExecutionMetadata type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "generated_name", ":ref:`ref_string`", "", "Unique, generated name for this task execution used by the backend."
   "external_resources", ":ref:`ref_flyteidl.event.ExternalResourceInfo`", "repeated", "Additional data on external resources on other back-ends or platforms (e.g. Hive, Qubole, etc) launched by this task execution."
   "resource_pool_info", ":ref:`ref_flyteidl.event.ResourcePoolInfo`", "repeated", "Includes additional data on concurrent resource management used during execution.. This is a repeated field because a plugin can request multiple resource allocations during execution."
   "plugin_identifier", ":ref:`ref_string`", "", "The identifier of the plugin used to execute this task."
   "instance_class", ":ref:`ref_flyteidl.event.TaskExecutionMetadata.InstanceClass`", "", ""







.. _ref_flyteidl.event.TaskNodeMetadata:

TaskNodeMetadata
------------------------------------------------------------------





.. csv-table:: TaskNodeMetadata type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "cache_status", ":ref:`ref_flyteidl.core.CatalogCacheStatus`", "", "Captures the status of caching for this execution."
   "catalog_key", ":ref:`ref_flyteidl.core.CatalogMetadata`", "", "This structure carries the catalog artifact information"
   "reservation_status", ":ref:`ref_flyteidl.core.CatalogReservation.Status`", "", "Captures the status of cache reservations for this execution."
   "checkpoint_uri", ":ref:`ref_string`", "", "The latest checkpoint location"
   "dynamic_workflow", ":ref:`ref_flyteidl.event.DynamicWorkflowNodeMetadata`", "", "In the case this task launched a dynamic workflow we capture its structure here."







.. _ref_flyteidl.event.WorkflowExecutionEvent:

WorkflowExecutionEvent
------------------------------------------------------------------





.. csv-table:: WorkflowExecutionEvent type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "execution_id", ":ref:`ref_flyteidl.core.WorkflowExecutionIdentifier`", "", "Workflow execution id"
   "producer_id", ":ref:`ref_string`", "", "the id of the originator (Propeller) of the event"
   "phase", ":ref:`ref_flyteidl.core.WorkflowExecution.Phase`", "", ""
   "occurred_at", ":ref:`ref_google.protobuf.Timestamp`", "", "This timestamp represents when the original event occurred, it is generated by the executor of the workflow."
   "output_uri", ":ref:`ref_string`", "", "URL to the output of the execution, it encodes all the information including Cloud source provider. ie., s3://..."
   "error", ":ref:`ref_flyteidl.core.ExecutionError`", "", "Error information for the execution"
   "output_data", ":ref:`ref_flyteidl.core.LiteralMap`", "", "Raw output data produced by this workflow execution."







.. _ref_flyteidl.event.WorkflowNodeMetadata:

WorkflowNodeMetadata
------------------------------------------------------------------

For Workflow Nodes we need to send information about the workflow that's launched



.. csv-table:: WorkflowNodeMetadata type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "execution_id", ":ref:`ref_flyteidl.core.WorkflowExecutionIdentifier`", "", ""






..
   end messages



.. _ref_flyteidl.event.TaskExecutionMetadata.InstanceClass:

TaskExecutionMetadata.InstanceClass
------------------------------------------------------------------

Includes the broad category of machine used for this specific task execution.

.. csv-table:: Enum TaskExecutionMetadata.InstanceClass values
   :header: "Name", "Number", "Description"
   :widths: auto

   "DEFAULT", "0", "The default instance class configured for the flyte application platform."
   "INTERRUPTIBLE", "1", "The instance class configured for interruptible tasks."


..
   end enums


..
   end HasExtensions


..
   end services




.. _ref_google/protobuf/timestamp.proto:

google/protobuf/timestamp.proto
==================================================================





.. _ref_google.protobuf.Timestamp:

Timestamp
------------------------------------------------------------------

A Timestamp represents a point in time independent of any time zone or local
calendar, encoded as a count of seconds and fractions of seconds at
nanosecond resolution. The count is relative to an epoch at UTC midnight on
January 1, 1970, in the proleptic Gregorian calendar which extends the
Gregorian calendar backwards to year one.

All minutes are 60 seconds long. Leap seconds are "smeared" so that no leap
second table is needed for interpretation, using a [24-hour linear
smear](https://developers.google.com/time/smear).

The range is from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59.999999999Z. By
restricting to that range, we ensure that we can convert to and from [RFC
3339](https://www.ietf.org/rfc/rfc3339.txt) date strings.

# Examples

Example 1: Compute Timestamp from POSIX `time()`.

    Timestamp timestamp;
    timestamp.set_seconds(time(NULL));
    timestamp.set_nanos(0);

Example 2: Compute Timestamp from POSIX `gettimeofday()`.

    struct timeval tv;
    gettimeofday(&tv, NULL);

    Timestamp timestamp;
    timestamp.set_seconds(tv.tv_sec);
    timestamp.set_nanos(tv.tv_usec * 1000);

Example 3: Compute Timestamp from Win32 `GetSystemTimeAsFileTime()`.

    FILETIME ft;
    GetSystemTimeAsFileTime(&ft);
    UINT64 ticks = (((UINT64)ft.dwHighDateTime) << 32) | ft.dwLowDateTime;

    // A Windows tick is 100 nanoseconds. Windows epoch 1601-01-01T00:00:00Z
    // is 11644473600 seconds before Unix epoch 1970-01-01T00:00:00Z.
    Timestamp timestamp;
    timestamp.set_seconds((INT64) ((ticks / 10000000) - 11644473600LL));
    timestamp.set_nanos((INT32) ((ticks % 10000000) * 100));

Example 4: Compute Timestamp from Java `System.currentTimeMillis()`.

    long millis = System.currentTimeMillis();

    Timestamp timestamp = Timestamp.newBuilder().setSeconds(millis / 1000)
        .setNanos((int) ((millis % 1000) * 1000000)).build();

Example 5: Compute Timestamp from Java `Instant.now()`.

    Instant now = Instant.now();

    Timestamp timestamp =
        Timestamp.newBuilder().setSeconds(now.getEpochSecond())
            .setNanos(now.getNano()).build();

Example 6: Compute Timestamp from current time in Python.

    timestamp = Timestamp()
    timestamp.GetCurrentTime()

# JSON Mapping

In JSON format, the Timestamp type is encoded as a string in the
[RFC 3339](https://www.ietf.org/rfc/rfc3339.txt) format. That is, the
format is "{year}-{month}-{day}T{hour}:{min}:{sec}[.{frac_sec}]Z"
where {year} is always expressed using four digits while {month}, {day},
{hour}, {min}, and {sec} are zero-padded to two digits each. The fractional
seconds, which can go up to 9 digits (i.e. up to 1 nanosecond resolution),
are optional. The "Z" suffix indicates the timezone ("UTC"); the timezone
is required. A proto3 JSON serializer should always use UTC (as indicated by
"Z") when printing the Timestamp type and a proto3 JSON parser should be
able to accept both UTC and other timezones (as indicated by an offset).

For example, "2017-01-15T01:30:15.01Z" encodes 15.01 seconds past
01:30 UTC on January 15, 2017.

In JavaScript, one can convert a Date object to this format using the
standard
[toISOString()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toISOString)
method. In Python, a standard `datetime.datetime` object can be converted
to this format using
[`strftime`](https://docs.python.org/2/library/time.html#time.strftime) with
the time format spec '%Y-%m-%dT%H:%M:%S.%fZ'. Likewise, in Java, one can use
the Joda Time's [`ISODateTimeFormat.dateTime()`](
http://www.joda.org/joda-time/apidocs/org/joda/time/format/ISODateTimeFormat.html#dateTime%2D%2D
) to obtain a formatter capable of generating timestamps in this format.



.. csv-table:: Timestamp type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "seconds", ":ref:`ref_int64`", "", "Represents seconds of UTC time since Unix epoch 1970-01-01T00:00:00Z. Must be from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59Z inclusive."
   "nanos", ":ref:`ref_int32`", "", "Non-negative fractions of a second at nanosecond resolution. Negative second values with fractions must still have non-negative nanos values that count forward in time. Must be from 0 to 999,999,999 inclusive."






..
   end messages


..
   end enums


..
   end HasExtensions


..
   end services




.. _ref_google/protobuf/duration.proto:

google/protobuf/duration.proto
==================================================================





.. _ref_google.protobuf.Duration:

Duration
------------------------------------------------------------------

A Duration represents a signed, fixed-length span of time represented
as a count of seconds and fractions of seconds at nanosecond
resolution. It is independent of any calendar and concepts like "day"
or "month". It is related to Timestamp in that the difference between
two Timestamp values is a Duration and it can be added or subtracted
from a Timestamp. Range is approximately +-10,000 years.

# Examples

Example 1: Compute Duration from two Timestamps in pseudo code.

    Timestamp start = ...;
    Timestamp end = ...;
    Duration duration = ...;

    duration.seconds = end.seconds - start.seconds;
    duration.nanos = end.nanos - start.nanos;

    if (duration.seconds < 0 && duration.nanos > 0) {
      duration.seconds += 1;
      duration.nanos -= 1000000000;
    } else if (duration.seconds > 0 && duration.nanos < 0) {
      duration.seconds -= 1;
      duration.nanos += 1000000000;
    }

Example 2: Compute Timestamp from Timestamp + Duration in pseudo code.

    Timestamp start = ...;
    Duration duration = ...;
    Timestamp end = ...;

    end.seconds = start.seconds + duration.seconds;
    end.nanos = start.nanos + duration.nanos;

    if (end.nanos < 0) {
      end.seconds -= 1;
      end.nanos += 1000000000;
    } else if (end.nanos >= 1000000000) {
      end.seconds += 1;
      end.nanos -= 1000000000;
    }

Example 3: Compute Duration from datetime.timedelta in Python.

    td = datetime.timedelta(days=3, minutes=10)
    duration = Duration()
    duration.FromTimedelta(td)

# JSON Mapping

In JSON format, the Duration type is encoded as a string rather than an
object, where the string ends in the suffix "s" (indicating seconds) and
is preceded by the number of seconds, with nanoseconds expressed as
fractional seconds. For example, 3 seconds with 0 nanoseconds should be
encoded in JSON format as "3s", while 3 seconds and 1 nanosecond should
be expressed in JSON format as "3.000000001s", and 3 seconds and 1
microsecond should be expressed in JSON format as "3.000001s".



.. csv-table:: Duration type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "seconds", ":ref:`ref_int64`", "", "Signed seconds of the span of time. Must be from -315,576,000,000 to +315,576,000,000 inclusive. Note: these bounds are computed from: 60 sec/min * 60 min/hr * 24 hr/day * 365.25 days/year * 10000 years"
   "nanos", ":ref:`ref_int32`", "", "Signed fractions of a second at nanosecond resolution of the span of time. Durations less than one second are represented with a 0 `seconds` field and a positive or negative `nanos` field. For durations of one second or more, a non-zero value for the `nanos` field must be of the same sign as the `seconds` field. Must be from -999,999,999 to +999,999,999 inclusive."






..
   end messages


..
   end enums


..
   end HasExtensions


..
   end services




.. _ref_google/protobuf/struct.proto:

google/protobuf/struct.proto
==================================================================





.. _ref_google.protobuf.ListValue:

ListValue
------------------------------------------------------------------

`ListValue` is a wrapper around a repeated field of values.

The JSON representation for `ListValue` is JSON array.



.. csv-table:: ListValue type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "values", ":ref:`ref_google.protobuf.Value`", "repeated", "Repeated field of dynamically typed values."







.. _ref_google.protobuf.Struct:

Struct
------------------------------------------------------------------

`Struct` represents a structured data value, consisting of fields
which map to dynamically typed values. In some languages, `Struct`
might be supported by a native representation. For example, in
scripting languages like JS a struct is represented as an
object. The details of that representation are described together
with the proto support for the language.

The JSON representation for `Struct` is JSON object.



.. csv-table:: Struct type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "fields", ":ref:`ref_google.protobuf.Struct.FieldsEntry`", "repeated", "Unordered map of dynamically typed values."







.. _ref_google.protobuf.Struct.FieldsEntry:

Struct.FieldsEntry
------------------------------------------------------------------





.. csv-table:: Struct.FieldsEntry type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "key", ":ref:`ref_string`", "", ""
   "value", ":ref:`ref_google.protobuf.Value`", "", ""







.. _ref_google.protobuf.Value:

Value
------------------------------------------------------------------

`Value` represents a dynamically typed value which can be either
null, a number, a string, a boolean, a recursive struct value, or a
list of values. A producer of value is expected to set one of these
variants. Absence of any variant indicates an error.

The JSON representation for `Value` is JSON value.



.. csv-table:: Value type fields
   :header: "Field", "Type", "Label", "Description"
   :widths: auto

   "null_value", ":ref:`ref_google.protobuf.NullValue`", "", "Represents a null value."
   "number_value", ":ref:`ref_double`", "", "Represents a double value."
   "string_value", ":ref:`ref_string`", "", "Represents a string value."
   "bool_value", ":ref:`ref_bool`", "", "Represents a boolean value."
   "struct_value", ":ref:`ref_google.protobuf.Struct`", "", "Represents a structured value."
   "list_value", ":ref:`ref_google.protobuf.ListValue`", "", "Represents a repeated `Value`."






..
   end messages



.. _ref_google.protobuf.NullValue:

NullValue
------------------------------------------------------------------

`NullValue` is a singleton enumeration to represent the null value for the
`Value` type union.

 The JSON representation for `NullValue` is JSON `null`.

.. csv-table:: Enum NullValue values
   :header: "Name", "Number", "Description"
   :widths: auto

   "NULL_VALUE", "0", "Null value."


..
   end enums


..
   end HasExtensions


..
   end services