Protocol Documentation#

flyteidl/service/admin.proto#

AdminService#

The following defines an RPC service that is also served over HTTP via grpc-gateway. Standard response codes for both are defined here: https://github.com/grpc-ecosystem/grpc-gateway/blob/master/runtime/errors.go

AdminService service methods#

Method Name

Request Type

Response Type

Description

CreateTask

TaskCreateRequest

TaskCreateResponse

Create and upload a Task definition

GetTask

ObjectGetRequest

Task

Fetch a Task definition.

ListTaskIds

NamedEntityIdentifierListRequest

NamedEntityIdentifierList

Fetch a list of NamedEntityIdentifier of task objects.

ListTasks

ResourceListRequest

TaskList

Fetch a list of Task definitions.

CreateWorkflow

WorkflowCreateRequest

WorkflowCreateResponse

Create and upload a Workflow definition

GetWorkflow

ObjectGetRequest

Workflow

Fetch a Workflow definition.

ListWorkflowIds

NamedEntityIdentifierListRequest

NamedEntityIdentifierList

Fetch a list of NamedEntityIdentifier of workflow objects.

ListWorkflows

ResourceListRequest

WorkflowList

Fetch a list of Workflow definitions.

CreateLaunchPlan

LaunchPlanCreateRequest

LaunchPlanCreateResponse

Create and upload a LaunchPlan definition

GetLaunchPlan

ObjectGetRequest

LaunchPlan

Fetch a LaunchPlan definition.

GetActiveLaunchPlan

ActiveLaunchPlanRequest

LaunchPlan

Fetch the active version of a LaunchPlan.

ListActiveLaunchPlans

ActiveLaunchPlanListRequest

LaunchPlanList

List active versions of LaunchPlan.

ListLaunchPlanIds

NamedEntityIdentifierListRequest

NamedEntityIdentifierList

Fetch a list of NamedEntityIdentifier of launch plan objects.

ListLaunchPlans

ResourceListRequest

LaunchPlanList

Fetch a list of LaunchPlan definitions.

UpdateLaunchPlan

LaunchPlanUpdateRequest

LaunchPlanUpdateResponse

Updates the status of a registered LaunchPlan.

CreateExecution

ExecutionCreateRequest

ExecutionCreateResponse

Triggers the creation of a Execution

RelaunchExecution

ExecutionRelaunchRequest

ExecutionCreateResponse

Triggers the creation of an identical Execution

RecoverExecution

ExecutionRecoverRequest

ExecutionCreateResponse

Recreates a previously-run workflow execution that will only start executing from the last known failure point. In Recover mode, users cannot change any input parameters or update the version of the execution. This is extremely useful to recover from system errors and byzantine faults like - Loss of K8s cluster, bugs in platform or instability, machine failures, downstream system failures (downstream services), or simply to recover executions that failed because of retry exhaustion and should complete if tried again. See ExecutionRecoverRequest for more details.

GetExecution

WorkflowExecutionGetRequest

Execution

Fetches a Execution.

UpdateExecution

ExecutionUpdateRequest

ExecutionUpdateResponse

Update execution belonging to project domain Execution.

GetExecutionData

WorkflowExecutionGetDataRequest

WorkflowExecutionGetDataResponse

Fetches input and output data for a Execution.

ListExecutions

ResourceListRequest

ExecutionList

Fetch a list of Execution.

TerminateExecution

ExecutionTerminateRequest

ExecutionTerminateResponse

Terminates an in-progress Execution.

GetNodeExecution

NodeExecutionGetRequest

NodeExecution

Fetches a NodeExecution.

ListNodeExecutions

NodeExecutionListRequest

NodeExecutionList

Fetch a list of NodeExecution.

ListNodeExecutionsForTask

NodeExecutionForTaskListRequest

NodeExecutionList

Fetch a list of NodeExecution launched by the reference TaskExecution.

GetNodeExecutionData

NodeExecutionGetDataRequest

NodeExecutionGetDataResponse

Fetches input and output data for a NodeExecution.

RegisterProject

ProjectRegisterRequest

ProjectRegisterResponse

Registers a Project with the Flyte deployment.

UpdateProject

Project

ProjectUpdateResponse

Updates an existing Project flyteidl.admin.Project should be passed but the domains property should be empty; it will be ignored in the handler as domains cannot be updated via this API.

ListProjects

ProjectListRequest

Projects

Fetches a list of Project

CreateWorkflowEvent

WorkflowExecutionEventRequest

WorkflowExecutionEventResponse

Indicates a WorkflowExecutionEvent has occurred.

CreateNodeEvent

NodeExecutionEventRequest

NodeExecutionEventResponse

Indicates a NodeExecutionEvent has occurred.

CreateTaskEvent

TaskExecutionEventRequest

TaskExecutionEventResponse

Indicates a TaskExecutionEvent has occurred.

GetTaskExecution

TaskExecutionGetRequest

TaskExecution

Fetches a TaskExecution.

ListTaskExecutions

TaskExecutionListRequest

TaskExecutionList

Fetches a list of TaskExecution.

GetTaskExecutionData

TaskExecutionGetDataRequest

TaskExecutionGetDataResponse

Fetches input and output data for a TaskExecution.

UpdateProjectDomainAttributes

ProjectDomainAttributesUpdateRequest

ProjectDomainAttributesUpdateResponse

Creates or updates custom MatchableAttributesConfiguration for a project and domain.

GetProjectDomainAttributes

ProjectDomainAttributesGetRequest

ProjectDomainAttributesGetResponse

Fetches custom MatchableAttributesConfiguration for a project and domain.

DeleteProjectDomainAttributes

ProjectDomainAttributesDeleteRequest

ProjectDomainAttributesDeleteResponse

Deletes custom MatchableAttributesConfiguration for a project and domain.

UpdateProjectAttributes

ProjectAttributesUpdateRequest

ProjectAttributesUpdateResponse

Creates or updates custom MatchableAttributesConfiguration at the project level

GetProjectAttributes

ProjectAttributesGetRequest

ProjectAttributesGetResponse

Fetches custom MatchableAttributesConfiguration for a project and domain.

DeleteProjectAttributes

ProjectAttributesDeleteRequest

ProjectAttributesDeleteResponse

Deletes custom MatchableAttributesConfiguration for a project and domain.

UpdateWorkflowAttributes

WorkflowAttributesUpdateRequest

WorkflowAttributesUpdateResponse

Creates or updates custom MatchableAttributesConfiguration for a project, domain and workflow.

GetWorkflowAttributes

WorkflowAttributesGetRequest

WorkflowAttributesGetResponse

Fetches custom MatchableAttributesConfiguration for a project, domain and workflow.

DeleteWorkflowAttributes

WorkflowAttributesDeleteRequest

WorkflowAttributesDeleteResponse

Deletes custom MatchableAttributesConfiguration for a project, domain and workflow.

ListMatchableAttributes

ListMatchableAttributesRequest

ListMatchableAttributesResponse

Lists custom MatchableAttributesConfiguration for a specific resource type.

ListNamedEntities

NamedEntityListRequest

NamedEntityList

Returns a list of NamedEntity objects.

GetNamedEntity

NamedEntityGetRequest

NamedEntity

Returns a NamedEntity object.

UpdateNamedEntity

NamedEntityUpdateRequest

NamedEntityUpdateResponse

Updates a NamedEntity object.

GetVersion

GetVersionRequest

GetVersionResponse

GetDescriptionEntity

ObjectGetRequest

DescriptionEntity

Fetch a DescriptionEntity object.

ListDescriptionEntities

DescriptionEntityListRequest

DescriptionEntityList

Fetch a list of DescriptionEntity definitions.

flyteidl/service/auth.proto#

OAuth2MetadataRequest#

OAuth2MetadataResponse#

OAuth2MetadataResponse defines an RFC-Compliant response for /.well-known/oauth-authorization-server metadata as defined in https://tools.ietf.org/html/rfc8414

OAuth2MetadataResponse type fields#

Field

Type

Label

Description

issuer

string

Defines the issuer string in all JWT tokens this server issues. The issuer can be admin itself or an external issuer.

authorization_endpoint

string

URL of the authorization server’s authorization endpoint [RFC6749]. This is REQUIRED unless no grant types are supported that use the authorization endpoint.

token_endpoint

string

URL of the authorization server’s token endpoint [RFC6749].

response_types_supported

string

repeated

Array containing a list of the OAuth 2.0 response_type values that this authorization server supports.

scopes_supported

string

repeated

JSON array containing a list of the OAuth 2.0 [RFC6749] scope values that this authorization server supports.

token_endpoint_auth_methods_supported

string

repeated

JSON array containing a list of client authentication methods supported by this token endpoint.

jwks_uri

string

URL of the authorization server’s JWK Set [JWK] document. The referenced document contains the signing key(s) the client uses to validate signatures from the authorization server.

code_challenge_methods_supported

string

repeated

JSON array containing a list of Proof Key for Code Exchange (PKCE) [RFC7636] code challenge methods supported by this authorization server.

grant_types_supported

string

repeated

JSON array containing a list of the OAuth 2.0 grant type values that this authorization server supports.

device_authorization_endpoint

string

URL of the authorization server’s device authorization endpoint, as defined in Section 3.1 of [RFC8628]

PublicClientAuthConfigRequest#

PublicClientAuthConfigResponse#

FlyteClientResponse encapsulates public information that flyte clients (CLIs… etc.) can use to authenticate users.

PublicClientAuthConfigResponse type fields#

Field

Type

Label

Description

client_id

string

client_id to use when initiating OAuth2 authorization requests.

redirect_uri

string

redirect uri to use when initiating OAuth2 authorization requests.

scopes

string

repeated

scopes to request when initiating OAuth2 authorization requests.

authorization_metadata_key

string

Authorization Header to use when passing Access Tokens to the server. If not provided, the client should use the default http Authorization header.

service_http_endpoint

string

ServiceHttpEndpoint points to the http endpoint for the backend. If empty, clients can assume the endpoint used to configure the gRPC connection can be used for the http one respecting the insecure flag to choose between SSL or no SSL connections.

audience

string

audience to use when initiating OAuth2 authorization requests.

AuthMetadataService#

The following defines an RPC service that is also served over HTTP via grpc-gateway. Standard response codes for both are defined here: https://github.com/grpc-ecosystem/grpc-gateway/blob/master/runtime/errors.go RPCs defined in this service must be anonymously accessible.

AuthMetadataService service methods#

Method Name

Request Type

Response Type

Description

GetOAuth2Metadata

OAuth2MetadataRequest

OAuth2MetadataResponse

Anonymously accessible. Retrieves local or external oauth authorization server metadata.

GetPublicClientConfig

PublicClientAuthConfigRequest

PublicClientAuthConfigResponse

Anonymously accessible. Retrieves the client information clients should use when initiating OAuth2 authorization requests.

flyteidl/service/dataproxy.proto#

CreateDownloadLinkRequest#

CreateDownloadLinkRequest defines the request parameters to create a download link (signed url)

CreateDownloadLinkRequest type fields#

Field

Type

Label

Description

artifact_type

ArtifactType

ArtifactType of the artifact requested.

expires_in

Duration

ExpiresIn defines a requested expiration duration for the generated url. The request will be rejected if this exceeds the platform allowed max. +optional. The default value comes from a global config.

node_execution_id

NodeExecutionIdentifier

NodeId is the unique identifier for the node execution. For a task node, this will retrieve the output of the most recent attempt of the task.

CreateDownloadLinkResponse#

CreateDownloadLinkResponse defines the response for the generated links

CreateDownloadLinkResponse type fields#

Field

Type

Label

Description

signed_url

string

repeated

SignedUrl specifies the url to use to download content from (e.g. https://my-bucket.s3.amazonaws.com/randomstring/suffix.tar?X-…)

expires_at

Timestamp

ExpiresAt defines when will the signed URL expire.

CreateDownloadLocationRequest#

CreateDownloadLocationRequest specified request for the CreateDownloadLocation API.

CreateDownloadLocationRequest type fields#

Field

Type

Label

Description

native_url

string

NativeUrl specifies the url in the format of the configured storage provider (e.g. s3://my-bucket/randomstring/suffix.tar)

expires_in

Duration

ExpiresIn defines a requested expiration duration for the generated url. The request will be rejected if this exceeds the platform allowed max. +optional. The default value comes from a global config.

CreateDownloadLocationResponse#

CreateDownloadLocationResponse type fields#

Field

Type

Label

Description

signed_url

string

SignedUrl specifies the url to use to download content from (e.g. https://my-bucket.s3.amazonaws.com/randomstring/suffix.tar?X-…)

expires_at

Timestamp

ExpiresAt defines when will the signed URL expires.

CreateUploadLocationRequest#

CreateUploadLocationRequest specified request for the CreateUploadLocation API.

CreateUploadLocationRequest type fields#

Field

Type

Label

Description

project

string

Project to create the upload location for +required

domain

string

Domain to create the upload location for. +required

filename

string

Filename specifies a desired suffix for the generated location. E.g. file.py or pre/fix/file.zip. +optional. By default, the service will generate a consistent name based on the provided parameters.

expires_in

Duration

ExpiresIn defines a requested expiration duration for the generated url. The request will be rejected if this exceeds the platform allowed max. +optional. The default value comes from a global config.

content_md5

bytes

ContentMD5 restricts the upload location to the specific MD5 provided. The ContentMD5 will also appear in the generated path. +required

CreateUploadLocationResponse#

CreateUploadLocationResponse type fields#

Field

Type

Label

Description

signed_url

string

SignedUrl specifies the url to use to upload content to (e.g. https://my-bucket.s3.amazonaws.com/randomstring/suffix.tar?X-…)

native_url

string

NativeUrl specifies the url in the format of the configured storage provider (e.g. s3://my-bucket/randomstring/suffix.tar)

expires_at

Timestamp

ExpiresAt defines when will the signed URL expires.

ArtifactType#

ArtifactType

Enum ArtifactType values#

Name

Number

Description

ARTIFACT_TYPE_UNDEFINED

0

ARTIFACT_TYPE_UNDEFINED is the default, often invalid, value for the enum.

ARTIFACT_TYPE_DECK

1

ARTIFACT_TYPE_DECK refers to the deck html file optionally generated after a task, a workflow or a launch plan finishes executing.

DataProxyService#

DataProxyService defines an RPC Service that allows access to user-data in a controlled manner.

DataProxyService service methods#

Method Name

Request Type

Response Type

Description

CreateUploadLocation

CreateUploadLocationRequest

CreateUploadLocationResponse

CreateUploadLocation creates a signed url to upload artifacts to for a given project/domain.

CreateDownloadLocation

CreateDownloadLocationRequest

CreateDownloadLocationResponse

CreateDownloadLocation creates a signed url to download artifacts.

CreateDownloadLink

CreateDownloadLinkRequest

CreateDownloadLinkResponse

CreateDownloadLocation creates a signed url to download artifacts.

flyteidl/service/identity.proto#

UserInfoRequest#

UserInfoResponse#

See the OpenID Connect spec at https://openid.net/specs/openid-connect-core-1_0.html#UserInfoResponse for more information.

UserInfoResponse type fields#

Field

Type

Label

Description

subject

string

Locally unique and never reassigned identifier within the Issuer for the End-User, which is intended to be consumed by the Client.

name

string

Full name

preferred_username

string

Shorthand name by which the End-User wishes to be referred to

given_name

string

Given name(s) or first name(s)

family_name

string

Surname(s) or last name(s)

email

string

Preferred e-mail address

picture

string

Profile picture URL

IdentityService#

IdentityService defines an RPC Service that interacts with user/app identities.

IdentityService service methods#

Method Name

Request Type

Response Type

Description

UserInfo

UserInfoRequest

UserInfoResponse

Retrieves user information about the currently logged in user.

flyteidl/service/signal.proto#

SignalService#

SignalService defines an RPC Service that may create, update, and retrieve signal(s).

SignalService service methods#

Method Name

Request Type

Response Type

Description

GetOrCreateSignal

SignalGetOrCreateRequest

Signal

Fetches or creates a Signal.

ListSignals

SignalListRequest

SignalList

Fetch a list of Signal definitions.

SetSignal

SignalSetRequest

SignalSetResponse

Sets the value on a Signal definition