FlyteAdmin serves the main Flyte API. It processes all client requests to the system. Clients include the FlyteConsole, which calls FlyteAdmin to list workflows, get execution details, etc., and FlyteKit, which calls FlyteAdmin to registering workflows, launch workflows etc.
Below, we’ll dive into each component defined in admin in more detail.
FlyteAdmin uses the grpc-gateway library to serve incoming gRPC and HTTP requests with identical handlers. For a more detailed overview of the API, including request and response entities, see the admin service definition. The RPC handlers are a thin shim that enforce some request structure validation and call out to appropriate manager. methods to process requests.
A detailed explanation of the service can be found in the admin service page.
The Admin API is broken up into entities:
Projects (and their respective domains)
Each API entity has an entity manager in FlyteAdmin reposible for implementing business logic for the entity. Entity managers handle full validation for create, update and get requests, and data persistence in the backing store (see the Repository section).
The managers utilize additional components to process requests. These additional components include:
:ref:`workflow engine <divedeep-admin-workflowengine>`: compiles workflows and launches workflow executions from launch plans.
:ref:`data <divedeep-admin-data>` (remote cloud storage): offloads data blobs to the configured cloud provider.
:ref:`runtime <divedeep-admin-config>`: loads values from a config file to assign task resources, initialization values, execution queues and more.
:ref:`async processes <divedeep-admin-async>`: provides functions for scheduling and executing workflows as well as enqueuing and triggering notifications
Serialized entities (tasks, workflows, launch plans) and executions (workflow-, node- and task-) are stored as protos defined here. We use the excellent gorm library to interface with our database, which currently supports a postgres implementation. The actual code for issuing queries with gorm can be found in the gormimpl directory.
The full set of database tables includes:
These database models inherit primary keys and indexes as defined in the corresponding models file.
The repositories code also includes transformers. These convert entities from the database format to a response format for the external API. If you change either of these structures, you will find you must change the corresponding transformers.
This section dives into detail for each top-level directories defined in
Notifications and schedules are handled by async routines that are reponsible for enqueing and subsequently processing dequeued messages.
For the sandbox development, no-op implementations of the notifications and schedule handlers are used to remove external cloud dependencies.
As the name implies,
common houses shared components used across different flyteadmin components in a single, top-level directory to avoid cyclic dependencies. These components include execution naming and phase utils, query filter definitions, query sorting definitions, and named constants.
Data interfaces are primarily handled by the storage library implemented in flytestdlib. However, neither this nor the underlying stow library expose HEAD support so the data package in admin exists as the layer responsible for additional, remote data operations.
The errors directory contains centrally defined errors that are designed for compatibility with gRPC statuses.
Values specific to the flyteadmin application as well as task and workflow registration and execution are configured in the runtime directory. These interfaces expose values configured in the
flyteadmin top-level key in the application config.