Extend Data Persistence layer

Flytekit provides a data persistence layer, which is used for recording metadata that is shared with backend Flyte. This persistence layer is also available for various types to store raw user data and is designed to be cross-cloud compatible. Moreover, it is design to be extensible and users can bring their own data persistence plugins by following the persistence interface. NOTE, this is bound to get more extensive for variety of use-cases, but the core set of apis are battle tested.

flytekit.core.data_persistence

The Data persistence module is used by core flytekit and most of the core TypeTransformers to manage data fetch & store, between the durable backend store and the runtime environment. This is designed to be a pluggable system, with a default simple implementation that ships with the core.

DataPersistence

Base abstract type for all DataPersistence operations.

DataPersistencePlugins

DataPersistencePlugins is the core plugin registry that stores all DataPersistence plugins.

DiskPersistence

The simplest form of persistence that is available with default flytekit - Disk-based persistence.

FileAccessProvider

This is the class that is available through the FlyteContext and can be used for persisting data to the remote durable store.

UnsupportedPersistenceOp

This exception is raised for all methods when a method is not supported by the data persistence layer

DataPersistence Extras

This module provides some default implementations of flytekit.DataPersistence. These implementations use command-line clients to download and upload data. The actual binaries need to be installed for these extras to work. The binaries are not bundled with flytekit to keep it lightweight.

Persistence Extras

GCSPersistence([default_prefix])

This DataPersistence plugin uses a preinstalled GSUtil binary in the container to download and upload data.

HttpPersistence(*args, **kwargs)

DataPersistence implementation for the HTTP protocol.

S3Persistence([default_prefix])

DataPersistence plugin for AWS S3 (and Minio).