flytekit.extras.persistence.GCSPersistence

class flytekit.extras.persistence.GCSPersistence(default_prefix=None, data_config=None)

This DataPersistence plugin uses a preinstalled GSUtil binary in the container to download and upload data.

The binary can be installed in multiple ways including simply,

pip install gsutil

Methods

Parameters

• default_prefix (Optional[str]) –

• data_config (Optional[flytekit.configuration.DataConfig]) –

construct_path(add_protocol, add_prefix, *paths)

if add_protocol is true then <protocol> is prefixed else Constructs a path in the format <base><delim>*args delim is dependent on the storage medium. each of the args is joined with the delim

Parameters

• add_protocol (bool) –

• add_prefix (bool) –

Return type

str

exists(remote_path)

Parameters

remote_path (Text) – remote gs:// path

Rtype bool

whether the gs file exists or not

get(from_path, to_path, recursive=False)

Retrieves data from from_path and writes to the given to_path (to_path is locally accessible)

Parameters

• from_path (str) –

• to_path (str) –

• recursive (bool) –

listdir(path, recursive=False)

Returns true if the given path exists, else false

Parameters

• path (str) –

• recursive (bool) –

Return type

Generator[str, None, None]

put(from_path, to_path, recursive=False)

Stores data from from_path and writes to the given to_path (from_path is locally accessible)

Parameters

• from_path (str) –

• to_path (str) –

• recursive (bool) –

Attributes

PROTOCOL = 'gs://'

default_prefix

name