flytekit.extras.persistence.GCSPersistence#
- class flytekit.extras.persistence.GCSPersistence(default_prefix=None, data_config=None)[source]#
This DataPersistence plugin uses a preinstalled GSUtil binary in the container to download and upload data.
The binary can be installed in multiple ways including simply,
pip install gsutil
Methods
- Parameters
data_config (Optional[flytekit.configuration.DataConfig]) –
- construct_path(add_protocol, add_prefix, *paths)[source]#
if add_protocol is true then <protocol> is prefixed else Constructs a path in the format <base><delim>*args delim is dependent on the storage medium. each of the args is joined with the delim
- exists(remote_path)[source]#
- Parameters
remote_path (Text) – remote gs:// path
- Rtype bool
whether the gs file exists or not
- get(from_path, to_path, recursive=False)[source]#
Retrieves data from from_path and writes to the given to_path (to_path is locally accessible)
- listdir(path, recursive=False)#
Returns true if the given path exists, else false
- put(from_path, to_path, recursive=False)[source]#
Stores data from from_path and writes to the given to_path (from_path is locally accessible)
Attributes
- PROTOCOL = 'gs://'#
- default_prefix#
- name#