flytekit.core.data_persistence.FileAccessProvider#

class flytekit.core.data_persistence.FileAccessProvider(local_sandbox_dir, raw_output_prefix, data_config=None)[source]#

This is the class that is available through the FlyteContext and can be used for persisting data to the remote durable store.

Parameters

local_sandbox_dir (Union[str, os.PathLike]) – A local temporary working directory, that should be used to store data
raw_output_prefix (str) –
data_config (Optional[flytekit.configuration.DataConfig]) –

Methods

construct_random_path(persist, file_path_or_file_name=None)[source]#

Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name

Parameters

persist (flytekit.core.data_persistence.DataPersistence) –
file_path_or_file_name (Optional[str]) –

Return type

str

download(remote_path, local_path)[source]#

Downloads from remote to local

Parameters

remote_path (str) –
local_path (str) –

download_directory(remote_path, local_path)[source]#

Downloads directory from given remote to local path

Parameters

remote_path (str) –
local_path (str) –

exists(path)[source]#

checks if the given path exists

Parameters: path (str) –
Return type: bool

get_data(remote_path, local_path, is_multipart=False)[source]#

Parameters

remote_path (Text) –
local_path (Text) –
is_multipart (bool) –

get_random_local_directory()[source]#

Return type: str

get_random_local_path(file_path_or_file_name=None)[source]#

Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name

Parameters: file_path_or_file_name (Optional[str]) –
Return type: str

get_random_remote_directory()[source]#

get_random_remote_path(file_path_or_file_name=None)[source]#

Constructs a randomized path on the configured raw_output_prefix (persistence layer). the random bit is a UUID and allows for disambiguating paths within the same directory.

Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name

Parameters: file_path_or_file_name (Optional[str]) –
Return type: str

static is_remote(path)[source]#

Deprecated. Lets find a replacement

Parameters: path (Union[str, os.PathLike]) –
Return type: bool

put_data(local_path, remote_path, is_multipart=False)[source]#

The implication here is that we’re always going to put data to the remote location, so we .remote to ensure we don’t use the true local proxy if the remote path is a file://

Parameters

local_path (Text) –
remote_path (Text) –
is_multipart (bool) –

upload(file_path, to_path)[source]#

Parameters

file_path (Text) –
to_path (Text) –

upload_directory(local_path, remote_path)[source]#

Parameters

local_path (Text) –
remote_path (Text) –

Attributes

data_config#

local_access#

local_sandbox_dir#

raw_output_prefix#