flytekit.core.data_persistence.FileAccessProvider

class flytekit.core.data_persistence.FileAccessProvider(local_sandbox_dir, raw_output_prefix)[source]

This is the class that is available through the FlyteContext and can be used for persisting data to the remote durable store.

Parameters

  • local_sandbox_dir (Union[str, os.PathLike]) – A local temporary working directory, that should be used to store data

  • raw_output_prefix (str) –

Methods

construct_random_path(persist, file_path_or_file_name=None)[source]

Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name

Parameters
Return type

str

download(remote_path, local_path)[source]

Downloads from remote to local

Parameters

  • remote_path (str) –

  • local_path (str) –

download_directory(remote_path, local_path)[source]

Downloads directory from given remote to local path

Parameters

  • remote_path (str) –

  • local_path (str) –

exists(path)[source]

checks if the given path exists

Parameters

path (str) –

Return type

bool

get_data(remote_path, local_path, is_multipart=False)[source]
Parameters

  • remote_path (Text) –

  • local_path (Text) –

  • is_multipart (bool) –

get_random_local_directory()[source]
Return type

str

get_random_local_path(file_path_or_file_name=None)[source]

Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name

Parameters

file_path_or_file_name (Optional[str]) –

Return type

str

get_random_remote_directory()[source]
get_random_remote_path(file_path_or_file_name=None)[source]

Constructs a randomized path on the configured raw_output_prefix (persistence layer). the random bit is a UUID and allows for disambiguating paths within the same directory.

Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name

Parameters

file_path_or_file_name (Optional[str]) –

Return type

str

static is_remote(path)[source]

Deprecated. Lets find a replacement

Parameters

path (Union[str, os.PathLike]) –

Return type

bool

put_data(local_path, remote_path, is_multipart=False)[source]

The implication here is that we’re always going to put data to the remote location, so we .remote to ensure we don’t use the true local proxy if the remote path is a file://

Parameters

  • local_path (Text) –

  • remote_path (Text) –

  • is_multipart (bool) –

upload(file_path, to_path)[source]
Parameters

  • file_path (Text) –

  • to_path (Text) –

upload_directory(local_path, remote_path)[source]
Parameters

  • local_path (Text) –

  • remote_path (Text) –

Attributes

local_access
local_sandbox_dir