flytekit.core.data_persistence.FileAccessProvider

class flytekit.core.data_persistence.FileAccessProvider(local_sandbox_dir, raw_output_prefix, data_config=None)

This is the class that is available through the FlyteContext and can be used for persisting data to the remote durable store.

Parameters

• local_sandbox_dir (Union[str, os.PathLike]) – A local temporary working directory, that should be used to store data

• raw_output_prefix (str) –

• data_config (Optional[flytekit.configuration.DataConfig]) –

Methods

download(remote_path, local_path)

Downloads from remote to local

Parameters

• remote_path (str) –

• local_path (str) –

download_directory(remote_path, local_path)

Downloads directory from given remote to local path

Parameters

• remote_path (str) –

• local_path (str) –

exists(path)

Parameters

path (str) –

Return type

bool

get(from_path, to_path, recursive=False)

Parameters

• from_path (str) –

• to_path (str) –

• recursive (bool) –

get_data(remote_path, local_path, is_multipart=False)

Parameters

• remote_path (str) –

• local_path (str) –

• is_multipart (bool) –

get_filesystem(protocol=None, anonymous=False, **kwargs)

Parameters

• protocol (Optional[str]) –

• anonymous (bool) –

Return type

Optional[fsspec.spec.AbstractFileSystem]

get_filesystem_for_path(path='', anonymous=False, **kwargs)

Parameters

• path (str) –

• anonymous (bool) –

Return type

fsspec.spec.AbstractFileSystem

get_random_local_directory()

Return type

str

get_random_local_path(file_path_or_file_name=None)

Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name

Parameters

file_path_or_file_name (Optional[str]) –

Return type

str

get_random_remote_directory()

get_random_remote_path(file_path_or_file_name=None)

Constructs a randomized path on the configured raw_output_prefix (persistence layer). the random bit is a UUID and allows for disambiguating paths within the same directory.

Use file_path_or_file_name, when you want a random directory, but want to preserve the leaf file name

Parameters

file_path_or_file_name (Optional[str]) –

Return type

str

static is_remote(path)

Deprecated. Let’s find a replacement

Parameters

path (Union[str, os.PathLike]) –

Return type

bool

put(from_path, to_path, recursive=False, **kwargs)

Parameters

• from_path (str) –

• to_path (str) –

• recursive (bool) –

put_data(local_path, remote_path, is_multipart=False, **kwargs)

The implication here is that we’re always going to put data to the remote location, so we .remote to ensure we don’t use the true local proxy if the remote path is a file://

Parameters

• local_path (Union[str, os.PathLike]) –

• remote_path (str) –

• is_multipart (bool) –

static recursive_paths(f, t)

Parameters

• f (str) –

• t (str) –

Return type

Tuple[str, str]

sep(file_system)

Parameters

file_system (Optional[fsspec.spec.AbstractFileSystem]) –

Return type

str

static strip_file_header(path, trim_trailing_sep=False)

Drops file:// if it exists from the file

Parameters

• path (str) –

• trim_trailing_sep (bool) –

Return type

str

upload(file_path, to_path)

Parameters

• file_path (Text) –

• to_path (Text) –

upload_directory(local_path, remote_path)

Parameters

• local_path (Text) –

• remote_path (Text) –

Attributes

data_config

local_access

local_sandbox_dir

This is a context based temp dir.

raw_output_prefix