flytekit.types.directory.types

Directory

Classes

Errors

• FlyteAssertion
• TypeTransformerFailedError

flytekit.types.directory.types.Any

Special type indicating an unconstrained type.

• Any is compatible with every type.
• Any assumed to have all methods.
• All values assumed to be instances of Any.

Note that all the above statements are true from the point of view of static type checkers. At runtime, Any should not be used with instance checks.

flytekit.types.directory.types.AsyncTypeTransformer

Base transformer type that should be implemented for every python native type that can be handled by flytekit

def AsyncTypeTransformer(
    name: str,
    t: Type[T],
    enable_type_assertions: bool,
):

Methods

assert_type()

def assert_type(
    t: Type[T],
    v: T,
):

async_to_literal()

def async_to_literal(
    ctx: FlyteContext,
    python_val: T,
    python_type: Type[T],
    expected: LiteralType,
):

Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch

async_to_python_value()

def async_to_python_value(
    ctx: FlyteContext,
    lv: Literal,
    expected_python_type: Type[T],
):

Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised

from_binary_idl()

def from_binary_idl(
    binary_idl_object: Binary,
    expected_python_type: Type[T],
):

This function primarily handles deserialization for untyped dicts, dataclasses, Pydantic BaseModels, and attribute access.｀

For untyped dict, dataclass, and pydantic basemodel: Life Cycle (Untyped Dict as example): python val -> msgpack bytes -> binary literal scalar -> msgpack bytes -> python val (to_literal) (from_binary_idl)

For attribute access: Life Cycle: python val -> msgpack bytes -> binary literal scalar -> resolved golang value -> binary literal scalar -> msgpack bytes -> python val (to_literal) (propeller attribute access) (from_binary_idl)

from_generic_idl()

def from_generic_idl(
    generic: Struct,
    expected_python_type: Type[T],
):

TODO: Support all Flyte Types. This is for dataclass attribute access from input created from the Flyte Console.

Note:

• This can be removed in the future when the Flyte Console support generate Binary IDL Scalar as input.

get_literal_type()

def get_literal_type(
    t: Type[T],
):

Converts the python type to a Flyte LiteralType

guess_python_type()

def guess_python_type(
    literal_type: LiteralType,
):

Converts the Flyte LiteralType to a python object type.

isinstance_generic()

def isinstance_generic(
    obj,
    generic_alias,
):

to_html()

def to_html(
    ctx: FlyteContext,
    python_val: T,
    expected_python_type: Type[T],
):

Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div

to_literal()

def to_literal(
    ctx: FlyteContext,
    python_val: typing.Any,
    python_type: Type[T],
    expected: LiteralType,
):

Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch

to_python_value()

def to_python_value(
    ctx: FlyteContext,
    lv: Literal,
    expected_python_type: Type[T],
):

Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised

Properties

flytekit.types.directory.types.Binary

def Binary(
    value,
    tag,
):

Methods

from_flyte_idl()

def from_flyte_idl(
    pb2_object,
):

serialize_to_string()

def serialize_to_string()

short_string()

def short_string()

to_flyte_idl()

def to_flyte_idl()

verbose_string()

def verbose_string()

Properties

flytekit.types.directory.types.Blob

def Blob(
    metadata,
    uri,
):

This literal model is used to represent binary data offloaded to some storage location which is identifiable with a unique string. See :py:class:flytekit.FlyteFile as an example.

Methods

from_flyte_idl()

def from_flyte_idl(
    proto,
):

serialize_to_string()

def serialize_to_string()

short_string()

def short_string()

to_flyte_idl()

def to_flyte_idl()

verbose_string()

def verbose_string()

Properties

flytekit.types.directory.types.BlobMetadata

This is metadata for the Blob literal.

def BlobMetadata(
    type,
):

Methods

from_flyte_idl()

def from_flyte_idl(
    proto,
):

serialize_to_string()

def serialize_to_string()

short_string()

def short_string()

to_flyte_idl()

def to_flyte_idl()

verbose_string()

def verbose_string()

Properties

flytekit.types.directory.types.BlobType

This type represents offloaded data and is typically used for things like files.

def BlobType(
    format,
    dimensionality,
):

Methods

from_flyte_idl()

def from_flyte_idl(
    proto,
):

serialize_to_string()

def serialize_to_string()

short_string()

def short_string()

to_flyte_idl()

def to_flyte_idl()

verbose_string()

def verbose_string()

Properties

flytekit.types.directory.types.DataClassJsonMixin

DataClassJsonMixin is an ABC that functions as a Mixin.

As with other ABCs, it should not be instantiated directly.

Methods

from_dict()

def from_dict(
    kvs: typing.Union[dict, list, str, int, float, bool, NoneType],
    infer_missing,
):

from_json()

def from_json(
    s: typing.Union[str, bytes, bytearray],
    parse_float,
    parse_int,
    parse_constant,
    infer_missing,
    kw,
):

schema()

def schema(
    infer_missing: bool,
    only,
    exclude,
    many: bool,
    context,
    load_only,
    dump_only,
    partial: bool,
    unknown,
):

to_dict()

def to_dict(
    encode_json,
):

to_json()

def to_json(
    skipkeys: bool,
    ensure_ascii: bool,
    check_circular: bool,
    allow_nan: bool,
    indent: typing.Union[int, str, NoneType],
    separators: typing.Tuple[str, str],
    default: typing.Callable,
    sort_keys: bool,
    kw,
):

flytekit.types.directory.types.FileExt

Used for annotating file extension types of FlyteFile. This is useful for extensions that have periods in them, e.g., “tar.gz”.

Example: TAR_GZ = Annotated[str, FileExt(“tar.gz”)]

def FileExt(
    ext: str,
):

Methods

check_and_convert_to_str()

def check_and_convert_to_str(
    item: typing.Union[typing.Type, str],
):

flytekit.types.directory.types.FlyteAssertion

Assertion failed.

def FlyteAssertion(
    args,
    timestamp: typing.Optional[float],
):

Properties

flytekit.types.directory.types.FlyteContext

This is an internal-facing context object, that most users will not have to deal with. It’s essentially a globally available grab bag of settings and objects that allows flytekit to do things like convert complex types, run and compile workflows, serialize Flyte entities, etc.

Even though this object as a current_context function on it, it should not be called directly. Please use the :py:class:flytekit.FlyteContextManager object instead.

Please do not confuse this object with the :py:class:flytekit.ExecutionParameters object.

def FlyteContext(
    file_access: FileAccessProvider,
    level: int,
    flyte_client: Optional['friendly_client.SynchronousFlyteClient'],
    compilation_state: Optional[CompilationState],
    execution_state: Optional[ExecutionState],
    serialization_settings: Optional[SerializationSettings],
    in_a_condition: bool,
    origin_stackframe: Optional[traceback.FrameSummary],
    output_metadata_tracker: Optional[OutputMetadataTracker],
    worker_queue: Optional[Controller],
):

Methods

current_context()

def current_context()

This method exists only to maintain backwards compatibility. Please use FlyteContextManager.current_context() instead.

Users of flytekit should be wary not to confuse the object returned from this function with :py:func:flytekit.current_context

enter_conditional_section()

def enter_conditional_section()

get_deck()

def get_deck()

Returns the deck that was created as part of the last execution.

The return value depends on the execution environment. In a notebook, the return value is compatible with IPython.display and should be rendered in the notebook.

.. code-block:: python

with flytekit.new_context() as ctx: my_task(…) ctx.get_deck()

OR if you wish to explicitly display

.. code-block:: python

from IPython import display display(ctx.get_deck())

get_origin_stackframe_repr()

def get_origin_stackframe_repr()

new_builder()

def new_builder()

new_compilation_state()

def new_compilation_state(
    prefix: str,
):

Creates and returns a default compilation state. For most of the code this should be the entrypoint of compilation, otherwise the code should always uses - with_compilation_state

new_execution_state()

def new_execution_state(
    working_dir: Optional[os.PathLike],
):

Creates and returns a new default execution state. This should be used at the entrypoint of execution, in all other cases it is preferable to use with_execution_state

set_stackframe()

def set_stackframe(
    s: traceback.FrameSummary,
):

with_client()

def with_client(
    c: SynchronousFlyteClient,
):

with_compilation_state()

def with_compilation_state(
    c: CompilationState,
):

with_execution_state()

def with_execution_state(
    es: ExecutionState,
):

with_file_access()

def with_file_access(
    fa: FileAccessProvider,
):

with_new_compilation_state()

def with_new_compilation_state()

with_output_metadata_tracker()

def with_output_metadata_tracker(
    t: OutputMetadataTracker,
):

with_serialization_settings()

def with_serialization_settings(
    ss: SerializationSettings,
):

with_worker_queue()

def with_worker_queue(
    wq: Controller,
):

Properties

flytekit.types.directory.types.FlyteContextManager

FlyteContextManager manages the execution context within Flytekit. It holds global state of either compilation or Execution. It is not thread-safe and can only be run as a single threaded application currently. Context’s within Flytekit is useful to manage compilation state and execution state. Refer to CompilationState and ExecutionState for more information. FlyteContextManager provides a singleton stack to manage these contexts.

Typical usage is

.. code-block:: python

FlyteContextManager.initialize() with FlyteContextManager.with_context(o) as ctx: pass

If required - not recommended you can use

FlyteContextManager.push_context()

but correspondingly a pop_context should be called

FlyteContextManager.pop_context()

Methods

add_signal_handler()

def add_signal_handler(
    handler: typing.Callable[[int, FrameType], typing.Any],
):

current_context()

def current_context()

get_origin_stackframe()

def get_origin_stackframe(
    limit,
):

initialize()

def initialize()

Re-initializes the context and erases the entire context

pop_context()

def pop_context()

push_context()

def push_context(
    ctx: FlyteContext,
    f: Optional[traceback.FrameSummary],
):

size()

def size()

with_context()

def with_context(
    b: FlyteContext.Builder,
):

flytekit.types.directory.types.FlyteDirToMultipartBlobTransformer

This transformer handles conversion between the Python native FlyteDirectory class defined above, and the Flyte IDL literal/type of Multipart Blob. Please see the FlyteDirectory comments for additional information.

.. caution:

The transformer will not check if the given path is actually a directory. This is because the path could be a remote reference.

def FlyteDirToMultipartBlobTransformer()

Methods

assert_type()

def assert_type(
    t: typing.Type[FlyteDirectory],
    v: typing.Union[FlyteDirectory, os.PathLike, str],
):

async_to_literal()

def async_to_literal(
    ctx: FlyteContext,
    python_val: FlyteDirectory,
    python_type: typing.Type[FlyteDirectory],
    expected: LiteralType,
):

Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch

async_to_python_value()

def async_to_python_value(
    ctx: FlyteContext,
    lv: Literal,
    expected_python_type: typing.Type[FlyteDirectory],
):

Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised

dict_to_flyte_directory()

def dict_to_flyte_directory(
    dict_obj: typing.Dict[str, str],
    expected_python_type: typing.Type[FlyteDirectory],
):

from_binary_idl()

def from_binary_idl(
    binary_idl_object: Binary,
    expected_python_type: typing.Type[FlyteDirectory],
):

If the input is from flytekit, the Life Cycle will be as follows:

Life Cycle: binary IDL -> resolved binary -> bytes -> expected Python object (flytekit customized (propeller processing) (flytekit binary IDL) (flytekit customized serialization) deserialization)

Example Code: @dataclass class DC: fd: FlyteDirectory

@workflow def wf(dc: DC): t_fd(dc.fd)

Note:

• The deserialization is the same as put a flyte directory in a dataclass, which will deserialize by the mashumaro’s API.

Related PR:

• Title: Override Dataclass Serialization/Deserialization Behavior for FlyteTypes via Mashumaro
• Link: https://github.com/flyteorg/flytekit/pull/2554

from_generic_idl()

def from_generic_idl(
    generic: Struct,
    expected_python_type: typing.Type[FlyteDirectory],
):

If the input is from Flyte Console, the Life Cycle will be as follows:

Life Cycle: json str -> protobuf struct -> resolved protobuf struct -> expected Python object (console user input) (console output) (propeller) (flytekit customized deserialization)

Example Code: @dataclass class DC: fd: FlyteDirectory

@workflow def wf(dc: DC): t_fd(dc.fd)

Note:

• The deserialization is the same as put a flyte directory in a dataclass, which will deserialize by the mashumaro’s API.

Related PR:

• Title: Override Dataclass Serialization/Deserialization Behavior for FlyteTypes via Mashumaro
• Link: https://github.com/flyteorg/flytekit/pull/2554

get_format()

def get_format(
    t: typing.Type[FlyteDirectory],
):

get_literal_type()

def get_literal_type(
    t: typing.Type[FlyteDirectory],
):

Converts the python type to a Flyte LiteralType

guess_python_type()

def guess_python_type(
    literal_type: LiteralType,
):

Converts the Flyte LiteralType to a python object type.

isinstance_generic()

def isinstance_generic(
    obj,
    generic_alias,
):

to_html()

def to_html(
    ctx: FlyteContext,
    python_val: T,
    expected_python_type: Type[T],
):

Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div

to_literal()

def to_literal(
    ctx: FlyteContext,
    python_val: typing.Any,
    python_type: Type[T],
    expected: LiteralType,
):

Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch

to_python_value()

def to_python_value(
    ctx: FlyteContext,
    lv: Literal,
    expected_python_type: Type[T],
):

Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised

Properties

flytekit.types.directory.types.FlyteDirectory

def FlyteDirectory(
    path: typing.Union[str, os.PathLike],
    downloader: typing.Optional[typing.Callable],
    remote_directory: typing.Optional[typing.Union[os.PathLike, str, typing.Literal[False]]],
):

Methods

crawl()

def crawl(
    maxdepth: typing.Optional[int],
    topdown: bool,
    kwargs,
):

Crawl returns a generator of all files prefixed by any sub-folders under the given “FlyteDirectory”. if details=True is passed, then it will return a dictionary as specified by fsspec.

Example:

list(fd.crawl()) [("/base", “file1”), ("/base", “dir1/file1”), ("/base", “dir2/file1”), ("/base", “dir1/dir/file1”)]

list(x.crawl(detail=True)) [(’/tmp/test’, {‘my-dir/ab.py’: {’name’: ‘/tmp/test/my-dir/ab.py’, ‘size’: 0, ’type’: ‘file’, ‘created’: 1677720780.2318847, ‘islink’: False, ‘mode’: 33188, ‘uid’: 501, ‘gid’: 0, ‘mtime’: 1677720780.2317934, ‘ino’: 1694329, ’nlink’: 1}})]

deserialize_flyte_dir()

def deserialize_flyte_dir(
    args,
    kwargs,
):

download()

def download()

extension()

def extension()

from_dict()

def from_dict(
    kvs: typing.Union[dict, list, str, int, float, bool, NoneType],
    infer_missing,
):

from_json()

def from_json(
    s: typing.Union[str, bytes, bytearray],
    parse_float,
    parse_int,
    parse_constant,
    infer_missing,
    kw,
):

from_source()

def from_source(
    source: str | os.PathLike,
):

Create a new FlyteDirectory object with the remote source set to the input

listdir()

def listdir(
    directory: FlyteDirectory,
):

This function will list all files and folders in the given directory, but without downloading the contents. In addition, it will return a list of FlyteFile and FlyteDirectory objects that have ability to lazily download the contents of the file/folder. For example:

.. code-block:: python

entity = FlyteDirectory.listdir(directory) for e in entity: print(“s3 object:”, e.remote_source)

s3 object: s3://test-flytedir/file1.txt

s3 object: s3://test-flytedir/file2.txt

s3 object: s3://test-flytedir/sub_dir

open(entity[0], “r”) # This will download the file to the local disk. open(entity[0], “r”) # flytekit will read data from the local disk if you open it again.

new()

def new(
    dirname: str | os.PathLike,
):

Create a new FlyteDirectory object in current Flyte working directory.

new_dir()

def new_dir(
    name: typing.Optional[str],
):

This will create a new folder under the current folder. If given a name, it will use the name given, otherwise it’ll pick a random string. Collisions are not checked.

new_file()

def new_file(
    name: typing.Optional[str],
):

This will create a new file under the current folder. If given a name, it will use the name given, otherwise it’ll pick a random string. Collisions are not checked.

new_remote()

def new_remote(
    stem: typing.Optional[str],
    alt: typing.Optional[str],
):

Create a new FlyteDirectory object using the currently configured default remote in the context (i.e. the raw_output_prefix configured in the current FileAccessProvider object in the context). This is used if you explicitly have a folder somewhere that you want to create files under. If you want to write a whole folder, you can let your task return a FlyteDirectory object, and let flytekit handle the uploading.

schema()

def schema(
    infer_missing: bool,
    only,
    exclude,
    many: bool,
    context,
    load_only,
    dump_only,
    partial: bool,
    unknown,
):

serialize_flyte_dir()

def serialize_flyte_dir(
    args,
    kwargs,
):

to_dict()

def to_dict(
    encode_json,
):

to_json()

def to_json(
    skipkeys: bool,
    ensure_ascii: bool,
    check_circular: bool,
    allow_nan: bool,
    indent: typing.Union[int, str, NoneType],
    separators: typing.Tuple[str, str],
    default: typing.Callable,
    sort_keys: bool,
    kw,
):

Properties

flytekit.types.directory.types.FlyteFile

def FlyteFile(
    path: typing.Union[str, os.PathLike],
    downloader: typing.Callable,
    remote_path: typing.Optional[typing.Union[os.PathLike, str, bool]],
    metadata: typing.Optional[dict[str, str]],
):

FlyteFile’s init method.

Methods

deserialize_flyte_file()

def deserialize_flyte_file(
    args,
    kwargs,
):

download()

def download()

extension()

def extension()

from_dict()

def from_dict(
    d,
    dialect,
):

from_json()

def from_json(
    data: typing.Union[str, bytes, bytearray],
    decoder: collections.abc.Callable[[typing.Union[str, bytes, bytearray]], dict[typing.Any, typing.Any]],
    from_dict_kwargs: typing.Any,
):

from_source()

def from_source(
    source: str | os.PathLike,
):

Create a new FlyteFile object with the remote source set to the input

new()

def new(
    filename: str | os.PathLike,
):

Create a new FlyteFile object in the current Flyte working directory

new_remote_file()

def new_remote_file(
    name: typing.Optional[str],
    alt: typing.Optional[str],
):

Create a new FlyteFile object with a remote path.

open()

def open(
    mode: str,
    cache_type: typing.Optional[str],
    cache_options: typing.Optional[typing.Dict[str, typing.Any]],
):

Returns a streaming File handle

.. code-block:: python

@task def copy_file(ff: FlyteFile) -> FlyteFile: new_file = FlyteFile.new_remote_file() with ff.open(“rb”, cache_type=“readahead”) as r: with new_file.open(“wb”) as w: w.write(r.read()) return new_file

serialize_flyte_file()

def serialize_flyte_file(
    args,
    kwargs,
):

to_dict()

def to_dict()

to_json()

def to_json(
    encoder: collections.abc.Callable[[typing.Any], typing.Union[str, bytes, bytearray]],
    to_dict_kwargs: typing.Any,
):

Properties

flytekit.types.directory.types.Literal

def Literal(
    scalar: typing.Optional[flytekit.models.literals.Scalar],
    collection: typing.Optional[flytekit.models.literals.LiteralCollection],
    map: typing.Optional[flytekit.models.literals.LiteralMap],
    hash: typing.Optional[str],
    metadata: typing.Optional[typing.Dict[str, str]],
    offloaded_metadata: typing.Optional[flytekit.models.literals.LiteralOffloadedMetadata],
):

This IDL message represents a literal value in the Flyte ecosystem.

Methods

from_flyte_idl()

def from_flyte_idl(
    pb2_object: flyteidl.core.literals_pb2.Literal,
):

serialize_to_string()

def serialize_to_string()

set_metadata()

def set_metadata(
    metadata: typing.Dict[str, str],
):

Note: This is a mutation on the literal

short_string()

def short_string()

to_flyte_idl()

def to_flyte_idl()

verbose_string()

def verbose_string()

Properties

flytekit.types.directory.types.LiteralType

def LiteralType(
    simple,
    schema,
    collection_type,
    map_value_type,
    blob,
    enum_type,
    union_type,
    structured_dataset_type,
    metadata,
    structure,
    annotation,
):

This is a oneof message, only one of the kwargs may be set, representing one of the Flyte types.

Methods

from_flyte_idl()

def from_flyte_idl(
    proto,
):

serialize_to_string()

def serialize_to_string()

short_string()

def short_string()

to_flyte_idl()

def to_flyte_idl()

verbose_string()

def verbose_string()

Properties

flytekit.types.directory.types.Path

PurePath subclass that can make system calls.

Path represents a filesystem path but unlike PurePath, also offers methods to do system calls on path objects. Depending on your system, instantiating a Path will return either a PosixPath or a WindowsPath object. You can also instantiate a PosixPath or WindowsPath directly, but cannot instantiate a WindowsPath on a POSIX system or vice versa.

def Path(
    args,
    kwargs,
):

Methods

absolute()

def absolute()

Return an absolute version of this path by prepending the current working directory. No normalization or symlink resolution is performed.

Use resolve() to get the canonical path to a file.

as_posix()

def as_posix()

Return the string representation of the path with forward (/) slashes.

as_uri()

def as_uri()

Return the path as a ‘file’ URI.

chmod()

def chmod(
    mode,
    follow_symlinks,
):

Change the permissions of the path, like os.chmod().

cwd()

def cwd()

Return a new path pointing to the current working directory.

exists()

def exists(
    follow_symlinks,
):

Whether this path exists.

This method normally follows symlinks; to check whether a symlink exists, add the argument follow_symlinks=False.

expanduser()

def expanduser()

Return a new path with expanded ~ and ~user constructs (as returned by os.path.expanduser)

glob()

def glob(
    pattern,
    case_sensitive,
):

Iterate over this subtree and yield all existing files (of any kind, including directories) matching the given relative pattern.

group()

def group()

Return the group name of the file gid.

hardlink_to()

def hardlink_to(
    target,
):

Make this path a hard link pointing to the same file as target.

Note the order of arguments (self, target) is the reverse of os.link’s.

home()

def home()

Return a new path pointing to the user’s home directory (as returned by os.path.expanduser(’~’)).

is_absolute()

def is_absolute()

True if the path is absolute (has both a root and, if applicable, a drive).

is_block_device()

def is_block_device()

Whether this path is a block device.

is_char_device()

def is_char_device()

Whether this path is a character device.

is_dir()

def is_dir()

Whether this path is a directory.

is_fifo()

def is_fifo()

Whether this path is a FIFO.

is_file()

def is_file()

Whether this path is a regular file (also True for symlinks pointing to regular files).

is_junction()

def is_junction()

Whether this path is a junction.

is_mount()

def is_mount()

Check if this path is a mount point

is_relative_to()

def is_relative_to(
    other,
    _deprecated,
):

Return True if the path is relative to another path or False.

is_reserved()

def is_reserved()

Return True if the path contains one of the special names reserved by the system, if any.

is_socket()

def is_socket()

Whether this path is a socket.

is_symlink()

def is_symlink()

Whether this path is a symbolic link.

iterdir()

def iterdir()

Yield path objects of the directory contents.

The children are yielded in arbitrary order, and the special entries ‘.’ and ‘..’ are not included.

joinpath()

def joinpath(
    pathsegments,
):

Combine this path with one or several arguments, and return a new path representing either a subpath (if all arguments are relative paths) or a totally different path (if one of the arguments is anchored).

lchmod()

def lchmod(
    mode,
):

Like chmod(), except if the path points to a symlink, the symlink’s permissions are changed, rather than its target’s.

lstat()

def lstat()

Like stat(), except if the path points to a symlink, the symlink’s status information is returned, rather than its target’s.

match()

def match(
    path_pattern,
    case_sensitive,
):

Return True if this path matches the given pattern.

mkdir()

def mkdir(
    mode,
    parents,
    exist_ok,
):

Create a new directory at this given path.

open()

def open(
    mode,
    buffering,
    encoding,
    errors,
    newline,
):

Open the file pointed to by this path and return a file object, as the built-in open() function does.

owner()

def owner()

Return the login name of the file owner.

read_bytes()

def read_bytes()

Open the file in bytes mode, read it, and close the file.

read_text()

def read_text(
    encoding,
    errors,
):

Open the file in text mode, read it, and close the file.

readlink()

def readlink()

Return the path to which the symbolic link points.

relative_to()

def relative_to(
    other,
    _deprecated,
    walk_up,
):

Return the relative path to another path identified by the passed arguments. If the operation is not possible (because this is not related to the other path), raise ValueError.

The walk_up parameter controls whether .. may be used to resolve the path.

rename()

def rename(
    target,
):

Rename this path to the target path.

The target path may be absolute or relative. Relative paths are interpreted relative to the current working directory, not the directory of the Path object.

Returns the new Path instance pointing to the target path.

replace()

def replace(
    target,
):

Rename this path to the target path, overwriting if that path exists.

The target path may be absolute or relative. Relative paths are interpreted relative to the current working directory, not the directory of the Path object.

Returns the new Path instance pointing to the target path.

resolve()

def resolve(
    strict,
):

Make the path absolute, resolving all symlinks on the way and also normalizing it.

rglob()

def rglob(
    pattern,
    case_sensitive,
):

Recursively yield all existing files (of any kind, including directories) matching the given relative pattern, anywhere in this subtree.

rmdir()

def rmdir()

Remove this directory. The directory must be empty.

samefile()

def samefile(
    other_path,
):

Return whether other_path is the same or not as this file (as returned by os.path.samefile()).

stat()

def stat(
    follow_symlinks,
):

Return the result of the stat() system call on this path, like os.stat() does.

symlink_to()

def symlink_to(
    target,
    target_is_directory,
):

Make this path a symlink pointing to the target path. Note the order of arguments (link, target) is the reverse of os.symlink.

touch()

def touch(
    mode,
    exist_ok,
):

Create this file with the given access mode, if it doesn’t exist.

unlink()

def unlink(
    missing_ok,
):

Remove this file or link. If the path is a directory, use rmdir() instead.

walk()

def walk(
    top_down,
    on_error,
    follow_symlinks,
):

Walk the directory tree from this directory, similar to os.walk().

with_name()

def with_name(
    name,
):

Return a new path with the file name changed.

with_segments()

def with_segments(
    pathsegments,
):

Construct a new path object from any number of path-like objects. Subclasses may override this method to customize how new path objects are created from methods like iterdir().

with_stem()

def with_stem(
    stem,
):

Return a new path with the stem changed.

with_suffix()

def with_suffix(
    suffix,
):

Return a new path with the file suffix changed. If the path has no suffix, add given suffix. If the given suffix is an empty string, remove the suffix from the path.

write_bytes()

def write_bytes(
    data,
):

Open the file in bytes mode, write to it, and close the file.

write_text()

def write_text(
    data,
    encoding,
    errors,
    newline,
):

Open the file in text mode, write to it, and close the file.

Properties

flytekit.types.directory.types.Scalar

def Scalar(
    primitive: typing.Optional[flytekit.models.literals.Primitive],
    blob: typing.Optional[flytekit.models.literals.Blob],
    binary: typing.Optional[flytekit.models.literals.Binary],
    schema: typing.Optional[flytekit.models.literals.Schema],
    union: typing.Optional[flytekit.models.literals.Union],
    none_type: typing.Optional[flytekit.models.literals.Void],
    error: typing.Optional[flytekit.models.types.Error],
    generic: typing.Optional[google.protobuf.struct_pb2.Struct],
    structured_dataset: typing.Optional[flytekit.models.literals.StructuredDataset],
):

Scalar wrapper around Flyte types. Only one can be specified.

Methods

from_flyte_idl()

def from_flyte_idl(
    pb2_object,
):

serialize_to_string()

def serialize_to_string()

short_string()

def short_string()

to_flyte_idl()

def to_flyte_idl()

verbose_string()

def verbose_string()

Properties

flytekit.types.directory.types.SerializableType

flytekit.types.directory.types.Struct

A ProtocolMessage

Methods

get_or_create_list()

def get_or_create_list(
    key,
):

Returns a list for this key, creating if it didn’t exist already.

get_or_create_struct()

def get_or_create_struct(
    key,
):

Returns a struct for this key, creating if it didn’t exist already.

items()

def items()

keys()

def keys()

update()

def update(
    dictionary,
):

values()

def values()

flytekit.types.directory.types.TypeEngine

Core Extensible TypeEngine of Flytekit. This should be used to extend the capabilities of FlyteKits type system. Users can implement their own TypeTransformers and register them with the TypeEngine. This will allow special handling of user objects

Methods

async_to_literal()

def async_to_literal(
    ctx: FlyteContext,
    python_val: typing.Any,
    python_type: Type[T],
    expected: LiteralType,
):

Converts a python value of a given type and expected LiteralType into a resolved Literal value.

async_to_python_value()

def async_to_python_value(
    ctx: FlyteContext,
    lv: Literal,
    expected_python_type: Type,
):

calculate_hash()

def calculate_hash(
    python_val: typing.Any,
    python_type: Type[T],
):

dict_to_literal_map()

def dict_to_literal_map(
    ctx: FlyteContext,
    d: typing.Dict[str, typing.Any],
    type_hints: Optional[typing.Dict[str, type]],
):

dict_to_literal_map_pb()

def dict_to_literal_map_pb(
    ctx: FlyteContext,
    d: typing.Dict[str, typing.Any],
    type_hints: Optional[typing.Dict[str, type]],
):

get_available_transformers()

def get_available_transformers()

Returns all python types for which transformers are available

get_transformer()

def get_transformer(
    python_type: Type,
):

Implements a recursive search for the transformer.

guess_python_type()

def guess_python_type(
    flyte_type: LiteralType,
):

Transforms a flyte-specific LiteralType to a regular python value.

guess_python_types()

def guess_python_types(
    flyte_variable_dict: typing.Dict[str, _interface_models.Variable],
):

Transforms a dictionary of flyte-specific Variable objects to a dictionary of regular python values.

lazy_import_transformers()

def lazy_import_transformers()

Only load the transformers if needed.

literal_map_to_kwargs()

def literal_map_to_kwargs(
    ctx: FlyteContext,
    lm: LiteralMap,
    python_types: typing.Optional[typing.Dict[str, type]],
    literal_types: typing.Optional[typing.Dict[str, _interface_models.Variable]],
):

named_tuple_to_variable_map()

def named_tuple_to_variable_map(
    t: typing.NamedTuple,
):

Converts a python-native NamedTuple to a flyte-specific VariableMap of named literals.

register()

def register(
    transformer: TypeTransformer,
    additional_types: Optional[typing.List[Type]],
):

This should be used for all types that respond with the right type annotation when you use type(…) function

register_additional_type()

def register_additional_type(
    transformer: TypeTransformer[T],
    additional_type: Type[T],
    override,
):

register_restricted_type()

def register_restricted_type(
    name: str,
    type: Type[T],
):

to_html()

def to_html(
    ctx: FlyteContext,
    python_val: typing.Any,
    expected_python_type: Type[typing.Any],
):

to_literal()

def to_literal(
    ctx: FlyteContext,
    python_val: typing.Any,
    python_type: Type[T],
    expected: LiteralType,
):

The current dance is because we are allowing users to call from an async function, this synchronous to_literal function, and allowing this to_literal function, to then invoke yet another async function, namely an async transformer.

to_literal_checks()

def to_literal_checks(
    python_val: typing.Any,
    python_type: Type[T],
    expected: LiteralType,
):

to_literal_type()

def to_literal_type(
    python_type: Type[T],
):

Converts a python type into a flyte specific LiteralType

to_python_value()

def to_python_value(
    ctx: FlyteContext,
    lv: Literal,
    expected_python_type: Type,
):

Converts a Literal value with an expected python type into a python value.

unwrap_offloaded_literal()

def unwrap_offloaded_literal(
    ctx: FlyteContext,
    lv: Literal,
):

flytekit.types.directory.types.TypeTransformerFailedError

Inappropriate argument type.

flytekit.types.directory.types.UUID

Instances of the UUID class represent UUIDs as specified in RFC 4122. UUID objects are immutable, hashable, and usable as dictionary keys. Converting a UUID to a string with str() yields something in the form ‘12345678-1234-1234-1234-123456789abc’. The UUID constructor accepts five possible forms: a similar string of hexadecimal digits, or a tuple of six integer fields (with 32-bit, 16-bit, 16-bit, 8-bit, 8-bit, and 48-bit values respectively) as an argument named ‘fields’, or a string of 16 bytes (with all the integer fields in big-endian order) as an argument named ‘bytes’, or a string of 16 bytes (with the first three fields in little-endian order) as an argument named ‘bytes_le’, or a single 128-bit integer as an argument named ‘int’.

UUIDs have these read-only attributes:

bytes the UUID as a 16-byte string (containing the six integer fields in big-endian byte order)

bytes_le the UUID as a 16-byte string (with time_low, time_mid, and time_hi_version in little-endian byte order)

fields a tuple of the six integer fields of the UUID, which are also available as six individual attributes and two derived attributes:

time_low the first 32 bits of the UUID time_mid the next 16 bits of the UUID time_hi_version the next 16 bits of the UUID clock_seq_hi_variant the next 8 bits of the UUID clock_seq_low the next 8 bits of the UUID node the last 48 bits of the UUID

time the 60-bit timestamp clock_seq the 14-bit sequence number

hex the UUID as a 32-character hexadecimal string

int the UUID as a 128-bit integer

urn the UUID as a URN as specified in RFC 4122

variant the UUID variant (one of the constants RESERVED_NCS, RFC_4122, RESERVED_MICROSOFT, or RESERVED_FUTURE)

version the UUID version number (1 through 5, meaningful only when the variant is RFC_4122)

is_safe An enum indicating whether the UUID has been generated in a way that is safe for multiprocessing applications, via uuid_generate_time_safe(3).

def UUID(
    hex,
    bytes,
    bytes_le,
    fields,
    int,
    version,
    is_safe,
):

Create a UUID from either a string of 32 hexadecimal digits, a string of 16 bytes as the ‘bytes’ argument, a string of 16 bytes in little-endian order as the ‘bytes_le’ argument, a tuple of six integers (32-bit time_low, 16-bit time_mid, 16-bit time_hi_version, 8-bit clock_seq_hi_variant, 8-bit clock_seq_low, 48-bit node) as the ‘fields’ argument, or a single 128-bit integer as the ‘int’ argument. When a string of hex digits is given, curly braces, hyphens, and a URN prefix are all optional. For example, these expressions all yield the same UUID:

UUID(’{12345678-1234-5678-1234-567812345678}’) UUID(‘12345678123456781234567812345678’) UUID(‘urn:uuid:12345678-1234-5678-1234-567812345678’) UUID(bytes=’\x12\x34\x56\x78’*4) UUID(bytes_le=’\x78\x56\x34\x12\x34\x12\x78\x56’ + ‘\x12\x34\x56\x78\x12\x34\x56\x78’) UUID(fields=(0x12345678, 0x1234, 0x5678, 0x12, 0x34, 0x567812345678)) UUID(int=0x12345678123456781234567812345678)

Exactly one of ‘hex’, ‘bytes’, ‘bytes_le’, ‘fields’, or ‘int’ must be given. The ‘version’ argument is optional; if given, the resulting UUID will have its variant and version set according to RFC 4122, overriding the given ‘hex’, ‘bytes’, ‘bytes_le’, ‘fields’, or ‘int’.

is_safe is an enum exposed as an attribute on the instance. It indicates whether the UUID has been generated in a way that is safe for multiprocessing applications, via uuid_generate_time_safe(3).

Properties