flytekit.core.promise
Directory
Classes
Class | Description |
---|---|
Annotated |
Add context-specific metadata to a type. |
Any |
Special type indicating an unconstrained type. |
AsyncTypeTransformer |
Base transformer type that should be implemented for every python native type that can be handled by flytekit. |
BaseAccelerator |
Base class for all accelerator types. |
Binary |
None. |
BranchEvalMode |
This is a 3-way class, with the None value meaning that we are not within a conditional context. |
ComparisonExpression |
ComparisonExpression refers to an expression of the form (lhs operator rhs), where lhs and rhs are operands. |
ComparisonOps |
Create a collection of name/value pairs. |
ConjunctionExpression |
A Conjunction Expression is an expression of the form either (A and B) or (A or B). |
ConjunctionOps |
Create a collection of name/value pairs. |
DictTransformer |
Transformer that transforms an univariate dictionary Dict[str, T] to a Literal Map or. |
Enum |
Create a collection of name/value pairs. |
ExecutionParameters |
This is a run-time user-centric context object that is accessible to every @task method. |
ExecutionState |
This is the context that is active when executing a task or a local workflow. |
FlyteContext |
This is an internal-facing context object, that most users will not have to deal with. |
FlyteContextManager |
FlyteContextManager manages the execution context within Flytekit. |
HasFlyteInterface |
Base class for protocol classes. |
Interface |
A Python native interface object, like inspect. |
Iterable |
None. |
ListTransformer |
Transformer that handles a univariate typing. |
Literal |
None. |
LocallyExecutable |
Base class for protocol classes. |
Node |
This class will hold all the things necessary to make an SdkNode but we won’t make one until we know things like. |
NodeOutput |
None. |
OutputMetadataTracker |
This class is for the users to set arbitrary metadata on output literals. |
Primitive |
None. |
Promise |
This object is a wrapper and exists for three main reasons. |
Protocol |
Base class for protocol classes. |
Resources |
None. |
Scalar |
None. |
SimpleType |
None. |
SupportsNodeCreation |
Base class for protocol classes. |
TypeEngine |
Core Extensible TypeEngine of Flytekit. |
TypeTransformer |
Base transformer type that should be implemented for every python native type that can be handled by flytekit. |
UnionTransformer |
Transformer that handles a typing. |
VoidPromise |
This object is returned for tasks that do not return any outputs (declared interface is empty). |
Errors
flytekit.core.promise.Annotated
Add context-specific metadata to a type.
Example: Annotated[int, runtime_check.Unsigned] indicates to the hypothetical runtime_check module that this type is an unsigned int. Every other consumer of this type can ignore this metadata and treat this type as int.
The first argument to Annotated must be a valid type.
Details:
- It’s an error to call
Annotated
with less than two arguments. - Access the metadata via the
__metadata__
attribute::
assert Annotated[int, ‘$’].metadata == (’$’,)
- Nested Annotated types are flattened::
assert Annotated[Annotated[T, Ann1, Ann2], Ann3] == Annotated[T, Ann1, Ann2, Ann3]
- Instantiating an annotated type is equivalent to instantiating the underlying type::
assert AnnotatedC, Ann1 == C(5)
- Annotated can be used as a generic type alias::
type Optimized[T] = Annotated[T, runtime.Optimize()]
type checker will treat Optimized[int]
as equivalent to Annotated[int, runtime.Optimize()]
type OptimizedList[T] = Annotated[list[T], runtime.Optimize()]
type checker will treat OptimizedList[int]
as equivalent to Annotated[list[int], runtime.Optimize()]
- Annotated cannot be used with an unpacked TypeVarTuple::
type Variadic[*Ts] = Annotated[*Ts, Ann1] # NOT valid
This would be equivalent to::
Annotated[T1, T2, T3, …, Ann1]
where T1, T2 etc. are TypeVars, which would be invalid, because only one type should be passed to Annotated.
flytekit.core.promise.Any
Special type indicating an unconstrained type.
- Any is compatible with every type.
- Any assumed to have all methods.
- All values assumed to be instances of Any.
Note that all the above statements are true from the point of view of static type checkers. At runtime, Any should not be used with instance checks.
flytekit.core.promise.AsyncTypeTransformer
Base transformer type that should be implemented for every python native type that can be handled by flytekit
def AsyncTypeTransformer(
name: str,
t: Type[T],
enable_type_assertions: bool,
):
Parameter | Type |
---|---|
name |
str |
t |
Type[T] |
enable_type_assertions |
bool |
Methods
Method | Description |
---|---|
assert_type() |
None |
async_to_literal() |
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type |
async_to_python_value() |
Converts the given Literal to a Python Type |
from_binary_idl() |
This function primarily handles deserialization for untyped dicts, dataclasses, Pydantic BaseModels, and attribute access |
from_generic_idl() |
TODO: Support all Flyte Types |
get_literal_type() |
Converts the python type to a Flyte LiteralType |
guess_python_type() |
Converts the Flyte LiteralType to a python object type |
isinstance_generic() |
None |
to_html() |
Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div |
to_literal() |
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type |
to_python_value() |
Converts the given Literal to a Python Type |
assert_type()
def assert_type(
t: Type[T],
v: T,
):
Parameter | Type |
---|---|
t |
Type[T] |
v |
T |
async_to_literal()
def async_to_literal(
ctx: FlyteContext,
python_val: T,
python_type: Type[T],
expected: LiteralType,
):
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
T |
python_type |
Type[T] |
expected |
LiteralType |
async_to_python_value()
def async_to_python_value(
ctx: FlyteContext,
lv: Literal,
expected_python_type: Type[T],
):
Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised
Parameter | Type |
---|---|
ctx |
FlyteContext |
lv |
Literal |
expected_python_type |
Type[T] |
from_binary_idl()
def from_binary_idl(
binary_idl_object: Binary,
expected_python_type: Type[T],
):
This function primarily handles deserialization for untyped dicts, dataclasses, Pydantic BaseModels, and attribute access.`
For untyped dict, dataclass, and pydantic basemodel: Life Cycle (Untyped Dict as example): python val -> msgpack bytes -> binary literal scalar -> msgpack bytes -> python val (to_literal) (from_binary_idl)
For attribute access: Life Cycle: python val -> msgpack bytes -> binary literal scalar -> resolved golang value -> binary literal scalar -> msgpack bytes -> python val (to_literal) (propeller attribute access) (from_binary_idl)
Parameter | Type |
---|---|
binary_idl_object |
Binary |
expected_python_type |
Type[T] |
from_generic_idl()
def from_generic_idl(
generic: Struct,
expected_python_type: Type[T],
):
TODO: Support all Flyte Types. This is for dataclass attribute access from input created from the Flyte Console.
Note:
- This can be removed in the future when the Flyte Console support generate Binary IDL Scalar as input.
Parameter | Type |
---|---|
generic |
Struct |
expected_python_type |
Type[T] |
get_literal_type()
def get_literal_type(
t: Type[T],
):
Converts the python type to a Flyte LiteralType
Parameter | Type |
---|---|
t |
Type[T] |
guess_python_type()
def guess_python_type(
literal_type: LiteralType,
):
Converts the Flyte LiteralType to a python object type.
Parameter | Type |
---|---|
literal_type |
LiteralType |
isinstance_generic()
def isinstance_generic(
obj,
generic_alias,
):
Parameter | Type |
---|---|
obj |
|
generic_alias |
to_html()
def to_html(
ctx: FlyteContext,
python_val: T,
expected_python_type: Type[T],
):
Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
T |
expected_python_type |
Type[T] |
to_literal()
def to_literal(
ctx: FlyteContext,
python_val: typing.Any,
python_type: Type[T],
expected: LiteralType,
):
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
typing.Any |
python_type |
Type[T] |
expected |
LiteralType |
to_python_value()
def to_python_value(
ctx: FlyteContext,
lv: Literal,
expected_python_type: Type[T],
):
Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised
Parameter | Type |
---|---|
ctx |
FlyteContext |
lv |
Literal |
expected_python_type |
Type[T] |
Properties
Property | Type | Description |
---|---|---|
is_async | ||
name | ||
python_type | ||
type_assertions_enabled |
flytekit.core.promise.BaseAccelerator
Base class for all accelerator types. This class is not meant to be instantiated directly.
Methods
Method | Description |
---|---|
to_flyte_idl() |
None |
to_flyte_idl()
def to_flyte_idl()
flytekit.core.promise.Binary
def Binary(
value,
tag,
):
Parameter | Type |
---|---|
value |
|
tag |
Methods
Method | Description |
---|---|
from_flyte_idl() |
|
serialize_to_string() |
None |
short_string() |
|
to_flyte_idl() |
|
verbose_string() |
from_flyte_idl()
def from_flyte_idl(
pb2_object,
):
Parameter | Type |
---|---|
pb2_object |
serialize_to_string()
def serialize_to_string()
short_string()
def short_string()
to_flyte_idl()
def to_flyte_idl()
verbose_string()
def verbose_string()
Properties
Property | Type | Description |
---|---|---|
is_empty | ||
tag | ||
value |
flytekit.core.promise.BranchEvalMode
This is a 3-way class, with the None value meaning that we are not within a conditional context. The other two values are
- Active - This means that the next
then
should run - Skipped - The next
then
should not run
flytekit.core.promise.ComparisonExpression
ComparisonExpression refers to an expression of the form (lhs operator rhs), where lhs and rhs are operands and operator can be any comparison expression like <, >, <=, >=, ==, !=
def ComparisonExpression(
lhs: Union['Promise', Any],
op: ComparisonOps,
rhs: Union['Promise', Any],
):
Parameter | Type |
---|---|
lhs |
Union['Promise', Any] |
op |
ComparisonOps |
rhs |
Union['Promise', Any] |
Methods
Method | Description |
---|---|
eval() |
None |
eval()
def eval()
Properties
Property | Type | Description |
---|---|---|
lhs | ||
op | ||
rhs |
flytekit.core.promise.ComparisonOps
Create a collection of name/value pairs.
Example enumeration:
class Color(Enum): … RED = 1 … BLUE = 2 … GREEN = 3
Access them by:
- attribute access:
Color.RED <Color.RED: 1>
- value lookup:
Color(1) <Color.RED: 1>
- name lookup:
Color[‘RED’] <Color.RED: 1>
Enumerations can be iterated over, and know how many members they have:
len(Color) 3
list(Color) [<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]
Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.
flytekit.core.promise.ConjunctionExpression
A Conjunction Expression is an expression of the form either (A and B) or (A or B). where A, B are two expressions (comparison or conjunctions) and (and, or) are logical truth operators.
A conjunctionExpression evaluates to True or False depending on the logical operator and the truth values of each of the expressions A & B
def ConjunctionExpression(
lhs: Union[ComparisonExpression, 'ConjunctionExpression'],
op: ConjunctionOps,
rhs: Union[ComparisonExpression, 'ConjunctionExpression'],
):
Parameter | Type |
---|---|
lhs |
Union[ComparisonExpression, 'ConjunctionExpression'] |
op |
ConjunctionOps |
rhs |
Union[ComparisonExpression, 'ConjunctionExpression'] |
Methods
Method | Description |
---|---|
eval() |
None |
eval()
def eval()
Properties
Property | Type | Description |
---|---|---|
lhs | ||
op | ||
rhs |
flytekit.core.promise.ConjunctionOps
Create a collection of name/value pairs.
Example enumeration:
class Color(Enum): … RED = 1 … BLUE = 2 … GREEN = 3
Access them by:
- attribute access:
Color.RED <Color.RED: 1>
- value lookup:
Color(1) <Color.RED: 1>
- name lookup:
Color[‘RED’] <Color.RED: 1>
Enumerations can be iterated over, and know how many members they have:
len(Color) 3
list(Color) [<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]
Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.
flytekit.core.promise.DictTransformer
Transformer that transforms an univariate dictionary Dict[str, T] to a Literal Map or transforms an untyped dictionary to a Binary Scalar Literal with a Struct Literal Type.
def DictTransformer()
Methods
Method | Description |
---|---|
assert_type() |
None |
async_to_literal() |
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type |
async_to_python_value() |
Converts the given Literal to a Python Type |
dict_to_binary_literal() |
Converts a Python dictionary to a Flyte-specific Literal using MessagePack encoding |
dict_to_generic_literal() |
This is deprecated from flytekit 1 |
extract_types() |
None |
from_binary_idl() |
This function primarily handles deserialization for untyped dicts, dataclasses, Pydantic BaseModels, and attribute access |
from_generic_idl() |
TODO: Support all Flyte Types |
get_literal_type() |
Transforms a native python dictionary to a flyte-specific LiteralType |
guess_python_type() |
Converts the Flyte LiteralType to a python object type |
is_pickle() |
None |
isinstance_generic() |
None |
to_html() |
Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div |
to_literal() |
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type |
to_python_value() |
Converts the given Literal to a Python Type |
assert_type()
def assert_type(
t: Type[T],
v: T,
):
Parameter | Type |
---|---|
t |
Type[T] |
v |
T |
async_to_literal()
def async_to_literal(
ctx: FlyteContext,
python_val: typing.Any,
python_type: Type[dict],
expected: LiteralType,
):
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
typing.Any |
python_type |
Type[dict] |
expected |
LiteralType |
async_to_python_value()
def async_to_python_value(
ctx: FlyteContext,
lv: Literal,
expected_python_type: Type[dict],
):
Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised
Parameter | Type |
---|---|
ctx |
FlyteContext |
lv |
Literal |
expected_python_type |
Type[dict] |
dict_to_binary_literal()
def dict_to_binary_literal(
ctx: FlyteContext,
v: dict,
python_type: Type[dict],
allow_pickle: bool,
):
Converts a Python dictionary to a Flyte-specific Literal
using MessagePack encoding.
Falls back to Pickle if encoding fails and allow_pickle
is True.
Parameter | Type |
---|---|
ctx |
FlyteContext |
v |
dict |
python_type |
Type[dict] |
allow_pickle |
bool |
dict_to_generic_literal()
def dict_to_generic_literal(
ctx: FlyteContext,
v: dict,
python_type: Type[dict],
allow_pickle: bool,
):
This is deprecated from flytekit 1.14.0.
Creates a flyte-specific Literal
value from a native python dictionary.
Note: This is deprecated and will be removed in the future.
Parameter | Type |
---|---|
ctx |
FlyteContext |
v |
dict |
python_type |
Type[dict] |
allow_pickle |
bool |
extract_types()
def extract_types(
t: Optional[Type[dict]],
):
Parameter | Type |
---|---|
t |
Optional[Type[dict]] |
from_binary_idl()
def from_binary_idl(
binary_idl_object: Binary,
expected_python_type: Type[T],
):
This function primarily handles deserialization for untyped dicts, dataclasses, Pydantic BaseModels, and attribute access.`
For untyped dict, dataclass, and pydantic basemodel: Life Cycle (Untyped Dict as example): python val -> msgpack bytes -> binary literal scalar -> msgpack bytes -> python val (to_literal) (from_binary_idl)
For attribute access: Life Cycle: python val -> msgpack bytes -> binary literal scalar -> resolved golang value -> binary literal scalar -> msgpack bytes -> python val (to_literal) (propeller attribute access) (from_binary_idl)
Parameter | Type |
---|---|
binary_idl_object |
Binary |
expected_python_type |
Type[T] |
from_generic_idl()
def from_generic_idl(
generic: Struct,
expected_python_type: Type[T],
):
TODO: Support all Flyte Types. This is for dataclass attribute access from input created from the Flyte Console.
Note:
- This can be removed in the future when the Flyte Console support generate Binary IDL Scalar as input.
Parameter | Type |
---|---|
generic |
Struct |
expected_python_type |
Type[T] |
get_literal_type()
def get_literal_type(
t: Type[dict],
):
Transforms a native python dictionary to a flyte-specific LiteralType
Parameter | Type |
---|---|
t |
Type[dict] |
guess_python_type()
def guess_python_type(
literal_type: LiteralType,
):
Converts the Flyte LiteralType to a python object type.
Parameter | Type |
---|---|
literal_type |
LiteralType |
is_pickle()
def is_pickle(
python_type: Type[dict],
):
Parameter | Type |
---|---|
python_type |
Type[dict] |
isinstance_generic()
def isinstance_generic(
obj,
generic_alias,
):
Parameter | Type |
---|---|
obj |
|
generic_alias |
to_html()
def to_html(
ctx: FlyteContext,
python_val: T,
expected_python_type: Type[T],
):
Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
T |
expected_python_type |
Type[T] |
to_literal()
def to_literal(
ctx: FlyteContext,
python_val: typing.Any,
python_type: Type[T],
expected: LiteralType,
):
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
typing.Any |
python_type |
Type[T] |
expected |
LiteralType |
to_python_value()
def to_python_value(
ctx: FlyteContext,
lv: Literal,
expected_python_type: Type[T],
):
Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised
Parameter | Type |
---|---|
ctx |
FlyteContext |
lv |
Literal |
expected_python_type |
Type[T] |
Properties
Property | Type | Description |
---|---|---|
is_async | ||
name | ||
python_type | ||
type_assertions_enabled |
flytekit.core.promise.Enum
Create a collection of name/value pairs.
Example enumeration:
class Color(Enum): … RED = 1 … BLUE = 2 … GREEN = 3
Access them by:
- attribute access:
Color.RED <Color.RED: 1>
- value lookup:
Color(1) <Color.RED: 1>
- name lookup:
Color[‘RED’] <Color.RED: 1>
Enumerations can be iterated over, and know how many members they have:
len(Color) 3
list(Color) [<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]
Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.
flytekit.core.promise.ExecutionParameters
This is a run-time user-centric context object that is accessible to every @task method. It can be accessed using
.. code-block:: python
flytekit.current_context()
This object provides the following
- a statsd handler
- a logging handler
- the execution ID as an :py:class:
flytekit.models.core.identifier.WorkflowExecutionIdentifier
object - a working directory for the user to write arbitrary files to
Please do not confuse this object with the :py:class:flytekit.FlyteContext
object.
def ExecutionParameters(
execution_date,
tmp_dir,
stats,
execution_id: typing.Optional[_identifier.WorkflowExecutionIdentifier],
logging,
raw_output_prefix,
output_metadata_prefix,
checkpoint,
decks,
task_id: typing.Optional[_identifier.Identifier],
enable_deck: bool,
kwargs,
):
Parameter | Type |
---|---|
execution_date |
|
tmp_dir |
|
stats |
|
execution_id |
typing.Optional[_identifier.WorkflowExecutionIdentifier] |
logging |
|
raw_output_prefix |
|
output_metadata_prefix |
|
checkpoint |
|
decks |
|
task_id |
typing.Optional[_identifier.Identifier] |
enable_deck |
bool |
kwargs |
**kwargs |
Methods
Method | Description |
---|---|
builder() |
None |
get() |
Returns task specific context if present else raise an error |
has_attr() |
None |
new_builder() |
None |
with_enable_deck() |
None |
with_task_sandbox() |
None |
builder()
def builder()
get()
def get(
key: str,
):
Returns task specific context if present else raise an error. The returned context will match the key
Parameter | Type |
---|---|
key |
str |
has_attr()
def has_attr(
attr_name: str,
):
Parameter | Type |
---|---|
attr_name |
str |
new_builder()
def new_builder(
current: Optional[ExecutionParameters],
):
Parameter | Type |
---|---|
current |
Optional[ExecutionParameters] |
with_enable_deck()
def with_enable_deck(
enable_deck: bool,
):
Parameter | Type |
---|---|
enable_deck |
bool |
with_task_sandbox()
def with_task_sandbox()
Properties
Property | Type | Description |
---|---|---|
checkpoint | ||
decks | ||
default_deck | ||
enable_deck | ||
execution_date | ||
execution_id | ||
logging | ||
output_metadata_prefix | ||
raw_output_prefix | ||
secrets | ||
stats | ||
task_id | ||
timeline_deck | ||
working_directory |
flytekit.core.promise.ExecutionState
This is the context that is active when executing a task or a local workflow. This carries the necessary state to execute. Some required things during execution deal with temporary directories, ExecutionParameters that are passed to the user etc.
Attributes: mode (ExecutionState.Mode): Defines the context in which the task is executed (local, hosted, etc). working_dir (os.PathLike): Specifies the remote, external directory where inputs, outputs and other protobufs are uploaded engine_dir (os.PathLike): branch_eval_mode Optional[BranchEvalMode]: Used to determine whether a branch node should execute. user_space_params Optional[ExecutionParameters]: Provides run-time, user-centric context such as a statsd handler, a logging handler, the current execution id and a working directory.
def ExecutionState(
working_dir: Union[os.PathLike, str],
mode: Optional[ExecutionState.Mode],
engine_dir: Optional[Union[os.PathLike, str]],
branch_eval_mode: Optional[BranchEvalMode],
user_space_params: Optional[ExecutionParameters],
):
Parameter | Type |
---|---|
working_dir |
Union[os.PathLike, str] |
mode |
Optional[ExecutionState.Mode] |
engine_dir |
Optional[Union[os.PathLike, str]] |
branch_eval_mode |
Optional[BranchEvalMode] |
user_space_params |
Optional[ExecutionParameters] |
Methods
Method | Description |
---|---|
branch_complete() |
Indicates that we are within a conditional / ifelse block and the active branch is not done |
is_local_execution() |
None |
take_branch() |
Indicates that we are within an if-else block and the current branch has evaluated to true |
with_params() |
Produces a copy of the current execution state and overrides the copy’s parameters with passed parameter values |
branch_complete()
def branch_complete()
Indicates that we are within a conditional / ifelse block and the active branch is not done. Default to SKIPPED
is_local_execution()
def is_local_execution()
take_branch()
def take_branch()
Indicates that we are within an if-else block and the current branch has evaluated to true. Useful only in local execution mode
with_params()
def with_params(
working_dir: Optional[os.PathLike],
mode: Optional[Mode],
engine_dir: Optional[os.PathLike],
branch_eval_mode: Optional[BranchEvalMode],
user_space_params: Optional[ExecutionParameters],
):
Produces a copy of the current execution state and overrides the copy’s parameters with passed parameter values.
Parameter | Type |
---|---|
working_dir |
Optional[os.PathLike] |
mode |
Optional[Mode] |
engine_dir |
Optional[os.PathLike] |
branch_eval_mode |
Optional[BranchEvalMode] |
user_space_params |
Optional[ExecutionParameters] |
flytekit.core.promise.FlyteContext
This is an internal-facing context object, that most users will not have to deal with. It’s essentially a globally available grab bag of settings and objects that allows flytekit to do things like convert complex types, run and compile workflows, serialize Flyte entities, etc.
Even though this object as a current_context
function on it, it should not be called directly. Please use the
:py:class:flytekit.FlyteContextManager
object instead.
Please do not confuse this object with the :py:class:flytekit.ExecutionParameters
object.
def FlyteContext(
file_access: FileAccessProvider,
level: int,
flyte_client: Optional['friendly_client.SynchronousFlyteClient'],
compilation_state: Optional[CompilationState],
execution_state: Optional[ExecutionState],
serialization_settings: Optional[SerializationSettings],
in_a_condition: bool,
origin_stackframe: Optional[traceback.FrameSummary],
output_metadata_tracker: Optional[OutputMetadataTracker],
worker_queue: Optional[Controller],
):
Parameter | Type |
---|---|
file_access |
FileAccessProvider |
level |
int |
flyte_client |
Optional['friendly_client.SynchronousFlyteClient'] |
compilation_state |
Optional[CompilationState] |
execution_state |
Optional[ExecutionState] |
serialization_settings |
Optional[SerializationSettings] |
in_a_condition |
bool |
origin_stackframe |
Optional[traceback.FrameSummary] |
output_metadata_tracker |
Optional[OutputMetadataTracker] |
worker_queue |
Optional[Controller] |
Methods
Method | Description |
---|---|
current_context() |
This method exists only to maintain backwards compatibility |
enter_conditional_section() |
None |
get_deck() |
Returns the deck that was created as part of the last execution |
get_origin_stackframe_repr() |
None |
new_builder() |
None |
new_compilation_state() |
Creates and returns a default compilation state |
new_execution_state() |
Creates and returns a new default execution state |
set_stackframe() |
None |
with_client() |
None |
with_compilation_state() |
None |
with_execution_state() |
None |
with_file_access() |
None |
with_new_compilation_state() |
None |
with_output_metadata_tracker() |
None |
with_serialization_settings() |
None |
with_worker_queue() |
None |
current_context()
def current_context()
This method exists only to maintain backwards compatibility. Please use
FlyteContextManager.current_context()
instead.
Users of flytekit should be wary not to confuse the object returned from this function
with :py:func:flytekit.current_context
enter_conditional_section()
def enter_conditional_section()
get_deck()
def get_deck()
Returns the deck that was created as part of the last execution.
The return value depends on the execution environment. In a notebook, the return value is compatible with IPython.display and should be rendered in the notebook.
.. code-block:: python
with flytekit.new_context() as ctx: my_task(…) ctx.get_deck()
OR if you wish to explicitly display
.. code-block:: python
from IPython import display display(ctx.get_deck())
get_origin_stackframe_repr()
def get_origin_stackframe_repr()
new_builder()
def new_builder()
new_compilation_state()
def new_compilation_state(
prefix: str,
):
Creates and returns a default compilation state. For most of the code this should be the entrypoint of compilation, otherwise the code should always uses - with_compilation_state
Parameter | Type |
---|---|
prefix |
str |
new_execution_state()
def new_execution_state(
working_dir: Optional[os.PathLike],
):
Creates and returns a new default execution state. This should be used at the entrypoint of execution, in all other cases it is preferable to use with_execution_state
Parameter | Type |
---|---|
working_dir |
Optional[os.PathLike] |
set_stackframe()
def set_stackframe(
s: traceback.FrameSummary,
):
Parameter | Type |
---|---|
s |
traceback.FrameSummary |
with_client()
def with_client(
c: SynchronousFlyteClient,
):
Parameter | Type |
---|---|
c |
SynchronousFlyteClient |
with_compilation_state()
def with_compilation_state(
c: CompilationState,
):
Parameter | Type |
---|---|
c |
CompilationState |
with_execution_state()
def with_execution_state(
es: ExecutionState,
):
Parameter | Type |
---|---|
es |
ExecutionState |
with_file_access()
def with_file_access(
fa: FileAccessProvider,
):
Parameter | Type |
---|---|
fa |
FileAccessProvider |
with_new_compilation_state()
def with_new_compilation_state()
with_output_metadata_tracker()
def with_output_metadata_tracker(
t: OutputMetadataTracker,
):
Parameter | Type |
---|---|
t |
OutputMetadataTracker |
with_serialization_settings()
def with_serialization_settings(
ss: SerializationSettings,
):
Parameter | Type |
---|---|
ss |
SerializationSettings |
with_worker_queue()
def with_worker_queue(
wq: Controller,
):
Parameter | Type |
---|---|
wq |
Controller |
Properties
Property | Type | Description |
---|---|---|
user_space_params |
flytekit.core.promise.FlyteContextManager
FlyteContextManager manages the execution context within Flytekit. It holds global state of either compilation
or Execution. It is not thread-safe and can only be run as a single threaded application currently.
Context’s within Flytekit is useful to manage compilation state and execution state. Refer to CompilationState
and ExecutionState
for more information. FlyteContextManager provides a singleton stack to manage these contexts.
Typical usage is
.. code-block:: python
FlyteContextManager.initialize() with FlyteContextManager.with_context(o) as ctx: pass
If required - not recommended you can use
FlyteContextManager.push_context()
but correspondingly a pop_context should be called
FlyteContextManager.pop_context()
Methods
Method | Description |
---|---|
add_signal_handler() |
None |
current_context() |
None |
get_origin_stackframe() |
None |
initialize() |
Re-initializes the context and erases the entire context |
pop_context() |
None |
push_context() |
None |
size() |
None |
with_context() |
None |
add_signal_handler()
def add_signal_handler(
handler: typing.Callable[[int, FrameType], typing.Any],
):
Parameter | Type |
---|---|
handler |
typing.Callable[[int, FrameType], typing.Any] |
current_context()
def current_context()
get_origin_stackframe()
def get_origin_stackframe(
limit,
):
Parameter | Type |
---|---|
limit |
initialize()
def initialize()
Re-initializes the context and erases the entire context
pop_context()
def pop_context()
push_context()
def push_context(
ctx: FlyteContext,
f: Optional[traceback.FrameSummary],
):
Parameter | Type |
---|---|
ctx |
FlyteContext |
f |
Optional[traceback.FrameSummary] |
size()
def size()
with_context()
def with_context(
b: FlyteContext.Builder,
):
Parameter | Type |
---|---|
b |
FlyteContext.Builder |
flytekit.core.promise.FlytePromiseAttributeResolveException
Assertion failed.
def FlytePromiseAttributeResolveException(
args,
timestamp: typing.Optional[float],
):
Parameter | Type |
---|---|
args |
*args |
timestamp |
typing.Optional[float] |
Properties
Property | Type | Description |
---|---|---|
timestamp |
flytekit.core.promise.HasFlyteInterface
Base class for protocol classes.
Protocol classes are defined as::
class Proto(Protocol): def meth(self) -> int: …
Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing).
For example::
class C: def meth(self) -> int: return 0
def func(x: Proto) -> int: return x.meth()
func(C()) # Passes static type check
See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as::
class GenProtoT: def meth(self) -> T: …
def HasFlyteInterface(
args,
kwargs,
):
Parameter | Type |
---|---|
args |
*args |
kwargs |
**kwargs |
Methods
Method | Description |
---|---|
construct_node_metadata() |
None |
construct_node_metadata()
def construct_node_metadata()
Properties
Property | Type | Description |
---|---|---|
interface | ||
name |
flytekit.core.promise.Interface
A Python native interface object, like inspect.signature but simpler.
def Interface(
inputs: Union[Optional[Dict[str, Type]], Optional[Dict[str, Tuple[Type, Any]]]],
outputs: Union[Optional[Dict[str, Type]], Optional[Dict[str, Optional[Type]]]],
output_tuple_name: Optional[str],
docstring: Optional[Docstring],
):
Parameter | Type |
---|---|
inputs |
Union[Optional[Dict[str, Type]], Optional[Dict[str, Tuple[Type, Any]]]] |
outputs |
Union[Optional[Dict[str, Type]], Optional[Dict[str, Optional[Type]]]] |
output_tuple_name |
Optional[str] |
docstring |
Optional[Docstring] |
Methods
Method | Description |
---|---|
remove_inputs() |
This method is useful in removing some variables from the Flyte backend inputs specification, as these are |
with_inputs() |
Use this to add additional inputs to the interface |
with_outputs() |
This method allows addition of extra outputs are expected from a task specification |
remove_inputs()
def remove_inputs(
vars: Optional[List[str]],
):
This method is useful in removing some variables from the Flyte backend inputs specification, as these are implicit local only inputs or will be supplied by the library at runtime. For example, spark-session etc It creates a new instance of interface with the requested variables removed
Parameter | Type |
---|---|
vars |
Optional[List[str]] |
with_inputs()
def with_inputs(
extra_inputs: Dict[str, Type],
):
Use this to add additional inputs to the interface. This is useful for adding additional implicit inputs that are added without the user requesting for them
Parameter | Type |
---|---|
extra_inputs |
Dict[str, Type] |
with_outputs()
def with_outputs(
extra_outputs: Dict[str, Type],
):
This method allows addition of extra outputs are expected from a task specification
Parameter | Type |
---|---|
extra_outputs |
Dict[str, Type] |
Properties
Property | Type | Description |
---|---|---|
default_inputs_as_kwargs | ||
docstring | ||
inputs | ||
inputs_with_defaults | ||
output_names | ||
output_tuple | ||
output_tuple_name | ||
outputs |
flytekit.core.promise.Iterable
flytekit.core.promise.ListTransformer
Transformer that handles a univariate typing.List[T]
def ListTransformer()
Methods
Method | Description |
---|---|
assert_type() |
None |
async_to_literal() |
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type |
async_to_python_value() |
Converts the given Literal to a Python Type |
from_binary_idl() |
This function primarily handles deserialization for untyped dicts, dataclasses, Pydantic BaseModels, and attribute access |
from_generic_idl() |
TODO: Support all Flyte Types |
get_literal_type() |
Only univariate Lists are supported in Flyte |
get_sub_type() |
Return the generic Type T of the List |
get_sub_type_or_none() |
Return the generic Type T of the List, or None if the generic type cannot be inferred |
guess_python_type() |
Converts the Flyte LiteralType to a python object type |
isinstance_generic() |
None |
to_html() |
Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div |
to_literal() |
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type |
to_python_value() |
Converts the given Literal to a Python Type |
assert_type()
def assert_type(
t: Type[T],
v: T,
):
Parameter | Type |
---|---|
t |
Type[T] |
v |
T |
async_to_literal()
def async_to_literal(
ctx: FlyteContext,
python_val: T,
python_type: Type[T],
expected: LiteralType,
):
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
T |
python_type |
Type[T] |
expected |
LiteralType |
async_to_python_value()
def async_to_python_value(
ctx: FlyteContext,
lv: Literal,
expected_python_type: Type[T],
):
Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised
Parameter | Type |
---|---|
ctx |
FlyteContext |
lv |
Literal |
expected_python_type |
Type[T] |
from_binary_idl()
def from_binary_idl(
binary_idl_object: Binary,
expected_python_type: Type[T],
):
This function primarily handles deserialization for untyped dicts, dataclasses, Pydantic BaseModels, and attribute access.`
For untyped dict, dataclass, and pydantic basemodel: Life Cycle (Untyped Dict as example): python val -> msgpack bytes -> binary literal scalar -> msgpack bytes -> python val (to_literal) (from_binary_idl)
For attribute access: Life Cycle: python val -> msgpack bytes -> binary literal scalar -> resolved golang value -> binary literal scalar -> msgpack bytes -> python val (to_literal) (propeller attribute access) (from_binary_idl)
Parameter | Type |
---|---|
binary_idl_object |
Binary |
expected_python_type |
Type[T] |
from_generic_idl()
def from_generic_idl(
generic: Struct,
expected_python_type: Type[T],
):
TODO: Support all Flyte Types. This is for dataclass attribute access from input created from the Flyte Console.
Note:
- This can be removed in the future when the Flyte Console support generate Binary IDL Scalar as input.
Parameter | Type |
---|---|
generic |
Struct |
expected_python_type |
Type[T] |
get_literal_type()
def get_literal_type(
t: Type[T],
):
Only univariate Lists are supported in Flyte
Parameter | Type |
---|---|
t |
Type[T] |
get_sub_type()
def get_sub_type(
t: Type[T],
):
Return the generic Type T of the List
Parameter | Type |
---|---|
t |
Type[T] |
get_sub_type_or_none()
def get_sub_type_or_none(
t: Type[T],
):
Return the generic Type T of the List, or None if the generic type cannot be inferred
Parameter | Type |
---|---|
t |
Type[T] |
guess_python_type()
def guess_python_type(
literal_type: LiteralType,
):
Converts the Flyte LiteralType to a python object type.
Parameter | Type |
---|---|
literal_type |
LiteralType |
isinstance_generic()
def isinstance_generic(
obj,
generic_alias,
):
Parameter | Type |
---|---|
obj |
|
generic_alias |
to_html()
def to_html(
ctx: FlyteContext,
python_val: T,
expected_python_type: Type[T],
):
Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
T |
expected_python_type |
Type[T] |
to_literal()
def to_literal(
ctx: FlyteContext,
python_val: typing.Any,
python_type: Type[T],
expected: LiteralType,
):
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
typing.Any |
python_type |
Type[T] |
expected |
LiteralType |
to_python_value()
def to_python_value(
ctx: FlyteContext,
lv: Literal,
expected_python_type: Type[T],
):
Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised
Parameter | Type |
---|---|
ctx |
FlyteContext |
lv |
Literal |
expected_python_type |
Type[T] |
Properties
Property | Type | Description |
---|---|---|
is_async | ||
name | ||
python_type | ||
type_assertions_enabled |
flytekit.core.promise.Literal
def Literal(
scalar: typing.Optional[flytekit.models.literals.Scalar],
collection: typing.Optional[flytekit.models.literals.LiteralCollection],
map: typing.Optional[flytekit.models.literals.LiteralMap],
hash: typing.Optional[str],
metadata: typing.Optional[typing.Dict[str, str]],
offloaded_metadata: typing.Optional[flytekit.models.literals.LiteralOffloadedMetadata],
):
This IDL message represents a literal value in the Flyte ecosystem.
Parameter | Type |
---|---|
scalar |
typing.Optional[flytekit.models.literals.Scalar] |
collection |
typing.Optional[flytekit.models.literals.LiteralCollection] |
map |
typing.Optional[flytekit.models.literals.LiteralMap] |
hash |
typing.Optional[str] |
metadata |
typing.Optional[typing.Dict[str, str]] |
offloaded_metadata |
typing.Optional[flytekit.models.literals.LiteralOffloadedMetadata] |
Methods
Method | Description |
---|---|
from_flyte_idl() |
|
serialize_to_string() |
None |
set_metadata() |
Note: This is a mutation on the literal |
short_string() |
|
to_flyte_idl() |
|
verbose_string() |
from_flyte_idl()
def from_flyte_idl(
pb2_object: flyteidl.core.literals_pb2.Literal,
):
Parameter | Type |
---|---|
pb2_object |
flyteidl.core.literals_pb2.Literal |
serialize_to_string()
def serialize_to_string()
set_metadata()
def set_metadata(
metadata: typing.Dict[str, str],
):
Note: This is a mutation on the literal
Parameter | Type |
---|---|
metadata |
typing.Dict[str, str] |
short_string()
def short_string()
to_flyte_idl()
def to_flyte_idl()
verbose_string()
def verbose_string()
Properties
Property | Type | Description |
---|---|---|
collection | ||
hash | ||
is_empty | ||
map | ||
metadata | ||
offloaded_metadata | ||
scalar | ||
value |
flytekit.core.promise.LocallyExecutable
Base class for protocol classes.
Protocol classes are defined as::
class Proto(Protocol): def meth(self) -> int: …
Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing).
For example::
class C: def meth(self) -> int: return 0
def func(x: Proto) -> int: return x.meth()
func(C()) # Passes static type check
See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as::
class GenProtoT: def meth(self) -> T: …
def LocallyExecutable(
args,
kwargs,
):
Parameter | Type |
---|---|
args |
*args |
kwargs |
**kwargs |
Methods
Method | Description |
---|---|
local_execute() |
None |
local_execution_mode() |
None |
local_execute()
def local_execute(
ctx: FlyteContext,
kwargs,
):
Parameter | Type |
---|---|
ctx |
FlyteContext |
kwargs |
**kwargs |
local_execution_mode()
def local_execution_mode()
flytekit.core.promise.Node
This class will hold all the things necessary to make an SdkNode but we won’t make one until we know things like ID, which from the registration step
def Node(
id: str,
metadata: _workflow_model.NodeMetadata,
bindings: List[_literal_models.Binding],
upstream_nodes: List[Node],
flyte_entity: Any,
):
Parameter | Type |
---|---|
id |
str |
metadata |
_workflow_model.NodeMetadata |
bindings |
List[_literal_models.Binding] |
upstream_nodes |
List[Node] |
flyte_entity |
Any |
Methods
Method | Description |
---|---|
runs_before() |
This is typically something we shouldn’t do |
with_overrides() |
None |
runs_before()
def runs_before(
other: Node,
):
This is typically something we shouldn’t do. This modifies an attribute of the other instance rather than self. But it’s done so only because we wanted this English function to be the same as the shift function. That is, calling node_1.runs_before(node_2) and node_1 » node_2 are the same. The shift operator going the other direction is not implemented to further avoid confusion. Right shift was picked rather than left shift because that’s what most users are familiar with.
Parameter | Type |
---|---|
other |
Node |
with_overrides()
def with_overrides(
node_name: Optional[str],
aliases: Optional[Dict[str, str]],
requests: Optional[Resources],
limits: Optional[Resources],
timeout: Optional[Union[int, datetime.timedelta, object]],
retries: Optional[int],
interruptible: Optional[bool],
name: Optional[str],
task_config: Optional[Any],
container_image: Optional[str],
accelerator: Optional[BaseAccelerator],
cache: Optional[bool],
cache_version: Optional[str],
cache_serialize: Optional[bool],
shared_memory: Optional[Union[L[True], str]],
pod_template: Optional[PodTemplate],
resources: Optional[Resources],
args,
kwargs,
):
Parameter | Type |
---|---|
node_name |
Optional[str] |
aliases |
Optional[Dict[str, str]] |
requests |
Optional[Resources] |
limits |
Optional[Resources] |
timeout |
Optional[Union[int, datetime.timedelta, object]] |
retries |
Optional[int] |
interruptible |
Optional[bool] |
name |
Optional[str] |
task_config |
Optional[Any] |
container_image |
Optional[str] |
accelerator |
Optional[BaseAccelerator] |
cache |
Optional[bool] |
cache_version |
Optional[str] |
cache_serialize |
Optional[bool] |
shared_memory |
Optional[Union[L[True], str]] |
pod_template |
Optional[PodTemplate] |
resources |
Optional[Resources] |
args |
*args |
kwargs |
**kwargs |
Properties
Property | Type | Description |
---|---|---|
bindings | ||
flyte_entity | ||
id | ||
metadata | ||
name | ||
outputs | ||
run_entity | ||
upstream_nodes |
flytekit.core.promise.NodeOutput
def NodeOutput(
node: Node,
var: str,
attr_path: Optional[List[Union[str, int]]],
):
Parameter | Type |
---|---|
node |
Node |
var |
str |
attr_path |
Optional[List[Union[str, int]]] |
Methods
Method | Description |
---|---|
deepcopy() |
None |
from_flyte_idl() |
|
serialize_to_string() |
None |
short_string() |
|
to_flyte_idl() |
|
verbose_string() |
|
with_attr() |
None |
deepcopy()
def deepcopy()
from_flyte_idl()
def from_flyte_idl(
pb2_object,
):
Parameter | Type |
---|---|
pb2_object |
serialize_to_string()
def serialize_to_string()
short_string()
def short_string()
to_flyte_idl()
def to_flyte_idl()
verbose_string()
def verbose_string()
with_attr()
def with_attr(
key,
):
Parameter | Type |
---|---|
key |
Properties
Property | Type | Description |
---|---|---|
attr_path | ||
is_empty | ||
node | ||
node_id | ||
var |
flytekit.core.promise.OutputMetadataTracker
This class is for the users to set arbitrary metadata on output literals.
Attributes: output_metadata Optional[TaskOutputMetadata]: is a sparse dictionary of metadata that the user wants to attach to each output of a task. The key is the output value (object) and the value is an OutputMetadata object.
def OutputMetadataTracker(
output_metadata: typing.Dict[typing.Any, OutputMetadata],
):
Parameter | Type |
---|---|
output_metadata |
typing.Dict[typing.Any, OutputMetadata] |
Methods
Method | Description |
---|---|
add() |
None |
get() |
None |
with_params() |
Produces a copy of the current object and set new things |
add()
def add(
obj: typing.Any,
metadata: OutputMetadata,
):
Parameter | Type |
---|---|
obj |
typing.Any |
metadata |
OutputMetadata |
get()
def get(
obj: typing.Any,
):
Parameter | Type |
---|---|
obj |
typing.Any |
with_params()
def with_params(
output_metadata: Optional[TaskOutputMetadata],
):
Produces a copy of the current object and set new things
Parameter | Type |
---|---|
output_metadata |
Optional[TaskOutputMetadata] |
flytekit.core.promise.Primitive
def Primitive(
integer: typing.Optional[int],
float_value: typing.Optional[float],
string_value: typing.Optional[str],
boolean: typing.Optional[bool],
datetime: typing.Optional[datetime.datetime],
duration: typing.Optional[datetime.timedelta],
):
This object proxies the primitives supported by the Flyte IDL system. Only one value can be set.
Parameter | Type |
---|---|
integer |
typing.Optional[int] |
float_value |
typing.Optional[float] |
string_value |
typing.Optional[str] |
boolean |
typing.Optional[bool] |
datetime |
typing.Optional[datetime.datetime] |
duration |
typing.Optional[datetime.timedelta] |
Methods
Method | Description |
---|---|
from_flyte_idl() |
|
serialize_to_string() |
None |
short_string() |
|
to_flyte_idl() |
|
verbose_string() |
from_flyte_idl()
def from_flyte_idl(
proto,
):
Parameter | Type |
---|---|
proto |
serialize_to_string()
def serialize_to_string()
short_string()
def short_string()
to_flyte_idl()
def to_flyte_idl()
verbose_string()
def verbose_string()
Properties
Property | Type | Description |
---|---|---|
boolean | ||
datetime | ||
duration | ||
float_value | ||
integer | ||
is_empty | ||
string_value | ||
value |
flytekit.core.promise.Promise
This object is a wrapper and exists for three main reasons. Let’s assume we’re dealing with a task like ::
@task def t1() -> (int, str): …
#. Handling the duality between compilation and local execution - when the task function is run in a local execution mode inside a workflow function, a Python integer and string are produced. When the task is being compiled as part of the workflow, the task call creates a Node instead, and the task returns two Promise objects that point to that Node. #. One needs to be able to call ::
x = t1().with_overrides(…)
If the task returns an integer or a (int, str)
tuple like t1
above, calling with_overrides
on the
result would throw an error. This Promise object adds that.
#. Assorted handling for conditionals.
def Promise(
var: str,
val: Union[NodeOutput, _literals_models.Literal],
type: typing.Optional[_type_models.LiteralType],
):
Parameter | Type |
---|---|
var |
str |
val |
Union[NodeOutput, _literals_models.Literal] |
type |
typing.Optional[_type_models.LiteralType] |
Methods
Method | Description |
---|---|
deepcopy() |
None |
eval() |
None |
is_() |
None |
is_false() |
None |
is_none() |
None |
is_true() |
None |
with_overrides() |
None |
with_var() |
None |
deepcopy()
def deepcopy()
eval()
def eval()
is_()
def is_(
v: bool,
):
Parameter | Type |
---|---|
v |
bool |
is_false()
def is_false()
is_none()
def is_none()
is_true()
def is_true()
with_overrides()
def with_overrides(
node_name: Optional[str],
aliases: Optional[Dict[str, str]],
requests: Optional[Resources],
limits: Optional[Resources],
timeout: Optional[Union[int, datetime.timedelta, object]],
retries: Optional[int],
interruptible: Optional[bool],
name: Optional[str],
task_config: Optional[Any],
container_image: Optional[str],
accelerator: Optional[BaseAccelerator],
cache: Optional[bool],
cache_version: Optional[str],
cache_serialize: Optional[bool],
args,
kwargs,
):
Parameter | Type |
---|---|
node_name |
Optional[str] |
aliases |
Optional[Dict[str, str]] |
requests |
Optional[Resources] |
limits |
Optional[Resources] |
timeout |
Optional[Union[int, datetime.timedelta, object]] |
retries |
Optional[int] |
interruptible |
Optional[bool] |
name |
Optional[str] |
task_config |
Optional[Any] |
container_image |
Optional[str] |
accelerator |
Optional[BaseAccelerator] |
cache |
Optional[bool] |
cache_version |
Optional[str] |
cache_serialize |
Optional[bool] |
args |
*args |
kwargs |
**kwargs |
with_var()
def with_var(
new_var: str,
):
Parameter | Type |
---|---|
new_var |
str |
Properties
Property | Type | Description |
---|---|---|
attr_path | ||
is_ready | ||
ref | ||
val | ||
var |
flytekit.core.promise.Protocol
Base class for protocol classes.
Protocol classes are defined as::
class Proto(Protocol): def meth(self) -> int: …
Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing).
For example::
class C: def meth(self) -> int: return 0
def func(x: Proto) -> int: return x.meth()
func(C()) # Passes static type check
See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as::
class GenProtoT: def meth(self) -> T: …
flytekit.core.promise.Resources
def Resources(
requests,
limits,
):
Parameter | Type |
---|---|
requests |
|
limits |
Methods
Method | Description |
---|---|
from_flyte_idl() |
|
serialize_to_string() |
None |
short_string() |
|
to_flyte_idl() |
|
verbose_string() |
from_flyte_idl()
def from_flyte_idl(
pb2_object,
):
Parameter | Type |
---|---|
pb2_object |
serialize_to_string()
def serialize_to_string()
short_string()
def short_string()
to_flyte_idl()
def to_flyte_idl()
verbose_string()
def verbose_string()
Properties
Property | Type | Description |
---|---|---|
is_empty | ||
limits | ||
requests |
flytekit.core.promise.Scalar
def Scalar(
primitive: typing.Optional[flytekit.models.literals.Primitive],
blob: typing.Optional[flytekit.models.literals.Blob],
binary: typing.Optional[flytekit.models.literals.Binary],
schema: typing.Optional[flytekit.models.literals.Schema],
union: typing.Optional[flytekit.models.literals.Union],
none_type: typing.Optional[flytekit.models.literals.Void],
error: typing.Optional[flytekit.models.types.Error],
generic: typing.Optional[google.protobuf.struct_pb2.Struct],
structured_dataset: typing.Optional[flytekit.models.literals.StructuredDataset],
):
Scalar wrapper around Flyte types. Only one can be specified.
Parameter | Type |
---|---|
primitive |
typing.Optional[flytekit.models.literals.Primitive] |
blob |
typing.Optional[flytekit.models.literals.Blob] |
binary |
typing.Optional[flytekit.models.literals.Binary] |
schema |
typing.Optional[flytekit.models.literals.Schema] |
union |
typing.Optional[flytekit.models.literals.Union] |
none_type |
typing.Optional[flytekit.models.literals.Void] |
error |
typing.Optional[flytekit.models.types.Error] |
generic |
typing.Optional[google.protobuf.struct_pb2.Struct] |
structured_dataset |
typing.Optional[flytekit.models.literals.StructuredDataset] |
Methods
Method | Description |
---|---|
from_flyte_idl() |
|
serialize_to_string() |
None |
short_string() |
|
to_flyte_idl() |
|
verbose_string() |
from_flyte_idl()
def from_flyte_idl(
pb2_object,
):
Parameter | Type |
---|---|
pb2_object |
serialize_to_string()
def serialize_to_string()
short_string()
def short_string()
to_flyte_idl()
def to_flyte_idl()
verbose_string()
def verbose_string()
Properties
Property | Type | Description |
---|---|---|
binary | ||
blob | ||
error | ||
generic | ||
is_empty | ||
none_type | ||
primitive | ||
schema | ||
structured_dataset | ||
union | ||
value |
flytekit.core.promise.SimpleType
flytekit.core.promise.SupportsNodeCreation
Base class for protocol classes.
Protocol classes are defined as::
class Proto(Protocol): def meth(self) -> int: …
Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing).
For example::
class C: def meth(self) -> int: return 0
def func(x: Proto) -> int: return x.meth()
func(C()) # Passes static type check
See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as::
class GenProtoT: def meth(self) -> T: …
def SupportsNodeCreation(
args,
kwargs,
):
Parameter | Type |
---|---|
args |
*args |
kwargs |
**kwargs |
Methods
Method | Description |
---|---|
construct_node_metadata() |
None |
construct_node_metadata()
def construct_node_metadata()
Properties
Property | Type | Description |
---|---|---|
name | ||
python_interface |
flytekit.core.promise.TypeEngine
Core Extensible TypeEngine of Flytekit. This should be used to extend the capabilities of FlyteKits type system. Users can implement their own TypeTransformers and register them with the TypeEngine. This will allow special handling of user objects
Methods
Method | Description |
---|---|
async_to_literal() |
Converts a python value of a given type and expected LiteralType into a resolved Literal value |
async_to_python_value() |
None |
calculate_hash() |
None |
dict_to_literal_map() |
None |
dict_to_literal_map_pb() |
None |
get_available_transformers() |
Returns all python types for which transformers are available |
get_transformer() |
Implements a recursive search for the transformer |
guess_python_type() |
Transforms a flyte-specific LiteralType to a regular python value |
guess_python_types() |
Transforms a dictionary of flyte-specific Variable objects to a dictionary of regular python values |
lazy_import_transformers() |
Only load the transformers if needed |
literal_map_to_kwargs() |
None |
named_tuple_to_variable_map() |
Converts a python-native NamedTuple to a flyte-specific VariableMap of named literals |
register() |
This should be used for all types that respond with the right type annotation when you use type( |
register_additional_type() |
None |
register_restricted_type() |
None |
to_html() |
None |
to_literal() |
The current dance is because we are allowing users to call from an async function, this synchronous |
to_literal_checks() |
None |
to_literal_type() |
Converts a python type into a flyte specific LiteralType |
to_python_value() |
Converts a Literal value with an expected python type into a python value |
unwrap_offloaded_literal() |
None |
async_to_literal()
def async_to_literal(
ctx: FlyteContext,
python_val: typing.Any,
python_type: Type[T],
expected: LiteralType,
):
Converts a python value of a given type and expected LiteralType
into a resolved Literal
value.
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
typing.Any |
python_type |
Type[T] |
expected |
LiteralType |
async_to_python_value()
def async_to_python_value(
ctx: FlyteContext,
lv: Literal,
expected_python_type: Type,
):
Parameter | Type |
---|---|
ctx |
FlyteContext |
lv |
Literal |
expected_python_type |
Type |
calculate_hash()
def calculate_hash(
python_val: typing.Any,
python_type: Type[T],
):
Parameter | Type |
---|---|
python_val |
typing.Any |
python_type |
Type[T] |
dict_to_literal_map()
def dict_to_literal_map(
ctx: FlyteContext,
d: typing.Dict[str, typing.Any],
type_hints: Optional[typing.Dict[str, type]],
):
Parameter | Type |
---|---|
ctx |
FlyteContext |
d |
typing.Dict[str, typing.Any] |
type_hints |
Optional[typing.Dict[str, type]] |
dict_to_literal_map_pb()
def dict_to_literal_map_pb(
ctx: FlyteContext,
d: typing.Dict[str, typing.Any],
type_hints: Optional[typing.Dict[str, type]],
):
Parameter | Type |
---|---|
ctx |
FlyteContext |
d |
typing.Dict[str, typing.Any] |
type_hints |
Optional[typing.Dict[str, type]] |
get_available_transformers()
def get_available_transformers()
Returns all python types for which transformers are available
get_transformer()
def get_transformer(
python_type: Type,
):
Implements a recursive search for the transformer.
Parameter | Type |
---|---|
python_type |
Type |
guess_python_type()
def guess_python_type(
flyte_type: LiteralType,
):
Transforms a flyte-specific LiteralType
to a regular python value.
Parameter | Type |
---|---|
flyte_type |
LiteralType |
guess_python_types()
def guess_python_types(
flyte_variable_dict: typing.Dict[str, _interface_models.Variable],
):
Transforms a dictionary of flyte-specific Variable
objects to a dictionary of regular python values.
Parameter | Type |
---|---|
flyte_variable_dict |
typing.Dict[str, _interface_models.Variable] |
lazy_import_transformers()
def lazy_import_transformers()
Only load the transformers if needed.
literal_map_to_kwargs()
def literal_map_to_kwargs(
ctx: FlyteContext,
lm: LiteralMap,
python_types: typing.Optional[typing.Dict[str, type]],
literal_types: typing.Optional[typing.Dict[str, _interface_models.Variable]],
):
Parameter | Type |
---|---|
ctx |
FlyteContext |
lm |
LiteralMap |
python_types |
typing.Optional[typing.Dict[str, type]] |
literal_types |
typing.Optional[typing.Dict[str, _interface_models.Variable]] |
named_tuple_to_variable_map()
def named_tuple_to_variable_map(
t: typing.NamedTuple,
):
Converts a python-native NamedTuple
to a flyte-specific VariableMap of named literals.
Parameter | Type |
---|---|
t |
typing.NamedTuple |
register()
def register(
transformer: TypeTransformer,
additional_types: Optional[typing.List[Type]],
):
This should be used for all types that respond with the right type annotation when you use type(…) function
Parameter | Type |
---|---|
transformer |
TypeTransformer |
additional_types |
Optional[typing.List[Type]] |
register_additional_type()
def register_additional_type(
transformer: TypeTransformer[T],
additional_type: Type[T],
override,
):
Parameter | Type |
---|---|
transformer |
TypeTransformer[T] |
additional_type |
Type[T] |
override |
register_restricted_type()
def register_restricted_type(
name: str,
type: Type[T],
):
Parameter | Type |
---|---|
name |
str |
type |
Type[T] |
to_html()
def to_html(
ctx: FlyteContext,
python_val: typing.Any,
expected_python_type: Type[typing.Any],
):
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
typing.Any |
expected_python_type |
Type[typing.Any] |
to_literal()
def to_literal(
ctx: FlyteContext,
python_val: typing.Any,
python_type: Type[T],
expected: LiteralType,
):
The current dance is because we are allowing users to call from an async function, this synchronous to_literal function, and allowing this to_literal function, to then invoke yet another async function, namely an async transformer.
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
typing.Any |
python_type |
Type[T] |
expected |
LiteralType |
to_literal_checks()
def to_literal_checks(
python_val: typing.Any,
python_type: Type[T],
expected: LiteralType,
):
Parameter | Type |
---|---|
python_val |
typing.Any |
python_type |
Type[T] |
expected |
LiteralType |
to_literal_type()
def to_literal_type(
python_type: Type[T],
):
Converts a python type into a flyte specific LiteralType
Parameter | Type |
---|---|
python_type |
Type[T] |
to_python_value()
def to_python_value(
ctx: FlyteContext,
lv: Literal,
expected_python_type: Type,
):
Converts a Literal value with an expected python type into a python value.
Parameter | Type |
---|---|
ctx |
FlyteContext |
lv |
Literal |
expected_python_type |
Type |
unwrap_offloaded_literal()
def unwrap_offloaded_literal(
ctx: FlyteContext,
lv: Literal,
):
Parameter | Type |
---|---|
ctx |
FlyteContext |
lv |
Literal |
flytekit.core.promise.TypeTransformer
Base transformer type that should be implemented for every python native type that can be handled by flytekit
def TypeTransformer(
name: str,
t: Type[T],
enable_type_assertions: bool,
):
Parameter | Type |
---|---|
name |
str |
t |
Type[T] |
enable_type_assertions |
bool |
Methods
Method | Description |
---|---|
assert_type() |
None |
from_binary_idl() |
This function primarily handles deserialization for untyped dicts, dataclasses, Pydantic BaseModels, and attribute access |
from_generic_idl() |
TODO: Support all Flyte Types |
get_literal_type() |
Converts the python type to a Flyte LiteralType |
guess_python_type() |
Converts the Flyte LiteralType to a python object type |
isinstance_generic() |
None |
to_html() |
Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div |
to_literal() |
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type |
to_python_value() |
Converts the given Literal to a Python Type |
assert_type()
def assert_type(
t: Type[T],
v: T,
):
Parameter | Type |
---|---|
t |
Type[T] |
v |
T |
from_binary_idl()
def from_binary_idl(
binary_idl_object: Binary,
expected_python_type: Type[T],
):
This function primarily handles deserialization for untyped dicts, dataclasses, Pydantic BaseModels, and attribute access.`
For untyped dict, dataclass, and pydantic basemodel: Life Cycle (Untyped Dict as example): python val -> msgpack bytes -> binary literal scalar -> msgpack bytes -> python val (to_literal) (from_binary_idl)
For attribute access: Life Cycle: python val -> msgpack bytes -> binary literal scalar -> resolved golang value -> binary literal scalar -> msgpack bytes -> python val (to_literal) (propeller attribute access) (from_binary_idl)
Parameter | Type |
---|---|
binary_idl_object |
Binary |
expected_python_type |
Type[T] |
from_generic_idl()
def from_generic_idl(
generic: Struct,
expected_python_type: Type[T],
):
TODO: Support all Flyte Types. This is for dataclass attribute access from input created from the Flyte Console.
Note:
- This can be removed in the future when the Flyte Console support generate Binary IDL Scalar as input.
Parameter | Type |
---|---|
generic |
Struct |
expected_python_type |
Type[T] |
get_literal_type()
def get_literal_type(
t: Type[T],
):
Converts the python type to a Flyte LiteralType
Parameter | Type |
---|---|
t |
Type[T] |
guess_python_type()
def guess_python_type(
literal_type: LiteralType,
):
Converts the Flyte LiteralType to a python object type.
Parameter | Type |
---|---|
literal_type |
LiteralType |
isinstance_generic()
def isinstance_generic(
obj,
generic_alias,
):
Parameter | Type |
---|---|
obj |
|
generic_alias |
to_html()
def to_html(
ctx: FlyteContext,
python_val: T,
expected_python_type: Type[T],
):
Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
T |
expected_python_type |
Type[T] |
to_literal()
def to_literal(
ctx: FlyteContext,
python_val: T,
python_type: Type[T],
expected: LiteralType,
):
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
T |
python_type |
Type[T] |
expected |
LiteralType |
to_python_value()
def to_python_value(
ctx: FlyteContext,
lv: Literal,
expected_python_type: Type[T],
):
Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised
Parameter | Type |
---|---|
ctx |
FlyteContext |
lv |
Literal |
expected_python_type |
Type[T] |
Properties
Property | Type | Description |
---|---|---|
is_async | ||
name | ||
python_type | ||
type_assertions_enabled |
flytekit.core.promise.TypeTransformerFailedError
Inappropriate argument type.
flytekit.core.promise.UnionTransformer
Transformer that handles a typing.Union[T1, T2, …]
def UnionTransformer()
Methods
Method | Description |
---|---|
assert_type() |
None |
async_to_literal() |
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type |
async_to_python_value() |
Converts the given Literal to a Python Type |
from_binary_idl() |
This function primarily handles deserialization for untyped dicts, dataclasses, Pydantic BaseModels, and attribute access |
from_generic_idl() |
TODO: Support all Flyte Types |
get_literal_type() |
Converts the python type to a Flyte LiteralType |
get_sub_type_in_optional() |
Return the generic Type T of the Optional type |
guess_python_type() |
Converts the Flyte LiteralType to a python object type |
is_optional_type() |
None |
isinstance_generic() |
None |
to_html() |
Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div |
to_literal() |
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type |
to_python_value() |
Converts the given Literal to a Python Type |
assert_type()
def assert_type(
t: Type[T],
v: T,
):
Parameter | Type |
---|---|
t |
Type[T] |
v |
T |
async_to_literal()
def async_to_literal(
ctx: FlyteContext,
python_val: T,
python_type: Type[T],
expected: LiteralType,
):
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
T |
python_type |
Type[T] |
expected |
LiteralType |
async_to_python_value()
def async_to_python_value(
ctx: FlyteContext,
lv: Literal,
expected_python_type: Type[T],
):
Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised
Parameter | Type |
---|---|
ctx |
FlyteContext |
lv |
Literal |
expected_python_type |
Type[T] |
from_binary_idl()
def from_binary_idl(
binary_idl_object: Binary,
expected_python_type: Type[T],
):
This function primarily handles deserialization for untyped dicts, dataclasses, Pydantic BaseModels, and attribute access.`
For untyped dict, dataclass, and pydantic basemodel: Life Cycle (Untyped Dict as example): python val -> msgpack bytes -> binary literal scalar -> msgpack bytes -> python val (to_literal) (from_binary_idl)
For attribute access: Life Cycle: python val -> msgpack bytes -> binary literal scalar -> resolved golang value -> binary literal scalar -> msgpack bytes -> python val (to_literal) (propeller attribute access) (from_binary_idl)
Parameter | Type |
---|---|
binary_idl_object |
Binary |
expected_python_type |
Type[T] |
from_generic_idl()
def from_generic_idl(
generic: Struct,
expected_python_type: Type[T],
):
TODO: Support all Flyte Types. This is for dataclass attribute access from input created from the Flyte Console.
Note:
- This can be removed in the future when the Flyte Console support generate Binary IDL Scalar as input.
Parameter | Type |
---|---|
generic |
Struct |
expected_python_type |
Type[T] |
get_literal_type()
def get_literal_type(
t: Type[T],
):
Converts the python type to a Flyte LiteralType
Parameter | Type |
---|---|
t |
Type[T] |
get_sub_type_in_optional()
def get_sub_type_in_optional(
t: Type[T],
):
Return the generic Type T of the Optional type
Parameter | Type |
---|---|
t |
Type[T] |
guess_python_type()
def guess_python_type(
literal_type: LiteralType,
):
Converts the Flyte LiteralType to a python object type.
Parameter | Type |
---|---|
literal_type |
LiteralType |
is_optional_type()
def is_optional_type(
t: Type,
):
Parameter | Type |
---|---|
t |
Type |
isinstance_generic()
def isinstance_generic(
obj,
generic_alias,
):
Parameter | Type |
---|---|
obj |
|
generic_alias |
to_html()
def to_html(
ctx: FlyteContext,
python_val: T,
expected_python_type: Type[T],
):
Converts any python val (dataframe, int, float) to a html string, and it will be wrapped in the HTML div
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
T |
expected_python_type |
Type[T] |
to_literal()
def to_literal(
ctx: FlyteContext,
python_val: typing.Any,
python_type: Type[T],
expected: LiteralType,
):
Converts a given python_val to a Flyte Literal, assuming the given python_val matches the declared python_type. Implementers should refrain from using type(python_val) instead rely on the passed in python_type. If these do not match (or are not allowed) the Transformer implementer should raise an AssertionError, clearly stating what was the mismatch
Parameter | Type |
---|---|
ctx |
FlyteContext |
python_val |
typing.Any |
python_type |
Type[T] |
expected |
LiteralType |
to_python_value()
def to_python_value(
ctx: FlyteContext,
lv: Literal,
expected_python_type: Type[T],
):
Converts the given Literal to a Python Type. If the conversion cannot be done an AssertionError should be raised
Parameter | Type |
---|---|
ctx |
FlyteContext |
lv |
Literal |
expected_python_type |
Type[T] |
Properties
Property | Type | Description |
---|---|---|
is_async | ||
name | ||
python_type | ||
type_assertions_enabled |
flytekit.core.promise.VoidPromise
This object is returned for tasks that do not return any outputs (declared interface is empty) VoidPromise cannot be interacted with and does not allow comparisons or any operations
def VoidPromise(
task_name: str,
ref: Optional[NodeOutput],
):
Parameter | Type |
---|---|
task_name |
str |
ref |
Optional[NodeOutput] |
Methods
Method | Description |
---|---|
runs_before() |
This is a placeholder and should do nothing |
with_overrides() |
None |
runs_before()
def runs_before(
args,
kwargs,
):
This is a placeholder and should do nothing. It is only here to enable local execution of workflows where a task returns nothing.
Parameter | Type |
---|---|
args |
*args |
kwargs |
**kwargs |
with_overrides()
def with_overrides(
args,
kwargs,
):
Parameter | Type |
---|---|
args |
*args |
kwargs |
**kwargs |
Properties
Property | Type | Description |
---|---|---|
ref | ||
task_name |