flytekit.clis.sdk_in_container.build

Directory

Classes

flytekit.clis.sdk_in_container.build.BuildCommand

A click command group for building a image for flyte workflows & tasks in a file.

def BuildCommand(
    args,
    kwargs,
):

Initialize RichGroup class.

Methods

add_command()

def add_command(
    cmd: click.core.Command,
    name: typing.Optional[str],
):

Registers another :class:Command with this group. If the name is not provided, the name of the command is used.

collect_usage_pieces()

def collect_usage_pieces(
    ctx: click.core.Context,
):

Returns all the pieces that go into the usage line and returns it as a list of strings.

command()

def command(
    args: `*args`,
    kwargs: `**kwargs`,
):

A shortcut decorator for declaring and attaching a command to the group. This takes the same arguments as :func:command and immediately registers the created command with this group by calling :meth:add_command.

To customize the command class used, set the :attr:command_class attribute.

.. versionchanged:: 8.1 This decorator can be applied without parentheses.

.. versionchanged:: 8.0 Added the :attr:command_class attribute.

format_commands()

def format_commands(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Extra format methods for multi methods that adds all the commands after the options.

format_epilog()

def format_epilog(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Writes the epilog into the formatter if it exists.

format_help()

def format_help(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Writes the help into the formatter if it exists.

This is a low-level method called by :meth:get_help.

This calls the following methods:

• :meth:format_usage
• :meth:format_help_text
• :meth:format_options
• :meth:format_epilog

format_help_text()

def format_help_text(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Writes the help text to the formatter if it exists.

format_options()

def format_options(
    ctx: click.core.Context,
    formatter: click.formatting.HelpFormatter,
):

Writes all the options into the formatter if they exist.

format_usage()

def format_usage(
    ctx: click.core.Context,
    formatter: click.formatting.HelpFormatter,
):

Writes the usage line into the formatter.

This is a low-level method called by :meth:get_usage.

get_command()

def get_command(
    ctx,
    filename,
):

Given a context and a command name, this returns a :class:Command object if it exists or returns None.

get_help()

def get_help(
    ctx: click.core.Context,
):

Formats the help into a string and returns it.

Calls :meth:format_help internally.

get_help_option()

def get_help_option(
    ctx: click.core.Context,
):

Returns the help option object.

Unless add_help_option is False.

.. versionchanged:: 8.1.8 The help option is now cached to avoid creating it multiple times.

get_help_option_names()

def get_help_option_names(
    ctx: click.core.Context,
):

Returns the names for the help option.

get_params()

def get_params(
    ctx: click.core.Context,
):

get_short_help_str()

def get_short_help_str(
    limit: int,
):

Gets short help for the command or makes it by shortening the long help string.

get_usage()

def get_usage(
    ctx: click.core.Context,
):

Formats the usage line into a string and returns it.

Calls :meth:format_usage internally.

group()

def group(
    args: `*args`,
    kwargs: `**kwargs`,
):

A shortcut decorator for declaring and attaching a group to the group. This takes the same arguments as :func:group and immediately registers the created group with this group by calling :meth:add_command.

To customize the group class used, set the :attr:group_class attribute.

.. versionchanged:: 8.1 This decorator can be applied without parentheses.

.. versionchanged:: 8.0 Added the :attr:group_class attribute.

invoke()

def invoke(
    ctx: click.core.Context,
):

Given a context, this invokes the attached callback (if it exists) in the right way.

list_commands()

def list_commands(
    ctx,
    args,
    kwargs,
):

Returns a list of subcommand names in the order they should appear.

main()

def main(
    args: `*args`,
    prog_name: typing.Optional[str],
    complete_var: typing.Optional[str],
    standalone_mode: bool,
    windows_expand_args: bool,
    extra: typing.Any,
):

This is the way to invoke a script with all the bells and whistles as a command line application. This will always terminate the application after a call. If this is not wanted, SystemExit needs to be caught.

This method is also available by directly calling the instance of a :class:Command.

make_context()

def make_context(
    info_name: typing.Optional[str],
    args: `*args`,
    parent: typing.Optional[click.core.Context],
    extra: typing.Any,
):

This function when given an info name and arguments will kick off the parsing and create a new :class:Context. It does not invoke the actual command callback though.

To quickly customize the context class used without overriding this method, set the :attr:context_class attribute.

make_parser()

def make_parser(
    ctx: click.core.Context,
):

Creates the underlying option parser for this command.

parse_args()

def parse_args(
    ctx: click.core.Context,
    args: `*args`,
):

Given a context and a list of arguments this creates the parser and parses the arguments, then modifies the context as necessary. This is automatically invoked by :meth:make_context.

resolve_command()

def resolve_command(
    ctx: click.core.Context,
    args: `*args`,
):

result_callback()

def result_callback(
    replace: bool,
):

Adds a result callback to the command. By default if a result callback is already registered this will chain them but this can be disabled with the replace parameter. The result callback is invoked with the return value of the subcommand (or the list of return values from all subcommands if chaining is enabled) as well as the parameters as they would be passed to the main callback.

Example::

@click.group() @click.option(’-i’, ‘–input’, default=23) def cli(input): return 42

@cli.result_callback() def process_result(result, input): return result + input

shell_complete()

def shell_complete(
    ctx: click.core.Context,
    incomplete: str,
):

Return a list of completions for the incomplete value. Looks at the names of options, subcommands, and chained multi-commands.

to_info_dict()

def to_info_dict(
    ctx: click.core.Context,
):

Gather information that could be useful for a tool generating user-facing documentation. This traverses the entire structure below this command.

Use :meth:click.Context.to_info_dict to traverse the entire CLI structure.

Properties

flytekit.clis.sdk_in_container.build.BuildParams

def BuildParams(
    config_file: typing.Optional[str],
    verbose: bool,
    pkgs: typing.List[str],
    project: str,
    domain: str,
    destination_dir: str,
    copy_all: bool,
    copy: typing.Optional[flytekit.constants.CopyFileDetection],
    image_config: flytekit.configuration.ImageConfig,
    service_account: str,
    wait_execution: bool,
    poll_interval: int,
    dump_snippet: bool,
    overwrite_cache: bool,
    envvars: typing.Dict[str, str],
    tags: typing.List[str],
    name: str,
    labels: typing.Dict[str, str],
    annotations: typing.Dict[str, str],
    raw_output_data_prefix: str,
    max_parallelism: int,
    disable_notifications: bool,
    remote: bool,
    limit: int,
    cluster_pool: str,
    execution_cluster_label: str,
    computed_params: flytekit.clis.sdk_in_container.run.RunLevelComputedParams,
    _remote: typing.Optional[flytekit.remote.remote.FlyteRemote],
    fast: bool,
):

Methods

from_dict()

def from_dict(
    d: typing.Dict[str, typing.Any],
):

options()

def options()

Return the set of base parameters added to every pyflyte run workflow subcommand.

remote_instance()

def remote_instance()

Properties

flytekit.clis.sdk_in_container.build.BuildWorkflowCommand

click multicommand at the python file layer, subcommands should be all the workflows in the file.

def BuildWorkflowCommand(
    filename: str,
    args,
    kwargs,
):

Initialize RichGroup class.

Methods

add_command()

def add_command(
    cmd: click.core.Command,
    name: typing.Optional[str],
):

Registers another :class:Command with this group. If the name is not provided, the name of the command is used.

collect_usage_pieces()

def collect_usage_pieces(
    ctx: click.core.Context,
):

Returns all the pieces that go into the usage line and returns it as a list of strings.

command()

def command(
    args: `*args`,
    kwargs: `**kwargs`,
):

A shortcut decorator for declaring and attaching a command to the group. This takes the same arguments as :func:command and immediately registers the created command with this group by calling :meth:add_command.

To customize the command class used, set the :attr:command_class attribute.

.. versionchanged:: 8.1 This decorator can be applied without parentheses.

.. versionchanged:: 8.0 Added the :attr:command_class attribute.

format_commands()

def format_commands(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Extra format methods for multi methods that adds all the commands after the options.

format_epilog()

def format_epilog(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Writes the epilog into the formatter if it exists.

format_help()

def format_help(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Writes the help into the formatter if it exists.

This is a low-level method called by :meth:get_help.

This calls the following methods:

• :meth:format_usage
• :meth:format_help_text
• :meth:format_options
• :meth:format_epilog

format_help_text()

def format_help_text(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Writes the help text to the formatter if it exists.

format_options()

def format_options(
    ctx: click.core.Context,
    formatter: click.formatting.HelpFormatter,
):

Writes all the options into the formatter if they exist.

format_usage()

def format_usage(
    ctx: click.core.Context,
    formatter: click.formatting.HelpFormatter,
):

Writes the usage line into the formatter.

This is a low-level method called by :meth:get_usage.

get_command()

def get_command(
    ctx,
    exe_entity,
):

This command uses the filename with which this command was created, and the string name of the entity passed after the Python filename on the command line, to load the Python object, and then return the Command that click should run.

get_help()

def get_help(
    ctx: click.core.Context,
):

Formats the help into a string and returns it.

Calls :meth:format_help internally.

get_help_option()

def get_help_option(
    ctx: click.core.Context,
):

Returns the help option object.

Unless add_help_option is False.

.. versionchanged:: 8.1.8 The help option is now cached to avoid creating it multiple times.

get_help_option_names()

def get_help_option_names(
    ctx: click.core.Context,
):

Returns the names for the help option.

get_params()

def get_params(
    ctx: click.core.Context,
):

get_short_help_str()

def get_short_help_str(
    limit: int,
):

Gets short help for the command or makes it by shortening the long help string.

get_usage()

def get_usage(
    ctx: click.core.Context,
):

Formats the usage line into a string and returns it.

Calls :meth:format_usage internally.

group()

def group(
    args: `*args`,
    kwargs: `**kwargs`,
):

A shortcut decorator for declaring and attaching a group to the group. This takes the same arguments as :func:group and immediately registers the created group with this group by calling :meth:add_command.

To customize the group class used, set the :attr:group_class attribute.

.. versionchanged:: 8.1 This decorator can be applied without parentheses.

.. versionchanged:: 8.0 Added the :attr:group_class attribute.

invoke()

def invoke(
    ctx: click.core.Context,
):

Given a context, this invokes the attached callback (if it exists) in the right way.

list_commands()

def list_commands(
    ctx,
):

Returns a list of subcommand names in the order they should appear.

main()

def main(
    args: `*args`,
    prog_name: typing.Optional[str],
    complete_var: typing.Optional[str],
    standalone_mode: bool,
    windows_expand_args: bool,
    extra: typing.Any,
):

This is the way to invoke a script with all the bells and whistles as a command line application. This will always terminate the application after a call. If this is not wanted, SystemExit needs to be caught.

This method is also available by directly calling the instance of a :class:Command.

make_context()

def make_context(
    info_name: typing.Optional[str],
    args: `*args`,
    parent: typing.Optional[click.core.Context],
    extra: typing.Any,
):

This function when given an info name and arguments will kick off the parsing and create a new :class:Context. It does not invoke the actual command callback though.

To quickly customize the context class used without overriding this method, set the :attr:context_class attribute.

make_parser()

def make_parser(
    ctx: click.core.Context,
):

Creates the underlying option parser for this command.

parse_args()

def parse_args(
    ctx: click.core.Context,
    args: `*args`,
):

Given a context and a list of arguments this creates the parser and parses the arguments, then modifies the context as necessary. This is automatically invoked by :meth:make_context.

resolve_command()

def resolve_command(
    ctx: click.core.Context,
    args: `*args`,
):

result_callback()

def result_callback(
    replace: bool,
):

Adds a result callback to the command. By default if a result callback is already registered this will chain them but this can be disabled with the replace parameter. The result callback is invoked with the return value of the subcommand (or the list of return values from all subcommands if chaining is enabled) as well as the parameters as they would be passed to the main callback.

Example::

@click.group() @click.option(’-i’, ‘–input’, default=23) def cli(input): return 42

@cli.result_callback() def process_result(result, input): return result + input

shell_complete()

def shell_complete(
    ctx: click.core.Context,
    incomplete: str,
):

Return a list of completions for the incomplete value. Looks at the names of options, subcommands, and chained multi-commands.

to_info_dict()

def to_info_dict(
    ctx: click.core.Context,
):

Gather information that could be useful for a tool generating user-facing documentation. This traverses the entire structure below this command.

Use :meth:click.Context.to_info_dict to traverse the entire CLI structure.

Properties

flytekit.clis.sdk_in_container.build.ImageConfig

We recommend you to use ImageConfig.auto(img_name=None) to create an ImageConfig. For example, ImageConfig.auto(img_name=““ghcr.io/flyteorg/flytecookbook:v1.0.0"”) will create an ImageConfig.

ImageConfig holds available images which can be used at registration time. A default image can be specified along with optional additional images. Each image in the config must have a unique name.

Attributes: default_image (Optional[Image]): The default image to be used as a container for task serialization. images (List[Image]): Optional, additional images which can be used in task container definitions.

def ImageConfig(
    default_image: Optional[Image],
    images: Optional[List[Image]],
):

Methods

auto()

def auto(
    config_file: typing.Union[str, ConfigFile, None],
    img_name: Optional[str],
):

Reads from config file or from img_name Note that this function does not take into account the flytekit default images (see the Dockerfiles at the base of this repo). To pick those up, see the auto_default_image function..

auto_default_image()

def auto_default_image()

create_from()

def create_from(
    default_image: Optional[Image],
    other_images: typing.Optional[typing.List[Image]],
):

find_image()

def find_image(
    name,
):

Return an image, by name, if it exists.

from_dict()

def from_dict(
    kvs: typing.Union[dict, list, str, int, float, bool, NoneType],
    infer_missing,
):

from_images()

def from_images(
    default_image: str,
    m: typing.Optional[typing.Dict[str, str]],
):

Allows you to programmatically create an ImageConfig. Usually only the default_image is required, unless your workflow uses multiple images

.. code:: python

ImageConfig.from_dict( “ghcr.io/flyteorg/flytecookbook:v1.0.0”, { “spark”: “ghcr.io/flyteorg/myspark:…”, “other”: “…”, } )

urn:

from_json()

def from_json(
    s: typing.Union[str, bytes, bytearray],
    parse_float,
    parse_int,
    parse_constant,
    infer_missing,
    kw,
):

schema()

def schema(
    infer_missing: bool,
    only,
    exclude,
    many: bool,
    context,
    load_only,
    dump_only,
    partial: bool,
    unknown,
):

to_dict()

def to_dict(
    encode_json,
):

to_json()

def to_json(
    skipkeys: bool,
    ensure_ascii: bool,
    check_circular: bool,
    allow_nan: bool,
    indent: typing.Union[int, str, NoneType],
    separators: typing.Tuple[str, str],
    default: typing.Callable,
    sort_keys: bool,
    kw,
):

validate_image()

def validate_image(
    _: typing.Any,
    param: str,
    values: tuple,
):

Validates the image to match the standard format. Also validates that only one default image is provided. a default image, is one that is specified as default=<image_uri> or just <image_uri>. All other images should be provided with a name, in the format name=<image_uri> This method can be used with the CLI

flytekit.clis.sdk_in_container.build.PythonFunctionWorkflow

Please read :std:ref:flyte:divedeep-workflows first for a high-level understanding of what workflows are in Flyte. This Python object represents a workflow defined by a function and decorated with the :py:func:@workflow <flytekit.workflow> decorator. Please see notes on that object for additional information.

def PythonFunctionWorkflow(
    workflow_function: Callable,
    metadata: WorkflowMetadata,
    default_metadata: WorkflowMetadataDefaults,
    docstring: Optional[Docstring],
    on_failure: Optional[Union[WorkflowBase, Task]],
    docs: Optional[Documentation],
    pickle_untyped: bool,
    default_options: Optional[Options],
):

Methods

add()

def add(
    t: flytekit.core.python_auto_container.PythonAutoContainerTask,
):

compile()

def compile(
    kwargs,
):

Supply static Python native values in the kwargs if you want them to be used in the compilation. This mimics a ‘closure’ in the traditional sense of the word.

construct_node_metadata()

def construct_node_metadata()

execute()

def execute(
    kwargs,
):

This function is here only to try to streamline the pattern between workflows and tasks. Since tasks call execute from dispatch_execute which is in local_execute, workflows should also call an execute inside local_execute. This makes mocking cleaner.

find_lhs()

def find_lhs()

get_all_tasks()

def get_all_tasks()

Future proof method. Just making it easy to access all tasks (Not required today as we auto register them)

load_task()

def load_task(
    loader_args: typing.List[str],
):

Given the set of identifier keys, should return one Python Task or raise an error if not found

loader_args()

def loader_args(
    settings: flytekit.configuration.SerializationSettings,
    t: flytekit.core.python_auto_container.PythonAutoContainerTask,
):

This is responsible for turning an instance of a task into args that the load_task function can reconstitute.

local_execute()

def local_execute(
    ctx: FlyteContext,
    kwargs,
):

local_execution_mode()

def local_execution_mode()

task_name()

def task_name(
    t: PythonAutoContainerTask,
):

Overridable function that can optionally return a custom name for a given task

Properties

flytekit.clis.sdk_in_container.build.PythonTask

Base Class for all Tasks with a Python native Interface. This should be directly used for task types, that do not have a python function to be executed. Otherwise refer to :py:class:flytekit.PythonFunctionTask.

def PythonTask(
    task_type: str,
    name: str,
    task_config: typing.Optional[~T],
    interface: typing.Optional[flytekit.core.interface.Interface],
    environment: typing.Optional[typing.Dict[str, str]],
    disable_deck: typing.Optional[bool],
    enable_deck: typing.Optional[bool],
    deck_fields: typing.Optional[typing.Tuple[flytekit.deck.deck.DeckField, ...]],
    kwargs,
):

Methods

compile()

def compile(
    ctx: flytekit.core.context_manager.FlyteContext,
    args,
    kwargs,
):

Generates a node that encapsulates this task in a workflow definition.

construct_node_metadata()

def construct_node_metadata()

Used when constructing the node that encapsulates this task as part of a broader workflow definition.

dispatch_execute()

def dispatch_execute(
    ctx: flytekit.core.context_manager.FlyteContext,
    input_literal_map: flytekit.models.literals.LiteralMap,
):

This method translates Flyte’s Type system based input values and invokes the actual call to the executor This method is also invoked during runtime.

• VoidPromise is returned in the case when the task itself declares no outputs.
• Literal Map is returned when the task returns either one more outputs in the declaration. Individual outputs may be none
• DynamicJobSpec is returned when a dynamic workflow is executed

execute()

def execute(
    kwargs,
):

This method will be invoked to execute the task.

find_lhs()

def find_lhs()

get_config()

def get_config(
    settings: flytekit.configuration.SerializationSettings,
):

Returns the task config as a serializable dictionary. This task config consists of metadata about the custom defined for this task.

get_container()

def get_container(
    settings: flytekit.configuration.SerializationSettings,
):

Returns the container definition (if any) that is used to run the task on hosted Flyte.

get_custom()

def get_custom(
    settings: flytekit.configuration.SerializationSettings,
):

Return additional plugin-specific custom data (if any) as a serializable dictionary.

get_extended_resources()

def get_extended_resources(
    settings: flytekit.configuration.SerializationSettings,
):

Returns the extended resources to allocate to the task on hosted Flyte.

get_input_types()

def get_input_types()

Returns the names and python types as a dictionary for the inputs of this task.

get_k8s_pod()

def get_k8s_pod(
    settings: flytekit.configuration.SerializationSettings,
):

Returns the kubernetes pod definition (if any) that is used to run the task on hosted Flyte.

get_sql()

def get_sql(
    settings: flytekit.configuration.SerializationSettings,
):

Returns the Sql definition (if any) that is used to run the task on hosted Flyte.

get_type_for_input_var()

def get_type_for_input_var(
    k: str,
    v: typing.Any,
):

Returns the python type for an input variable by name.

get_type_for_output_var()

def get_type_for_output_var(
    k: str,
    v: typing.Any,
):

Returns the python type for the specified output variable by name.

local_execute()

def local_execute(
    ctx: flytekit.core.context_manager.FlyteContext,
    kwargs,
):

This function is used only in the local execution path and is responsible for calling dispatch execute. Use this function when calling a task with native values (or Promises containing Flyte literals derived from Python native values).

local_execution_mode()

def local_execution_mode()

post_execute()

def post_execute(
    user_params: typing.Optional[flytekit.core.context_manager.ExecutionParameters],
    rval: typing.Any,
):

Post execute is called after the execution has completed, with the user_params and can be used to clean-up, or alter the outputs to match the intended tasks outputs. If not overridden, then this function is a No-op

pre_execute()

def pre_execute(
    user_params: typing.Optional[flytekit.core.context_manager.ExecutionParameters],
):

This is the method that will be invoked directly before executing the task method and before all the inputs are converted. One particular case where this is useful is if the context is to be modified for the user process to get some user space parameters. This also ensures that things like SparkSession are already correctly setup before the type transformers are called

This should return either the same context of the mutated context

sandbox_execute()

def sandbox_execute(
    ctx: flytekit.core.context_manager.FlyteContext,
    input_literal_map: flytekit.models.literals.LiteralMap,
):

Call dispatch_execute, in the context of a local sandbox execution. Not invoked during runtime.

Properties

flytekit.clis.sdk_in_container.build.RunCommand

A click command group for registering and executing flyte workflows & tasks in a file.

def RunCommand(
    args,
    kwargs,
):

Initialize RichGroup class.

Methods

add_command()

def add_command(
    cmd: click.core.Command,
    name: typing.Optional[str],
):

Registers another :class:Command with this group. If the name is not provided, the name of the command is used.

collect_usage_pieces()

def collect_usage_pieces(
    ctx: click.core.Context,
):

Returns all the pieces that go into the usage line and returns it as a list of strings.

command()

def command(
    args: `*args`,
    kwargs: `**kwargs`,
):

A shortcut decorator for declaring and attaching a command to the group. This takes the same arguments as :func:command and immediately registers the created command with this group by calling :meth:add_command.

To customize the command class used, set the :attr:command_class attribute.

.. versionchanged:: 8.1 This decorator can be applied without parentheses.

.. versionchanged:: 8.0 Added the :attr:command_class attribute.

format_commands()

def format_commands(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Extra format methods for multi methods that adds all the commands after the options.

format_epilog()

def format_epilog(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Writes the epilog into the formatter if it exists.

format_help()

def format_help(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Writes the help into the formatter if it exists.

This is a low-level method called by :meth:get_help.

This calls the following methods:

• :meth:format_usage
• :meth:format_help_text
• :meth:format_options
• :meth:format_epilog

format_help_text()

def format_help_text(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Writes the help text to the formatter if it exists.

format_options()

def format_options(
    ctx: click.core.Context,
    formatter: click.formatting.HelpFormatter,
):

Writes all the options into the formatter if they exist.

format_usage()

def format_usage(
    ctx: click.core.Context,
    formatter: click.formatting.HelpFormatter,
):

Writes the usage line into the formatter.

This is a low-level method called by :meth:get_usage.

get_command()

def get_command(
    ctx,
    filename,
):

Given a context and a command name, this returns a :class:Command object if it exists or returns None.

get_help()

def get_help(
    ctx: click.core.Context,
):

Formats the help into a string and returns it.

Calls :meth:format_help internally.

get_help_option()

def get_help_option(
    ctx: click.core.Context,
):

Returns the help option object.

Unless add_help_option is False.

.. versionchanged:: 8.1.8 The help option is now cached to avoid creating it multiple times.

get_help_option_names()

def get_help_option_names(
    ctx: click.core.Context,
):

Returns the names for the help option.

get_params()

def get_params(
    ctx: click.core.Context,
):

get_short_help_str()

def get_short_help_str(
    limit: int,
):

Gets short help for the command or makes it by shortening the long help string.

get_usage()

def get_usage(
    ctx: click.core.Context,
):

Formats the usage line into a string and returns it.

Calls :meth:format_usage internally.

group()

def group(
    args: `*args`,
    kwargs: `**kwargs`,
):

A shortcut decorator for declaring and attaching a group to the group. This takes the same arguments as :func:group and immediately registers the created group with this group by calling :meth:add_command.

To customize the group class used, set the :attr:group_class attribute.

.. versionchanged:: 8.1 This decorator can be applied without parentheses.

.. versionchanged:: 8.0 Added the :attr:group_class attribute.

invoke()

def invoke(
    ctx: click.core.Context,
):

Given a context, this invokes the attached callback (if it exists) in the right way.

list_commands()

def list_commands(
    ctx,
    add_remote: bool,
):

Returns a list of subcommand names in the order they should appear.

main()

def main(
    args: `*args`,
    prog_name: typing.Optional[str],
    complete_var: typing.Optional[str],
    standalone_mode: bool,
    windows_expand_args: bool,
    extra: typing.Any,
):

This is the way to invoke a script with all the bells and whistles as a command line application. This will always terminate the application after a call. If this is not wanted, SystemExit needs to be caught.

This method is also available by directly calling the instance of a :class:Command.

make_context()

def make_context(
    info_name: typing.Optional[str],
    args: `*args`,
    parent: typing.Optional[click.core.Context],
    extra: typing.Any,
):

This function when given an info name and arguments will kick off the parsing and create a new :class:Context. It does not invoke the actual command callback though.

To quickly customize the context class used without overriding this method, set the :attr:context_class attribute.

make_parser()

def make_parser(
    ctx: click.core.Context,
):

Creates the underlying option parser for this command.

parse_args()

def parse_args(
    ctx: click.core.Context,
    args: `*args`,
):

Given a context and a list of arguments this creates the parser and parses the arguments, then modifies the context as necessary. This is automatically invoked by :meth:make_context.

resolve_command()

def resolve_command(
    ctx: click.core.Context,
    args: `*args`,
):

result_callback()

def result_callback(
    replace: bool,
):

Adds a result callback to the command. By default if a result callback is already registered this will chain them but this can be disabled with the replace parameter. The result callback is invoked with the return value of the subcommand (or the list of return values from all subcommands if chaining is enabled) as well as the parameters as they would be passed to the main callback.

Example::

@click.group() @click.option(’-i’, ‘–input’, default=23) def cli(input): return 42

@cli.result_callback() def process_result(result, input): return result + input

shell_complete()

def shell_complete(
    ctx: click.core.Context,
    incomplete: str,
):

Return a list of completions for the incomplete value. Looks at the names of options, subcommands, and chained multi-commands.

to_info_dict()

def to_info_dict(
    ctx: click.core.Context,
):

Gather information that could be useful for a tool generating user-facing documentation. This traverses the entire structure below this command.

Use :meth:click.Context.to_info_dict to traverse the entire CLI structure.

Properties

flytekit.clis.sdk_in_container.build.RunLevelParams

This class is used to store the parameters that are used to run a workflow / task / launchplan.

def RunLevelParams(
    config_file: typing.Optional[str],
    verbose: bool,
    pkgs: typing.List[str],
    project: str,
    domain: str,
    destination_dir: str,
    copy_all: bool,
    copy: typing.Optional[flytekit.constants.CopyFileDetection],
    image_config: flytekit.configuration.ImageConfig,
    service_account: str,
    wait_execution: bool,
    poll_interval: int,
    dump_snippet: bool,
    overwrite_cache: bool,
    envvars: typing.Dict[str, str],
    tags: typing.List[str],
    name: str,
    labels: typing.Dict[str, str],
    annotations: typing.Dict[str, str],
    raw_output_data_prefix: str,
    max_parallelism: int,
    disable_notifications: bool,
    remote: bool,
    limit: int,
    cluster_pool: str,
    execution_cluster_label: str,
    computed_params: flytekit.clis.sdk_in_container.run.RunLevelComputedParams,
    _remote: typing.Optional[flytekit.remote.remote.FlyteRemote],
):

Methods

from_dict()

def from_dict(
    d: typing.Dict[str, typing.Any],
):

options()

def options()

Return the set of base parameters added to every pyflyte run workflow subcommand.

remote_instance()

def remote_instance()

Properties

flytekit.clis.sdk_in_container.build.SerializationSettings

These settings are provided while serializing a workflow and task, before registration. This is required to get runtime information at serialization time, as well as some defaults.

Attributes: project (str): The project (if any) with which to register entities under. domain (str): The domain (if any) with which to register entities under. version (str): The version (if any) with which to register entities under. image_config (ImageConfig): The image config used to define task container images. env (Optional[Dict[str, str]]): Environment variables injected into task container definitions. flytekit_virtualenv_root (Optional[str]): During out of container serialize the absolute path of the flytekit virtualenv at serialization time won’t match the in-container value at execution time. This optional value is used to provide the in-container virtualenv path python_interpreter (Optional[str]): The python executable to use. This is used for spark tasks in out of container execution. entrypoint_settings (Optional[EntrypointSettings]): Information about the command, path and version of the entrypoint program. fast_serialization_settings (Optional[FastSerializationSettings]): If the code is being serialized so that it can be fast registered (and thus omit building a Docker image) this object contains additional parameters for serialization. source_root (Optional[str]): The root directory of the source code.

def SerializationSettings(
    image_config: ImageConfig,
    project: typing.Optional[str],
    domain: typing.Optional[str],
    version: typing.Optional[str],
    env: Optional[Dict[str, str]],
    git_repo: Optional[str],
    python_interpreter: str,
    flytekit_virtualenv_root: Optional[str],
    fast_serialization_settings: Optional[FastSerializationSettings],
    source_root: Optional[str],
):

Methods

default_entrypoint_settings()

def default_entrypoint_settings(
    interpreter_path: str,
):

Assumes the entrypoint is installed in a virtual-environment where the interpreter is

for_image()

def for_image(
    image: str,
    version: str,
    project: str,
    domain: str,
    python_interpreter_path: str,
):

from_dict()

def from_dict(
    kvs: typing.Union[dict, list, str, int, float, bool, NoneType],
    infer_missing,
):

from_json()

def from_json(
    s: typing.Union[str, bytes, bytearray],
    parse_float,
    parse_int,
    parse_constant,
    infer_missing,
    kw,
):

from_transport()

def from_transport(
    s: str,
):

new_builder()

def new_builder()

Creates a SerializationSettings.Builder that copies the existing serialization settings parameters and allows for customization.

schema()

def schema(
    infer_missing: bool,
    only,
    exclude,
    many: bool,
    context,
    load_only,
    dump_only,
    partial: bool,
    unknown,
):

should_fast_serialize()

def should_fast_serialize()

Whether or not the serialization settings specify that entities should be serialized for fast registration.

to_dict()

def to_dict(
    encode_json,
):

to_json()

def to_json(
    skipkeys: bool,
    ensure_ascii: bool,
    check_circular: bool,
    allow_nan: bool,
    indent: typing.Union[int, str, NoneType],
    separators: typing.Tuple[str, str],
    default: typing.Callable,
    sort_keys: bool,
    kw,
):

venv_root_from_interpreter()

def venv_root_from_interpreter(
    interpreter_path: str,
):

Computes the path of the virtual environment root, based on the passed in python interpreter path for example /opt/venv/bin/python3 -> /opt/venv

with_serialized_context()

def with_serialized_context()

Use this method to create a new SerializationSettings that has an environment variable set with the SerializedContext This is useful in transporting SerializedContext to serialized and registered tasks. The setting will be available in the env field with the key SERIALIZED_CONTEXT_ENV_VAR :return: A newly constructed SerializationSettings, or self, if it already has the serializationSettings

Properties

flytekit.clis.sdk_in_container.build.WorkflowCommand

click multicommand at the python file layer, subcommands should be all the workflows in the file.

def WorkflowCommand(
    filename: str,
    args,
    kwargs,
):

Initialize RichGroup class.

Methods

add_command()

def add_command(
    cmd: click.core.Command,
    name: typing.Optional[str],
):

Registers another :class:Command with this group. If the name is not provided, the name of the command is used.

collect_usage_pieces()

def collect_usage_pieces(
    ctx: click.core.Context,
):

Returns all the pieces that go into the usage line and returns it as a list of strings.

command()

def command(
    args: `*args`,
    kwargs: `**kwargs`,
):

A shortcut decorator for declaring and attaching a command to the group. This takes the same arguments as :func:command and immediately registers the created command with this group by calling :meth:add_command.

To customize the command class used, set the :attr:command_class attribute.

.. versionchanged:: 8.1 This decorator can be applied without parentheses.

.. versionchanged:: 8.0 Added the :attr:command_class attribute.

format_commands()

def format_commands(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Extra format methods for multi methods that adds all the commands after the options.

format_epilog()

def format_epilog(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Writes the epilog into the formatter if it exists.

format_help()

def format_help(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Writes the help into the formatter if it exists.

This is a low-level method called by :meth:get_help.

This calls the following methods:

• :meth:format_usage
• :meth:format_help_text
• :meth:format_options
• :meth:format_epilog

format_help_text()

def format_help_text(
    ctx: rich_click.rich_context.RichContext,
    formatter: rich_click.rich_help_formatter.RichHelpFormatter,
):

Writes the help text to the formatter if it exists.

format_options()

def format_options(
    ctx: click.core.Context,
    formatter: click.formatting.HelpFormatter,
):

Writes all the options into the formatter if they exist.

format_usage()

def format_usage(
    ctx: click.core.Context,
    formatter: click.formatting.HelpFormatter,
):

Writes the usage line into the formatter.

This is a low-level method called by :meth:get_usage.

get_command()

def get_command(
    ctx,
    exe_entity,
):

This command uses the filename with which this command was created, and the string name of the entity passed after the Python filename on the command line, to load the Python object, and then return the Command that click should run.

get_help()

def get_help(
    ctx: click.core.Context,
):

Formats the help into a string and returns it.

Calls :meth:format_help internally.

get_help_option()

def get_help_option(
    ctx: click.core.Context,
):

Returns the help option object.

Unless add_help_option is False.

.. versionchanged:: 8.1.8 The help option is now cached to avoid creating it multiple times.

get_help_option_names()

def get_help_option_names(
    ctx: click.core.Context,
):

Returns the names for the help option.

get_params()

def get_params(
    ctx: click.core.Context,
):

get_short_help_str()

def get_short_help_str(
    limit: int,
):

Gets short help for the command or makes it by shortening the long help string.

get_usage()

def get_usage(
    ctx: click.core.Context,
):

Formats the usage line into a string and returns it.

Calls :meth:format_usage internally.

group()

def group(
    args: `*args`,
    kwargs: `**kwargs`,
):

A shortcut decorator for declaring and attaching a group to the group. This takes the same arguments as :func:group and immediately registers the created group with this group by calling :meth:add_command.

To customize the group class used, set the :attr:group_class attribute.

.. versionchanged:: 8.1 This decorator can be applied without parentheses.

.. versionchanged:: 8.0 Added the :attr:group_class attribute.

invoke()

def invoke(
    ctx: click.core.Context,
):

Given a context, this invokes the attached callback (if it exists) in the right way.

list_commands()

def list_commands(
    ctx,
):

Returns a list of subcommand names in the order they should appear.

main()

def main(
    args: `*args`,
    prog_name: typing.Optional[str],
    complete_var: typing.Optional[str],
    standalone_mode: bool,
    windows_expand_args: bool,
    extra: typing.Any,
):

This is the way to invoke a script with all the bells and whistles as a command line application. This will always terminate the application after a call. If this is not wanted, SystemExit needs to be caught.

This method is also available by directly calling the instance of a :class:Command.

make_context()

def make_context(
    info_name: typing.Optional[str],
    args: `*args`,
    parent: typing.Optional[click.core.Context],
    extra: typing.Any,
):