flytekit.sensor.sensor_engine
Directory
Classes
Class | Description |
---|---|
SensorEngine |
This is the base class for all async connectors. |
flytekit.sensor.sensor_engine.SensorEngine
This is the base class for all async connectors. It defines the interface that all connectors must implement. The connector service is responsible for invoking connectors. The propeller will communicate with the connector service to create tasks, get the status of tasks, and delete tasks.
All the connectors should be registered in the ConnectorRegistry. Connector Service will look up the connector based on the task type. Every task type can only have one connector.
def SensorEngine()
Methods
Method | Description |
---|---|
create() |
Return a resource meta that can be used to get the status of the task. |
delete() |
Delete the task. |
get() |
Return the status of the task, and return the outputs in some cases. |
get_logs() |
Return the metrics for the task. |
get_metrics() |
Return the metrics for the task. |
create()
def create(
task_template: flytekit.models.task.TaskTemplate,
inputs: typing.Optional[flytekit.models.literals.LiteralMap],
kwarg,
) -> flytekit.sensor.base_sensor.SensorMetadata
Return a resource meta that can be used to get the status of the task.
Parameter | Type |
---|---|
task_template |
flytekit.models.task.TaskTemplate |
inputs |
typing.Optional[flytekit.models.literals.LiteralMap] |
kwarg |
delete()
def delete(
resource_meta: flytekit.sensor.base_sensor.SensorMetadata,
kwargs,
)
Delete the task. This call should be idempotent. It should raise an error if fails to delete the task.
Parameter | Type |
---|---|
resource_meta |
flytekit.sensor.base_sensor.SensorMetadata |
kwargs |
**kwargs |
get()
def get(
resource_meta: flytekit.sensor.base_sensor.SensorMetadata,
kwargs,
) -> flytekit.extend.backend.base_connector.Resource
Return the status of the task, and return the outputs in some cases. For example, bigquery job can’t write the structured dataset to the output location, so it returns the output literals to the propeller, and the propeller will write the structured dataset to the blob store.
Parameter | Type |
---|---|
resource_meta |
flytekit.sensor.base_sensor.SensorMetadata |
kwargs |
**kwargs |
get_logs()
def get_logs(
resource_meta: flytekit.extend.backend.base_connector.ResourceMeta,
kwargs,
) -> flyteidl.admin.agent_pb2.GetTaskLogsResponse
Return the metrics for the task.
Parameter | Type |
---|---|
resource_meta |
flytekit.extend.backend.base_connector.ResourceMeta |
kwargs |
**kwargs |
get_metrics()
def get_metrics(
resource_meta: flytekit.extend.backend.base_connector.ResourceMeta,
kwargs,
) -> flyteidl.admin.agent_pb2.GetTaskMetricsResponse
Return the metrics for the task.
Parameter | Type |
---|---|
resource_meta |
flytekit.extend.backend.base_connector.ResourceMeta |
kwargs |
**kwargs |
Properties
Property | Type | Description |
---|---|---|
metadata_type |
||
task_category |
task category that the connector supports |