Flyte Datacatalog Configuration

Section: application

grpcPort (int)

On which grpc port to serve Catalog

Default Value:

"8081"

grpcServerReflection (bool)

Enable GRPC Server Reflection

Default Value:

"true"

grpcMaxRecvMsgSizeMBs (int)

The max receive message size; if unset defaults to gRPC server default value

Default Value:

"0"

httpPort (int)

On which http port to serve Catalog

Default Value:

"8080"

secure (bool)

Whether to run Catalog in secure mode or not

Default Value:

"false"

readHeaderTimeoutSeconds (int)

The amount of time allowed to read request headers.

Default Value:

"32"

Section: database

host (string)

Default Value:

""

port (int)

Default Value:

"0"

dbname (string)

Default Value:

""

username (string)

Default Value:

""

password (string)

Default Value:

""

passwordPath (string)

Default Value:

""

options (string)

Default Value:

""

debug (bool)

Default Value:

"false"

enableForeignKeyConstraintWhenMigrating (bool)

Whether to enable gorm foreign keys when migrating the db

Default Value:

"false"

maxIdleConnections (int)

maxIdleConnections sets the maximum number of connections in the idle connection pool.

Default Value:

"10"

maxOpenConnections (int)

maxOpenConnections sets the maximum number of open connections to the database.

Default Value:

"100"

connMaxLifeTime (config.Duration)

sets the maximum amount of time a connection may be reused

Default Value:

1h0m0s

postgres (database.PostgresConfig)

Default Value:

dbname: postgres
debug: false
host: localhost
options: sslmode=disable
password: postgres
passwordPath: ""
port: 30001
readReplicaHost: localhost
username: postgres

sqlite (database.SQLiteConfig)

Default Value:

file: ""

config.Duration

Duration (int64)

Default Value:

1h0m0s

database.PostgresConfig

host (string)

The host name of the database server

Default Value:

localhost
readReplicaHost (string)

The host name of the read replica database server

Default Value:

localhost
port (int)

The port name of the database server

Default Value:

"30001"
dbname (string)

The database name

Default Value:

postgres
username (string)

The database user who is connecting to the server.

Default Value:

postgres
password (string)

The database password.

Default Value:

postgres
passwordPath (string)

Points to the file containing the database password.

Default Value:

""
options (string)

See http://gorm.io/docs/connecting_to_the_database.html for available options passed, in addition to the above.

Default Value:

sslmode=disable
debug (bool)

Whether or not to start the database connection with debug mode enabled.

Default Value:

"false"

database.SQLiteConfig

file (string)

The path to the file (existing or new) where the DB should be created / stored. If existing, then this will be reused, else a new will be created

Default Value:

""

Section: datacatalog

storage-prefix (string)

StoragePrefix specifies the prefix where DataCatalog stores offloaded ArtifactData in CloudStorage. If not specified, the data will be stored in the base container directly.

Default Value:

metadata

metrics-scope (string)

Scope that the metrics will record under.

Default Value:

datacatalog

profiler-port (int)

Port that the profiling service is listening on.

Default Value:

"10254"

heartbeat-grace-period-multiplier (int)

Number of heartbeats before a reservation expires without an extension.

Default Value:

"3"

max-reservation-heartbeat (config.Duration)

The maximum available reservation extension heartbeat interval.

Default Value:

10s

Section: logger

show-source (bool)

Includes source code location in logs.

Default Value:

"false"

mute (bool)

Mutes all logs regardless of severity. Intended for benchmarks/tests only.

Default Value:

"false"

level (int)

Sets the minimum logging level.

Default Value:

"3"

formatter (logger.FormatterConfig)

Sets logging format.

Default Value:

type: json

logger.FormatterConfig

type (string)

Sets logging format type.

Default Value:

json

Section: otel

type (string)

Sets the type of exporter to configure [noop/file/jaeger/otlpgrpc/otlphttp].

Default Value:

noop

file (otelutils.FileConfig)

Configuration for exporting telemetry traces to a file

Default Value:

filename: /tmp/trace.txt

jaeger (otelutils.JaegerConfig)

Configuration for exporting telemetry traces to a jaeger

Default Value:

endpoint: http://localhost:14268/api/traces

otlpgrpc (otelutils.OtlpGrpcConfig)

Configuration for exporting telemetry traces to an OTLP gRPC collector

Default Value:

endpoint: http://localhost:4317

otlphttp (otelutils.OtlpHttpConfig)

Configuration for exporting telemetry traces to an OTLP HTTP collector

Default Value:

endpoint: http://localhost:4318/v1/traces

sampler (otelutils.SamplerConfig)

Configuration for the sampler to use for the tracer

Default Value:

parentSampler: always
traceIdRatio: 0.01

otelutils.FileConfig

filename (string)

Filename to store exported telemetry traces

Default Value:

/tmp/trace.txt

otelutils.JaegerConfig

endpoint (string)

Endpoint for the jaeger telemetry trace ingestor

Default Value:

http://localhost:14268/api/traces

otelutils.OtlpGrpcConfig

endpoint (string)

Endpoint for the OTLP telemetry trace collector

Default Value:

http://localhost:4317

otelutils.OtlpHttpConfig

endpoint (string)

Endpoint for the OTLP telemetry trace collector

Default Value:

http://localhost:4318/v1/traces

otelutils.SamplerConfig

parentSampler (string)

Sets the parent sampler to use for the tracer

Default Value:

always
traceIdRatio (float64)

Default Value:

"0.01"

Section: storage

type (string)

Sets the type of storage to configure [s3/minio/local/mem/stow].

Default Value:

s3

connection (storage.ConnectionConfig)

Default Value:

access-key: ""
auth-type: iam
disable-ssl: false
endpoint: ""
region: us-east-1
secret-key: ""

stow (storage.StowConfig)

Storage config for stow backend.

Default Value:

{}

container (string)

Initial container (in s3 a bucket) to create -if it doesn't exist-.'

Default Value:

""

enable-multicontainer (bool)

If this is true, then the container argument is overlooked and redundant. This config will automatically open new connections to new containers/buckets as they are encountered

Default Value:

"false"

cache (storage.CachingConfig)

Default Value:

max_size_mbs: 0
target_gc_percent: 0

limits (storage.LimitsConfig)

Sets limits for stores.

Default Value:

maxDownloadMBs: 2

defaultHttpClient (storage.HTTPClientConfig)

Sets the default http client config.

Default Value:

headers: null
timeout: 0s

signedUrl (storage.SignedURLConfig)

Sets config for SignedURL.

Default Value:

{}

storage.CachingConfig

max_size_mbs (int)

Maximum size of the cache where the Blob store data is cached in-memory. If not specified or set to 0, cache is not used

Default Value:

"0"
target_gc_percent (int)

Sets the garbage collection target percentage.

Default Value:

"0"

storage.ConnectionConfig

endpoint (config.URL)

URL for storage client to connect to.

Default Value:

""
auth-type (string)

Auth Type to use [iam,accesskey].

Default Value:

iam
access-key (string)

Access key to use. Only required when authtype is set to accesskey.

Default Value:

""
secret-key (string)

Secret to use when accesskey is set.

Default Value:

""
region (string)

Region to connect to.

Default Value:

us-east-1
disable-ssl (bool)

Disables SSL connection. Should only be used for development.

Default Value:

"false"

config.URL

URL (url.URL)

Default Value:

ForceQuery: false
Fragment: ""
Host: ""
OmitHost: false
Opaque: ""
Path: ""
RawFragment: ""
RawPath: ""
RawQuery: ""
Scheme: ""
User: null

url.URL

Scheme (string)

Default Value:

""
Opaque (string)

Default Value:

""
User (url.Userinfo)

Default Value:

null
Host (string)

Default Value:

""
Path (string)

Default Value:

""
RawPath (string)

Default Value:

""
OmitHost (bool)

Default Value:

"false"
ForceQuery (bool)

Default Value:

"false"
RawQuery (string)

Default Value:

""
Fragment (string)

Default Value:

""
RawFragment (string)

Default Value:

""

storage.HTTPClientConfig

headers (map[string][]string)

Default Value:

null
timeout (config.Duration)

Sets time out on the http client.

Default Value:

0s

storage.LimitsConfig

maxDownloadMBs (int64)

Maximum allowed download size (in MBs) per call.

Default Value:

"2"

storage.SignedURLConfig

stowConfigOverride (map[string]string)

Default Value:

null

storage.StowConfig

kind (string)

Kind of Stow backend to use. Refer to github/flyteorg/stow

Default Value:

""
config (map[string]string)

Configuration for stow backend. Refer to github/flyteorg/stow

Default Value:

{}