MLServer CLI

The MLServer package includes a mlserver CLI designed to help with some of the common tasks involved with a model’s lifecycle. Below, you can find the full list of supported subcommands. Note that you can also get a similar high-level outline at any time by running:

mlserver --help

Commands

mlserver

Command-line interface to manage MLServer models.

mlserver [OPTIONS] COMMAND [ARGS]...

Options

--version

Show the version and exit.

build

Build a Docker image for a custom MLServer runtime.

mlserver build [OPTIONS] FOLDER

Options

-t, --tag <tag>
--no-cache

Arguments

FOLDER

Required argument

dockerfile

Generate a Dockerfile

mlserver dockerfile [OPTIONS] FOLDER

Options

-i, --include-dockerignore

Arguments

FOLDER

Required argument

infer

Execute batch inference requests against V2 inference server (experimental).

mlserver infer [OPTIONS]

Options

-u, --url <url>

URL of the MLServer to send inference requests to. Should not contain http or https.

-m, --model-name <model_name>

Required Name of the model to send inference requests to.

-i, --input-data-path <input_data_path>

Required Local path to the input file containing inference requests to be processed.

-o, --output-data-path <output_data_path>

Required Local path to the output file for the inference responses to be written to.

-w, --workers <workers>
-r, --retries <retries>
-s, --batch-size <batch_size>

Send inference requests grouped together as micro-batches.

-b, --binary-data

Send inference requests as binary data (not fully supported).

-v, --verbose

Verbose mode.

-vv, --extra-verbose

Extra verbose mode (shows detailed requests and responses).

-t, --transport <transport>

Transport type to use to send inference requests. Can be ‘rest’ or ‘grpc’ (not yet supported).

Options:

rest | grpc

-H, --request-headers <request_headers>

Headers to be set on each inference request send to the server. Multiple options are allowed as: -H ‘Header1: Val1’ -H ‘Header2: Val2’. When setting up as environmental provide as ‘Header1:Val1 Header2:Val2’.

--timeout <timeout>

Connection timeout to be passed to tritonclient.

--batch-interval <batch_interval>

Minimum time interval (in seconds) between requests made by each worker.

--batch-jitter <batch_jitter>

Maximum random jitter (in seconds) added to batch interval between requests.

--use-ssl

Use SSL in communications with inference server.

--insecure

Disable SSL verification in communications. Use with caution.

Environment variables

MLSERVER_INFER_URL

Provide a default for --url

MLSERVER_INFER_MODEL_NAME

Provide a default for --model-name

MLSERVER_INFER_INPUT_DATA_PATH

Provide a default for --input-data-path

MLSERVER_INFER_OUTPUT_DATA_PATH

Provide a default for --output-data-path

MLSERVER_INFER_WORKERS

Provide a default for --workers

MLSERVER_INFER_RETRIES

Provide a default for --retries

MLSERVER_INFER_BATCH_SIZE

Provide a default for --batch-size

MLSERVER_INFER_BINARY_DATA

Provide a default for --binary-data

MLSERVER_INFER_VERBOSE

Provide a default for --verbose

MLSERVER_INFER_EXTRA_VERBOSE

Provide a default for --extra-verbose

MLSERVER_INFER_TRANSPORT

Provide a default for --transport

MLSERVER_INFER_REQUEST_HEADERS

Provide a default for --request-headers

MLSERVER_INFER_CONNECTION_TIMEOUT

Provide a default for --timeout

MLSERVER_INFER_BATCH_INTERVAL

Provide a default for --batch-interval

MLSERVER_INFER_BATCH_JITTER

Provide a default for --batch-jitter

MLSERVER_INFER_USE_SSL

Provide a default for --use-ssl

MLSERVER_INFER_INSECURE

Provide a default for --insecure

init

Generate a base project template

mlserver init [OPTIONS]

Options

-t, --template <template>

start

Start serving a machine learning model with MLServer.

mlserver start [OPTIONS] FOLDER

Arguments

FOLDER

Required argument