ML Inference
Category: AI & ML
Node id: ml-inference
Calls external ML model endpoints for predictions (supports TensorFlow Serving, Triton, and generic JSON)
What it does
- Generic ML model endpoint integration via HTTP
- Support for TensorFlow Serving, Triton, and JSON formats
- Single and batch prediction modes
- Custom HTTP headers for authentication
- Configurable request timeout
Use cases
- Running predictions against hosted ML models
- Batch inference for patient risk scoring
- Integrating custom NLP or image classification models
- Calling TensorFlow Serving or Triton endpoints
Runtime
- Performance tier:
variable
Input ports
| Name | Type | Required | Description |
|---|---|---|---|
features | json | No | Feature map for a single prediction |
feature_array | json | No | Array of feature sets for batch prediction |
raw_input | text | No | Raw text input for text-based models |
Output ports
| Name | Type | Required | Description |
|---|---|---|---|
result | json | No | Main result containing predictions, model_name, http_status, and any additional endpoint response fields |
success | boolean | No | Whether the operation succeeded |
Settings
Basic
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Model Name | model_name | string | No | "" | Name of the model to invoke | |
| Input Format | input_format | enum | No | json | json, tensorflow_serving, triton | Request body format |
Auth
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Model Endpoint | model_endpoint | string | Yes | URL of the ML model serving endpoint | ||
| API Key | api_key | secret | No | Bearer token for the model endpoint | ||
| Custom Headers | custom_headers | json | No | Additional HTTP headers as key-value pairs |
Advanced
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Timeout (seconds) | timeout_seconds | number | No | 120 | min 1 | Request timeout in seconds |
Errors
| Code | Description |
|---|---|
ENDPOINT_REQUIRED | model_endpoint configuration is missing |
INPUT_REQUIRED | features, feature_array, or raw_input must be provided |
INFERENCE_FAILED | HTTP request to the ML endpoint failed |
HTTP_ERROR | ML endpoint returned an HTTP error status (4xx/5xx) |
Runtime notes
- Result shape depends on the ML endpoint; predictions key is common but not guaranteed
- If the response is not valid JSON, predictions is returned as a raw string
- model_name and http_status are injected into the result map alongside endpoint fields
- At least one of features, feature_array, or raw_input must be provided
See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.
Quelque chose n'est pas clair ? Envoyez vos commentaires sur la documentation par courriel.