Chapter 1. Default logging alerts

CollectorNodeDown

Prometheus could not scrape vector <instance> for more than 10m.

Vector is reporting that Prometheus could not scrape a specific Vector instance.

Critical

DiskBufferUsage

Collectors potentially consuming too much node disk, <value>

Collectors are consuming too much node disk on the host.

Warning

CollectorHigh403ForbiddenResponseRate

High rate of "HTTP 403 Forbidden" responses detected for collector <instance> in namespace <namespace> for output <label>. The rate of 403 responses is <rate> over the last 2 minutes, persisting for more than 5 minutes. This could indicate an authorization issue.

At least 10% of sent requests responded with "HTTP 403 Forbidden" for collector "<intance>" in namespace <namespace> for the output "<output>".

Critical

LokiRequestErrors

{{ $labels.job }} {{ $labels.route }} is experiencing <value>% errors.

At least 10% of requests result in 5xx server errors.

critical

LokiStackWriteRequestErrors

<value>% of write requests from {{ $labels.job }} in <namespace> are returned with server errors.

At least 10% of write requests to the lokistack-gateway result in 5xx server errors.

critical

LokiStackReadRequestErrors

<value>% of query requests from {{ $labels.job }} in <namespace> are returned with server errors.

At least 10% of query requests to the lokistack-gateway result in 5xx server errors.

critical

LokiRequestPanics

{{ $labels.job }} is experiencing an increase of <value> panics.

A panic was triggered.

critical

LokiRequestLatency

{{ $labels.job }} {{ $labels.route }} is experiencing <value>s 99th percentile latency.

The 99th percentile is experiencing latency higher than 1 second.

critical

LokiTenantRateLimit

{{ $labels.job }} {{ $labels.route }} is experiencing 429 errors.

At least 10% of requests are received the rate limit error code.

warning

LokiStorageSlowWrite

The storage path is experiencing slow write response rates.

The storage path is experiencing slow read response rates.

warning

LokiWritePathHighLoad

The write path is experiencing high load.`

The write path is experiencing high load causing backpressure storage flushing.

warning

LokiReadPathHighLoad

The read path is experiencing high load.

The read path has a high volume of queries, causing longer response times.

warning

LokiDiscardedSamplesWarning

Loki in namespace "<namespace>" is discarding samples in the "<tenant>" tenant during ingestion. Samples are discarded because of "<reason>" at a rate of <value> samples per second.

Loki is discarding samples during ingestion because they fail validation.

warning

LokistackSchemaUpgradesRequired

The LokiStack "{{ $labels.stack_name }}" in namespace "<namespace>" is using a storage schema configuration that does not contain the latest schema version. It is recommended to update the schema configuration to update the schema version to the latest

One or more of the deployed LokiStacks contains an outdated storage schema configuration.

warning