Diagnosing and Resolving Elasticsearch Index Lifecycle Management Rollover Failures

Automated index rollover stalling is the most common production bottleneck in high-throughput search pipelines. When rollover fails, write queues back up and query latency spikes due to oversized primary shards. Index Lifecycle Management (ILM) directly governs storage efficiency and routing performance. This guide builds on the node and shard model in Elasticsearch fundamentals for engineers and aligns ILM execution with broader Search Engine Selection & Architecture principles. It isolates the exact failure modes and provides immediate remediation steps.

ILM phase timeline from hot to delete An index advances through hot, warm, cold, and delete phases as it ages, with the rollover trigger and min_age thresholds marked. Hot rollover 50gb/30d priority 100 Warm shrink + forcemerge min_age 31d Cold allocate low-cost min_age 90d Delete remove index min_age 180d Index age increases left to right stall here = WAITING_FOR_ALIAS

Prerequisites for ILM Policy Execution

ILM requires a healthy cluster state before any policy execution begins. The search cluster must maintain green or yellow status with active master nodes. Index templates must explicitly define the write alias used for routing. Misconfigured aliases are the primary cause of rollover blocks. You must establish baseline routing standards by reviewing Elasticsearch Fundamentals for Engineers.

Apply the exact template configuration below to bootstrap alias routing.

curl -X PUT "localhost:9200/_index_template/app_logs_template" \
  -H 'Content-Type: application/json' \
  -d '{
    "index_patterns": ["app_logs-*"],
    "template": {
      "settings": {
        "index.lifecycle.name": "app_logs_policy",
        "index.lifecycle.rollover_alias": "app_logs_write"
      },
      "aliases": {
        "app_logs_write": {"is_write_index": true}
      }
    }
  }'

Verify cluster health thresholds before proceeding. Disk watermarks must remain below 85% to prevent automatic read-only blocks.

Structuring Production-Ready ILM Policies

Production policies must define explicit phase transitions with non-overlapping thresholds. Overlapping triggers cause phase deadlocks where the state machine cannot advance. Use the exact JSON structure below for hot, warm, cold, and delete phases.

curl -X PUT "localhost:9200/_ilm/policy/app_logs_policy" \
  -H 'Content-Type: application/json' \
  -d '{
    "policy": {
      "phases": {
        "hot": {
          "actions": {
            "rollover": {"max_size": "50gb", "max_age": "30d", "max_docs": 20000000},
            "set_priority": {"priority": 100}
          }
        },
        "warm": {
          "min_age": "31d",
          "actions": {
            "shrink": {"number_of_shards": 1},
            "forcemerge": {"max_num_segments": 1},
            "allocate": {"number_of_replicas": 1, "require": {"data": "warm"}}
          }
        },
        "cold": {
          "min_age": "90d",
          "actions": {"allocate": {"require": {"data": "cold"}}}
        },
        "delete": {
          "min_age": "180d",
          "actions": {"delete": {}}
        }
      }
    }
  }'

Ensure max_size, max_age, and max_docs do not conflict with cluster capacity limits. The shrink action requires a single primary shard in the target phase.

Step-by-Step Diagnostic Workflow for Stuck Rollovers

Execute the following API sequence to isolate stuck indices. Start by querying the ILM state machine for the target index.

curl -X GET "localhost:9200/_ilm/explain/app_logs-000001?pretty"

Parse the step_info and phase fields. A WAITING_FOR_ALIAS error indicates the write alias is detached or missing. An ILM_POLICY_NOT_FOUND error means the policy was deleted or never attached. A CLUSTER_BLOCK state points to disk watermark violations or read-only indices.

Verify index metrics and alias routing simultaneously.

curl -s "localhost:9200/_cat/indices/app_logs-*?v&h=index,health,status,docs.count,store.size"
curl -s "localhost:9200/_cat/aliases/app_logs_write?v"

Use jq to extract critical failure codes from the explain response.

curl -s "localhost:9200/_ilm/explain/app_logs-000001" | jq '.indices[].step_info.type'

Resolution Paths and Production Rollout

Force the state machine to advance after correcting underlying issues. Retry the failed step immediately.

curl -X POST "localhost:9200/_ilm/retry/app_logs-000001"

Reassign the write alias if it points to a stale index. This operation guarantees zero query downtime.

curl -X POST "localhost:9200/_aliases" -H 'Content-Type: application/json' -d '{
 "actions": [
 { "remove": { "index": "app_logs-000001", "alias": "app_logs_write" } },
 { "add": { "index": "app_logs-000002", "alias": "app_logs_write", "is_write_index": true } }
 ]
}'

Reattach a missing policy directly to the index settings.

curl -X PUT "localhost:9200/app_logs-000001/_settings" -H 'Content-Type: application/json' -d '{
 "index.lifecycle.name": "app_logs_policy",
 "index.lifecycle.rollover_alias": "app_logs_write"
}'

Roll back failed shrink operations by restoring the original shard count. Delete the partially merged index and retry only after confirming disk watermarks are below 85%.

Post-Implementation Validation & Monitoring

Confirm successful rollover by verifying the new index exists and holds the write alias. Monitor shard allocation across node tiers to prevent hot node saturation.

curl -s "localhost:9200/_cat/shards/app_logs-*?v&h=index,shard,prirep,state,node"

Track disk usage trends using the search cluster stats API. Set DevOps alert thresholds at 70% disk utilization per node.

Query phase transition latency to detect policy drift.

curl -s "localhost:9200/_ilm/status"
curl -s "localhost:9200/_cat/indices/app_logs-*?v&h=index,lifecycle.phase,lifecycle.step"

Integrate ILM metrics into your observability stack. Alert on step_info.error fields and lifecycle_date_millis gaps exceeding 15 minutes. Continuous validation prevents silent pipeline degradation.

Aliases are what make the policy invisible to applications

A lifecycle policy rolls indices over constantly, so applications must never name one. Two aliases do the work: a write alias pointing at the current hot index, which rollover moves automatically, and a read alias spanning every index in the set, which lets a single query cover the whole retention window regardless of how many indices it currently comprises.

# The bootstrap index carries both aliases; rollover maintains the write one.
curl -s -X PUT 'localhost:9200/events-000001' -H 'Content-Type: application/json' -d '{
  "aliases": {
    "events-write": { "is_write_index": true },
    "events":       {}
  }
}'

Getting this wrong is the most common lifecycle-management failure and it presents confusingly: writes succeed for weeks and then start landing in an index the policy has already moved to a colder tier, or reads silently stop covering older data because the read alias was never applied to the rolled-over indices. Index templates are what keep the aliases attached automatically as new indices are created, and verifying that on the first rollover — rather than assuming it — saves discovering the gap months later.

What each phase actually costs

The four phases are not merely labels; each one changes what the index can do, and the constraints are easy to discover too late.

Hot is the only phase that accepts writes. Everything routed to a write alias lands here, and the phase’s job is to roll over before the shard grows past a workable size.

Warm stops writes and optimises for read cost. Force-merging to a single segment in this phase is the largest available win — it removes the per-segment overhead from every subsequent query — but it is expensive to perform and irreversible in the sense that the merged segment will never be split again. Shrinking shard count here is also common and equally one-way.

Cold trades latency for cost, typically by moving data to cheaper nodes or converting to a searchable snapshot. Queries still work; they are simply slower, sometimes by an order of magnitude. Anyone querying cold data through the same dashboard as hot data will report the search as broken, which makes it worth surfacing the tier in the UI rather than hiding it.

Delete is the phase teams forget to configure, and the omission is expensive: an index lifecycle policy without a delete phase manages the aging of data that then accumulates forever. Deciding the retention period is a compliance and cost question rather than a technical one, but it must be decided by someone.

What changes at each lifecycle phase Hot accepts writes, warm force-merges and shrinks, cold moves to cheaper storage with slower queries, and delete removes the index. hot accepts writes warm force-merge, shrink cold cheaper, slower delete often omitted warm-phase actions are one-way: a merged or shrunk index cannot be returned to its earlier shape and a policy without a delete phase manages aging data that never goes away
Each phase changes what the index can do. The two most consequential facts are that warm actions are irreversible and that delete is optional but rarely should be.

One further habit: verify the policy is actually progressing rather than assuming it. A policy can be attached, valid, and stuck — waiting on an action that cannot complete because a node lacks the required attribute, or because a shrink cannot allocate. The explain API reports the current step and any error per index, and checking it a day after any policy change catches the stuck case while it is still only a day of unmanaged data.

When lifecycle management is the wrong tool

Index lifecycle management exists for time-series data: logs, metrics, events — corpora where documents arrive continuously, age predictably, and eventually stop being queried. Applied to that shape it is excellent. Applied to a catalog, it is machinery in search of a problem.

The distinguishing question is whether documents are replaced or accumulated. A product catalog replaces: a product’s record is updated in place and the corpus size tracks the number of products, not the passage of time. Rolling that over daily produces many small indices holding overlapping versions of the same entities and makes every query a cross-index search. A log stream accumulates: today’s events are never updated, yesterday’s are queried less, and last quarter’s can move to cheaper storage or be deleted outright.

Which corpora suit lifecycle management Accumulating time-series data suits rollover and tiering, while a catalog of replaced records does not and should use a single index behind an alias. accumulating corpus logs, events, metrics documents never updated → rollover, tier, delete replacing corpus catalogs, users, documents records updated in place → one index behind an alias applying rollover to a replacing corpus scatters versions of one entity across many indices and turns every lookup into a cross-index search
The decision is about document lifetime, not data volume. A large catalog still wants one index; a small log stream still wants rollover.

Choosing the rollover trigger

Rollover can fire on size, document count, or age, and using all three is the norm. Size is the primary trigger because it is what actually bounds shard size; age is the safety net that stops a quiet index from staying open for months; document count is useful only when documents are uniform enough for the count to predict the size.

{
  "policy": { "phases": { "hot": { "actions": { "rollover": {
    "max_primary_shard_size": "30gb",
    "max_age": "7d",
    "max_docs": 50000000
  } } } } }
}

The trigger that fires first wins, which is why max_age should be generous rather than tight: a seven-day age trigger on a stream that would take three weeks to reach 30 GB produces three undersized indices instead of one correctly sized one. Set age to bound the worst case, not to define the normal case.

Rollover triggers and their roles Size is the primary trigger, age is a safety net for quiet periods, and document count is only useful with uniform documents. max_primary_shard_size primary — bounds shard size max_age safety net for quiet streams max_docs only with uniform documents Whichever fires first wins — so a tight age trigger silently overrides your carefully chosen size target.
Three triggers, one winner per rollover. Most misconfigured policies have an age trigger that fires long before the size trigger ever could.