Skip to main content

Monitoring

Health Checks​

bash
# Cluster health
curl -X GET "localhost:9200/_cluster/health?pretty"
# Node information
curl -X GET "localhost:9200/_nodes?pretty"
# Index statistics
curl -X GET "localhost:9200/_stats?pretty"
# Pending tasks
curl -X GET "localhost:9200/_cluster/pending_tasks?pretty"

Performance Monitoring​

bash

# Monitor thread pools
curl -X GET "localhost:9200/_nodes/thread_pool?pretty"
# Monitor JVM statistics
curl -X GET "localhost:9200/_nodes/stats/jvm?pretty"
# Monitor filesystem stats
curl -X GET "localhost:9200/_nodes/stats/fs?pretty"

Backup and Restore​

json

# Create repository
{
  "type": "fs",
  "settings": {
    "location": "/backup/elasticsearch",
    "compress": true
  }
}
# Create snapshot
{
  "indices": "index_1,index_2",
  "ignore_unavailable": true,
  "include_global_state": false
}

Performance Tuning​

JVM Tuning

bash
# Set heap size (50% of RAM, max 32GB)
-Xms16g
-Xmx16g
# Use G1GC for large heaps
-XX:+UseG1GC
-XX:G1HeapRegionSize=32m
-XX:MaxGCPauseMillis=50

Index Optimization​


yaml
# Index settings for performance
index.refresh_interval: 30s
index.number_of_replicas: 0  # During bulk indexing
index.translog.durability: async
index.translog.sync_interval: 60s

Query Performance​

bash
# Enable slow log monitoring
index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms

Troubleshooting​

Common Issues

High Memory Usage:​

  • Check heap size configuration

  • Monitor GC activity

  • Review index mappings for excessive fields

Slow Search Performance:​

  • Analyze slow query logs

  • Check index statistics

  • Review query patterns

Cluster Stability:​

  • Monitor split-brain scenarios

  • Check network connectivity

  • Review discovery configuration

Log Analysis​

bash

# Main log locations
/var/log/elasticsearch/production-cluster.log
/var/log/elasticsearch/production-cluster_deprecation.log
/var/log/elasticsearch/production-cluster_slow.log
# Key log patterns to monitor
grep "OutOfMemoryError" /var/log/elasticsearch/*.log
grep "CircuitBreakerException" /var/log/elasticsearch/*.log
grep "ElasticsearchTimeoutException" /var/log/elasticsearch/*.log

Integration Examples​

Logstash Configuration​

ruby

input {
  beats {
    port => 5044
  }
}
filter {
  if [fields][logtype] == "apache" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  }
}
output {
  elasticsearch {
    hosts => ["es-node1:9200", "es-node2:9200", "es-node3:9200"]
    index => "logs-%{+YYYY.MM.dd}"
    user => "logstash_user"
    password => "logstash_password"
  }
}

Kibana Integration​

yaml

kibana.yml

server.host: "0.0.0.0" server.port: 5601 elasticsearch.hosts: ["http://es-node1:9200", "http://es-node2:9200"] elasticsearch.username: "kibana_user" elasticsearch.password: "kibana_password"