Elasticsearch & Logstash & Kibana

ELK is a popular stack consisting of Elasticsearch, Logstash, and Kibana used for log collection, indexing, searching, and visualization.

Elasticsearch is a distributed search and analytics engine that stores data in JSON documents and supports full-text and structured queries.

Logstash is a data processing pipeline that collects, transforms, and forwards logs or events to Elasticsearch.

Kibana is the visualization and exploration layer for Elasticsearch data.

It is commonly used for centralized logging, operational analytics, metrics exploration, and troubleshooting.

Because it provides a scalable, queryable way to store and inspect large amounts of logs and events.

An index is a logical collection of documents in Elasticsearch.

A document is a JSON object stored in Elasticsearch.

A field is a key in a JSON document, such as timestamp, status, or message.

A shard is a partition of an index used to distribute data across nodes.

A replica is a copy of a shard used for redundancy and availability.

They improve fault tolerance and allow failover if a node goes down.

A cluster is a set of Elasticsearch nodes that work together to store and index data.

A node is a single Elasticsearch instance running in a cluster.

A master node is responsible for cluster state management and orchestration.

A data node stores and indexes the actual data.

Ingestion is the process of sending data into Elasticsearch.

Indexing is the process of parsing, storing, and making documents searchable.

Searching means querying Elasticsearch for documents or aggregations.

An analyzer is a component that tokenizes and normalizes text before indexing or searching.

Full-text search indexes text and enables keyword matching, relevance scoring, and tokenization.

A mapping defines the fields and data types in an index.

Earlier versions used document types, but newer versions use indices and mappings without types.

Because it defines how fields are indexed and searched.

A string field usually uses text or keyword mappings.

A keyword field is used for exact-match filtering and aggregations.

A text field is analyzed for full-text search.

A numeric field stores numbers such as integer, float, or long.

A date field stores timestamps and supports date range queries.

A boolean field stores true/false values.

A nested field stores arrays of objects with independent query semantics.

A geo field stores geographic coordinates or shapes.

A query is a request for documents that match certain conditions.

A filter is usually a query that can be cached and used for exact matching.

A bool query combines clauses like must, should, must_not, and filter.

A match query performs full-text matching on a field.

A term query matches exact values, often used for keyword fields.

A range query matches values between lower and upper bounds, often used for timestamps and numeric fields.

A wildcard query matches text patterns using wildcards like * or ?.

A regex query matches data with regular-expression patterns.

Sorting controls the order of search results.

A score is the relevance score assigned to a document by Elasticsearch.

Relevance is how well a document matches a query relative to others.

A composite aggregation combines bucket results from multiple dimensions.

A bucket aggregation groups documents by certain criteria.

A metric aggregation computes statistics like average, sum, min, max, or percentiles.

A histogram groups documents by a time or value interval.

A date histogram groups data into time buckets such as minute, hour, or day.

A term aggregation groups documents by a field value.

A stats aggregation calculates count, min, max, sum, and average for a field.

A percentile aggregation shows thresholds like p95 or p99 of a metric.

A log event is a single record of an event or log message.

Structured logging stores logs as structured JSON rather than plain text.

It makes log filtering and analysis easier and more precise.

A Logstash pipeline is a sequence of inputs, filters, and outputs.

An input plugin reads data from a source, such as file, syslog, beats, Kafka, or HTTP.

A filter plugin transforms or enriches the event.

An output plugin sends data to a destination such as Elasticsearch.

Beats is a lightweight data shipper used to collect logs and metrics from hosts or applications.

Filebeat ships log files to Logstash or Elasticsearch.

Metricbeat ships system and application metrics.

Heartbeat checks service availability and liveness.

Packetbeat captures network packet data.

Because Beats are lightweight, agent-based log shippers.

An index pattern tells Kibana which indices to search and visualize.

Kibana Discover lets users search and inspect log data interactively.

A Dashboard is a set of visualizations showing metrics, logs, and trends.

A Visualization is a chart or graph built from Elasticsearch queries.

Search in Kibana is based on Elasticsearch queries and filters.

It is how Logstash collects log streams, such as file input or Beats input.

A codec defines how data is encoded or decoded in the pipeline.

A grok filter parses unstructured text using patterns to extract fields.

Because many logs are human-readable text and need structured extraction.

A mutate filter changes fields, adds, renames, or removes values.

A date filter parses dates into proper timestamp fields.

A geoip filter enriches events with geographic metadata based on IP addresses.

A drop filter can drop certain events based on conditions.

A condition often uses if or else blocks based on field values.

The output plugin sends processed log data to Elasticsearch for indexing.

Because containers and clusters create huge volumes of logs and metrics that need centralized processing.

Application logs, system logs, access logs, error logs, and security logs.

Because distributed systems produce logs across many hosts and containers.

Log volume is the amount of log data generated by services and infrastructure.

Retention is how long logs are kept before being deleted or archived.

Because keeping all logs indefinitely is expensive and may violate compliance needs.

A log pipeline is the path from source to storage and visualization.

It is the speed at which Elasticsearch can ingest and make data searchable.

Cluster health indicates whether the Elasticsearch cluster is green, yellow, or red.

All primary shards are assigned and available.

All primary shards are assigned but some replicas are unassigned.

At least one primary shard is unassigned, meaning some data is unavailable.

It indicates whether the cluster is healthy and data is being indexed safely.

It is the assignment of shards to nodes in the cluster.

To spread load across nodes and maximize availability.

It describes tiered storage strategies for different service levels or retention needs.

An index template defines default mappings and settings for new indices.

They reduce mistakes and keep indices consistent.

A rollover index is a new index created when an existing one reaches size or time thresholds.

To keep indices manageable and improve search performance.

A name pattern such as logs-2026.10.* organizes documents by time.

Bulk indexing sends many documents in one request for better performance.

They improve ingestion throughput and reduce overhead.

The refresh interval controls when new documents become visible to searches.

Because it trades ingestion speed for search latency.

Elasticsearch search is near real-time, meaning documents become visible shortly after indexing.

It is the volume of documents per unit time that Elasticsearch can ingest.

Lucene is the search engine beneath Elasticsearch.

Because Lucene provides the underlying indexing, search, and scoring capabilities.

The query DSL is the Elasticsearch domain-specific language for writing queries.

A filter context is used for exact matching and often caches results.

A query context computes relevance and scoring.

Because filters are efficient and often reusable, while queries compute relevance.

A script query runs a script to decide whether a document matches.

They are slower and more complex than standard queries.

A function score query modifies the relevance score using custom functions.

A geospatial query filters or scores documents by geographic location.

A nested query searches within nested objects.

Because it supports document structures where arrays of objects are semantically grouped.

An alias points to one or more indices, which helps with time-based index rollover and zero-downtime updates.

They let you redirect searches to new indices without changing queries.

A mapping conflict occurs when a field is mapped differently across indices or updates.

They can break indexing or cause unexpected search behavior.

Text is meant for full-text search; keyword is for exact matching and sorting.

Because tokens are normalized and lowercased so searches are more useful.

Stemming reduces words to their root form, such as "running" to "run".

Tokenization splits text into searchable pieces or tokens.

A stop word is a common word removed from indexing, such as "the" or "and".

Field data is used for aggregations and sorting on certain field types.

A runtime field is computed at query time rather than stored in the index.

A scripted field is calculated using scripts during a query.

Because they run at query time and can be computationally expensive.

It queries documents from Elasticsearch.

It counts matching documents.

It allows high-throughput ingestion of many documents in one request.

It provides compact human-readable cluster, node, and index information.

Because it is convenient for quick cluster diagnostics.

Monitoring tracks cluster health, shard utilization, memory, and indexing throughput.

A cluster state contains topology and metadata about nodes, indices, and shards.

Node role may be master, data, ingest, coordinating-only, or combinations.

To improve scalability and isolate workloads.

An ingest pipeline transforms documents before they reach an index.

A Logstash pipeline can have multiple filters such as grok, mutate, geoip, and date.

Because it normalizes, enriches, and structures the data before indexing.

It can misparse or corrupt field values, leading to poor indexing and search results.

It means some data is intentionally not stored in Elasticsearch because it is irrelevant or too noisy.

To discard irrelevant or duplicate events before they reach Elasticsearch.

A transformation reorganizes or enriches raw logs into structured fields.

Telemetry is the collection of operational data like logs, events, metrics, and traces.

It links related log events across services or components for troubleshooting.

A trace ID allows associating log events with a specific request or transaction.

They aid debugging distributed systems and correlate logs across services.

A multi-line log is a single event spread across several lines.

Because they may otherwise be indexed as multiple unrelated events.

A JSON log is structured and easy to parse and query.

They are easier to index and query than plain text logs.

It parses JSON payloads directly into structured key-value fields.

It reads comma-separated values, often used for structured exports.

A `jdbc` input reads database rows into Logstash and Elasticsearch.

A Beats input collects events from Beats agents such as Filebeat, Metricbeat, and Heartbeat.

It may be due to bad field names, parse failures, or syntax issues in Logstash config.

It sends processed documents to one or more Elasticsearch endpoints.

Because fields and mappings are crucial for search quality and aggregation accuracy.

It drops data matching a condition, often used for noisy or sensitive logs.

It changes fields such as adding, renaming, or converting values.

The time associated with the event, often derived from the log or the ingestion time.

Because time-based queries and dashboards depend on a valid event time.

It parses dates into Elasticsearch-compatible timestamp values.

It turns raw text into structured fields such as service, host, severity, and message.

It is the destination of processed data, often Elasticsearch.

It happens when one filter transforms fields in a way that breaks later processing.

It adds metadata such as environment, hostname, or region to logs.

Because it makes them more context-rich and easier to operationalize.

A log signature is a pattern or fingerprint that identifies a recurring event pattern.

It connects events from multiple systems related to the same transaction or incident.

A dashboard alert identifies anomalies or threshold breaches in cluster data.

A threshold rule triggers an alert when a metric or log count exceeds a value.

TTL (time-to-live) defines how long a document should stay before expiration.

To manage storage cost and lifecycle of the logs.

Archival stores old logs in cheaper storage for later or compliance-driven access.

Because search speed, cost, and retention needs differ across data age and access frequency.

Sizing is the process of choosing hardware and shard counts based on expected data volume and query load.

More shards improve parallelism but increase overhead and management cost.

Because uneven distribution can create hot spots and slow queries.

Elasticsearch heap size must be tuned carefully; too high or too low can hurt performance.

Elasticsearch runs on the JVM and is sensitive to heap, GC, and memory pressure.

A short refresh interval makes search more real-time but increases indexing overhead.

Backpressure occurs when the pipeline cannot keep up with event rates.

By buffering, scaling out, rate limiting, or rebalancing data pipelines.

Durability means data is stored safely and can survive node or shard failures.

They protect against node failure and can help cluster recovery.

It moves shards across nodes to keep the cluster balanced and healthy.

It involves good mappings, efficient filters, reduced field data, and proper index design.

It is the time taken to complete a search query.

They help answer business and operational questions like “which service failed the most?” or “what is the p99 latency?”

They describe the latency thresholds for the slowest 5% or 1% of requests.

Because average latency can hide spikes and bad tail behavior.

A hot shard is a shard receiving disproportionate traffic or load.

A hotspot is a node or shard receiving much more traffic than its peers.

• cluster health
• shard allocation
• index lifecycle
• disk pressure
• mapping drift
• ingest backlog
• query performance

ELK is not just log storage; it is an operational analytics platform that turns raw event streams into searchable, filterable, and visualizable evidence of system health, behavior, and errors.