← All posts

ElasticSearch

tech

This is what I know (up to the point of writing) about Elasticsearch:

  • It is based on Apache Lucene (obvious but still surprising).
  • It leverages the Log Structured Merge Tree (LSM Tree) architecture.
  • It uses a mechanism called “circuit breaking” to estimate the memory usage of a request and cancel it if it exceeds the memory limit.
  • Through using it, I have learned more about pipelines. Elasticsearch can use scripts (in Painless or other supported languages) to define custom behavior, which can be applied to many operations, including reindexing.
  • The reindex API can use scripts (or ingest pipelines) to modify documents during the reindexing process; I utilized this to convert index data types.
  • When converting a field from a long to a text type using the reindex strategy, it is important to note that a nested keyword sub-field must be explicitly declared in the new index mapping to allow exact-match searching later (our toolbelt currently searches using this keyword). Because the original long field does not automatically generate this sub-field, it must be added manually.
  • It can automatically map normal string fields to both text and keyword types.
  • A shard in Elasticsearch is a Lucene index, and a Lucene index is broken down into segments. Segments are immutable internal storage elements where the index data is kept. Smaller segments are periodically merged into larger ones to manage the index size and expunge deleted documents.
  • When indexing documents, an updated document is simply inserted into the index’s newest segment. Later, a process called “Segment Merge” consolidates these segments into a smaller number of larger segments by cleaning up documents marked for deletion and removing older versions of updated documents. (See article).
  • When searching, you might need a Point in Time (PIT) to “freeze” a specific data state for consistent search results. However, this holds active file locks on the segments until the PIT expires. While the background merge process still runs, it cannot delete those locked segments from the disk.
  • The Elasticsearch Python client is a thin wrapper around the ES REST API and provides simple Helpers for convenience.