Setting up ILM policies for App Search in Elastic Search

What’s a Rich Text element?

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

Static and dynamic content editing

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to customize formatting for each rich text

Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.

At SpotDraft we use App Search on top of Elastic Search to give our users fast and reliable search. Search is at the core of Contract Management, as we let users search by various AI extracted and manually inputted Key Points on each contract. In this article we discuss how to manage App Search Indexes in Elastic Search.

Identifying the Reason for High Memory Usage

We started seeing issues around high JVM memory load and had 1000s of indexes even though the data size was only in the 100K documents range. Since we were a paying user of Elastic Cloud, we reached out to their support and were told that we had too many shards and we should get consulting support from them to improve this (or fix it ourselves). 

Following their index lifecycle management and ILM documentation, we realized that the issue was that there were a lot of app search logs that were saved causing this heavy memory and disk usage. However there isn’t much documentation online on how to manage indexes for App Search.

Since we were using Elastic.co we can go to Kibana to see what indexes are in the system. Here we can clearly see that a lot of app-search-app-logs-* indexes are present and taking up a lot of room.


List of indexes taking up a lot of room

We can even drill down to each index and what the creation date is and the stats around when it was last queried and written to

Stats for an individual index as seen in Kibana

Setting up ILM Policy Using Kibana

Since we already ship logs elsewhere, we didn’t need to save these logs in elastic. So we can set up an ILM policy to auto delete them after 30 days. Thankfully App Search creates a index template named “app-search-logs” which we can use for this.


Creating a new ILM Policy
Adding a template to an ILM Policy

However this only applies to new indexes. In order to update the existing indexes, we can use the Elastic Search console in Kibana.

Applying ILM policies to existing indexes in bulk

Setting up ILM Policy without Kibana

Same can also be done without Kibana using curl


curl -X PUT "localhost:9200/_ilm/policy/delete-after-15-days?pretty" \
  -H 'Content-Type: application/json' \
  -d'{"policy": {"phases": {"delete": {"min_age": "15d","actions": {"delete": {}}}}}}'


PUT .app-search-app-logs-*/_settings 
{
  "index": {
    "lifecycle": {
      "name": "delete-after-15-days"
    }
  }
}


PUT .app-search-analytics-logs-*/_settings 
{
  "index": {
    "lifecycle": {
      "name": "delete-after-15-days"
    }
  }
}


After making this change our Elastic Search memory usage dropped by 50% and the latency also dropped by 40%.

Happy searching!