-    ≉  Estimated 4 min read

Elasticsearch Range Query

An accident in the Elasticsearch universe. Instead writing to an daily index it was index to a yearly index. Now I had to check the date range of the documents. The Elasticsearch Date Math is a great help for the Range Query.

Detect Boundaries

First check lower and upper bound

Getting lower bound with sorting on date field

GET fo-log-2017/_search
{
  "_source": "datetime_host", 
  "size": 1,
   "sort": [
    {
      "datetime_host": {
        "order": "asc"
      }
    }
  ]
}

Getting upper bound

GET fo-log-2017/_search
{
  "_source": "datetime_host", 
  "size": 1,
   "sort": [
    {
      "datetime_host": {
        "order": "desc"
      }
    }
  ]
}

Get Docs Count

Check how many documents exist for a specific day

GET fo-log-2017/_search
{
  "size": 0, 
  "query": {
    "range": {
      "datetime_host": {
        "gte": "2017-07-24 00:00",
        "lte": "2017-07-25 00:00",
        "format": "yyyy-MM-dd HH:mm"
      }
    }
  }
}

Example output

{
  "took": 59,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "hits": {
    "total": 9576222,
    "max_score": 0,
    "hits": []
  }
}

Using Date Math

GET fo-log-2017/_search
{
  "size": 0, 
  "query": {
    "range": {
      "datetime_host": {
        "gte": "now/d",
        "lte": "now+1d/d",
        "format": "yyyy-MM-dd"
      }
    }
  }
}
{
  "took": 9,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "hits": {
    "total": 1627667,
    "max_score": 0,
    "hits": []
  }
}

Reindex with Range Query

Now use it to transfer the data to the daily index

POST _reindex
{
  "source": {
    "index": "fo-log-2017",
    "query": {
      "range": {
        "datetime_host": {
          "gte": "2017-07-25 00:00",
          "lte": "2017-07-26 00:00",
          "format": "yyyy-MM-dd HH:mm"
        }
      }
    }
  },
  "dest": {
    "index": "fo-log-2017.07.25"
  }
}

Delete with Range Query

The range query can also be utilized in the Delete By Query API. For example wrong indexed documents of the wrong month.

curl -XPOST "http://localhost:9200/fo-log-2017.05.24/_delete_by_query" -H 'Content-Type: application/json' -d'
{
  "query": {
    "range": {
      "datetime_host": {
        "gte": "2017-07-24 00:00",
        "lte": "2017-07-25 00:00",
        "format": "yyyy-MM-dd HH:mm"
      }
    }
  }
}'

Tan-Vinh Nguyen

Just a coder

Similar Stories


Data

Reindex Subset Data in Elasticsearch

The Elasticsearch Reindex API is a powerful way to index a subset of data from existing data. If you think of a long term statistics solution, you can aggregate data and store the aggregated val... Read on

IT

Elasticsearch Date Processor Pipeline

I write some configuration documents with the Elasticsearch low level Java Rest Client. The documents are missing a timestamp, therefore I define a simple pipeline, which adds the Read on

IT

Import Currency codes into Elasticsearch

Working in the financial business requires to have the currency code master-data accessible for various reasons. The ISO 4217 currency codes can be obtained from the Read on