Elastic search- Rolling index or time base indexes

Common use case for rolling index is logging. Most applications generate a lot of logs  and its convenient to search them easily. It also useful to remove old logs for availability. 

Template and Aliasis

A template is a scheme setup. 

{
"template": "log_*", //any index which is created with log_ name will automatically inherit the //template from the mapping block and the new indexes will be added to the alias log
"aliases": {
"log": {
}
},
"settings": {
"index": {
"number_of_shards": 5,
"number_of_replicas": 1
}
},
"mappings": {
"log_event": {
"properties": {
"log_name": {
"type": "string"
},
"log_text": {
"type":"string"
},
"log_date": {
"type": "date"
}
}
}
}
}

Any indexes which is created with the naming schema should inherit the schema below. In this example indexes which is created dynamically with log_* will inherit the mapping information we have. 

Alias is a shortcut name you can install infornt of multiple indexes. In this way if we write a query against the alias the query will be automatically circulated to all the indexes assingned to the alias. 

Dynamic index is created when you insert a document into a index that doesnt exist yet. In that situation elastic search will create a index for you automatically. And takes the best guess of the property field types. So instead of letting elastic search to guess at the property field types, we set up a template so that it knows what field type to use. Now if I insert a document with correct properties to the index name that matches the wildcard elastic search will create it on demand. 

In post man I am going to insert a log document to a index that doesnt exist yet. but it does match the template we setup. 

http://localhost:9200/log_2013/log_event

{
"log_name": "my_app",
"log_text": "null pointer exception"
"article_date": "2015-10-13t00:00:00z"
}


output

{
  "_index": "log_2013",
  "_type": "log_event",
  "_id": "AVnES-4-bB-_0C_ORt6J",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "created": true
}

http://localhost:9200/log_2014/log_event

{
"log_name": "my_app",
"log_text": "null pointer exception",
"article_date": "2014-10-13t00:00:00z"
}

output

{
  "_index": "log_2014",
  "_type": "log_event",
  "_id": "AVnET4DsbB-_0C_ORt6K",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "created": true
}


http://localhost:9200/log/

{
"took": 8,
"timed_out": false,
"_shards": {
"total": 10,
"successful": 10,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "log_2014",
"_type": "log_event",
"_id": "AVnET4DsbB-_0C_ORt6K",
"_score": 1,
"_source": {
"log_name": "my_app",
"log_text": "null pointer exception",
"article_date": "2014-10-13t00:00:00z"
}
}
,
{
"_index": "log_2013",
"_type": "log_event",
"_id": "AVnES-4-bB-_0C_ORt6J",
"_score": 1,
"_source": {
"log_name": "my_app",
"log_text": "null pointer exception",
"article_date": "2015-10-13t00:00:00z"
}
}
]
}
}



http://localhost:9200/log_2013

{
"took": 3,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "log_2013",
"_type": "log_event",
"_id": "AVnES-4-bB-_0C_ORt6J",
"_score": 1,
"_source": {
"log_name": "my_app",
"log_text": "null pointer exception",
"article_date": "2015-10-13t00:00:00z"
}
}
]
}
}

This is a very powerful way of adding and removing indexes in the cluster without the user notices.