elasticsearchRecently AWS announced support for Elasticsearch 5.1 in their Elasticsearch Service. Today, I tried to upgrade an existing CloudFormation stack, previously using Elasticsearch 2.3, to the new version and, after a very long wait, CloudFormation rolled back the stack with the following error: “Creating Elasticsearch Domain did not stabilize“. Here’s what I did to solve it.

First, I made sure it is actually possible to even create a working Elasticsearch 5.1 Domain in AWS. It is. You can create a domain using AWS Console and that domain will reach an “Active” state.

Then, I used AWS CLI describe-elasticsearch-domain call to compare the failed domain created by CloudFormation (Rollback on failure set to No helps a lot with debugging). From the comparison it looked like the domain created by AWS Console had the AdvancedOptions set to default values:

"AdvancedOptions": {
    "rest.action.multi.allow_explicit_index": "true", 
    "indices.fielddata.cache.size": ""
},

while the domain from CloudFormation had that option empty (which was correct, since our template did not set the values):

"AdvancedOptions": {}, 

Since this was the only difference between both domains, I added the AdvancedOptions definition to our CloudFormation template, setting them to the default values, as above. With that change, the stack and the domain was created successfully!

Example of how to set this using troposphere:

elasticsearch.Domain(
    "Elasticsearch",
    ...
    AdvancedOptions={  
        "rest.action.multi.allow_explicit_index": "true",
        "indices.fielddata.cache.size": ""
    }
)

Or directly in CloudFormation’s JSON template:

"Elasticsearch": {
    "Properties": {
        ...
        "AdvancedOptions": {
            "indices.fielddata.cache.size": "",
             "rest.action.multi.allow_explicit_index": "true"
        },
}

Even though those are default values, and CloudFormation documentation specifies this property is not required, it looks like it is not possible to create Elasticsearch 5.1 domain without those being set.


Update: Bug reported

Since this post was published, I have been in contact with AWS support and reported the above issue as a bug. You can follow the progress of the report on AWS Forums: https://forums.aws.amazon.com/thread.jspa?messageID=768527


Remember, you can always use our RSS feed to keep up to date, follow us on Twitter or sign up to our newsletter below (no spam, promise):



6 Comments

  1. Tariq

    Your post saved me hours! Thanks! Looks like AWS still does not have a fix for this bug.

  2. Anonymous

    +1, thank you for taking the time to post this

  3. kkkbmm

    I have the same problem with ES 5.3, spend quite some time until saw this post, thank you very much.

  4. Alan

    Thank you!

  5. Gavin

    +1 – Helpful on quickly figuring out the CFn needed to get 5.3 properly built (replacing 2.3).

  6. Jorge

    Thanks for this! Hours debugging this one…


Leave a comment