1. 2017-01-17 - Backup your Elasticsearch data with Amazon S3; Tags: Backup your Elasticsearch data with Amazon S3
    Loading...

    Backup your Elasticsearch data with Amazon S3

    As I mentioned before, how easy it is to backup your Elasticsearch data with the snapshot and restore API, today’s post demonstrates how to backup the data to Amazon S3 file storage.

    Install the plugin

    First you need to install the elasticsearch plugin for that:

    ```bashsudo bin/elasticsearch-plugin install repository-s3

    
    ## Create a user for S3
    
    There are various ways how to access the S3 storage. I demonstrate the simplest one.
    
    Log in into your AWS console and go to Security. Create a user with an access key. I named the user elasticsearch.
    
    ![Add AWS S3 user](https://media.cinhtau.net/elasticsearch/s3-add-user.png){:class="img-responsive"}
    
    And assign the user to the group backup with permission for S3.
    
    ![User permissions](https://media.cinhtau.net/elasticsearch/s3-group-permission-1024x354.png){:class="img-responsive"}
    
    After that AWS will generate the access and secret key for the Elasticsearch user.
    
    ## Configure your S3 access
    
    You need to alter your `elasticsearch.yml` for that. Following settings are exemplary and doesn't represent real values.
    
    ```yaml
    cloud:
        aws:
            access_key: AAAABBBB1234CCCC5678
            secret_key: Ahfk380HqZR7sUYdeH2Xw*ZxyY8fwlF5QVQoxiJ$
            s3.region: eu-central
    

    Do the backup

    Following steps demonstrates the backup process.

    Check plugin

    In the Kibana console you can check if the repository-s3 is installed.

    GET _cat/plugins
    
    23Y9vRH repository-s3 5.1.2
    23Y9vRH x-pack        5.1.2
    

    Now we can define the S3 bucket. Replace my-s3-bucket with your bucket name and maybe replace the region, as you need it. The repository will also be verified.

    PUT _snapshot/s3
    {
      "type": "s3",
      "settings": {
        "bucket": "my-s3-bucket",
        "compress": true,
        "region": "eu-central-1"
      }
    }
    

    To verify it manually:

    POST /_snapshot/s3/_verify
    

    Backup Kibana

    We do a backup and name the snapshot upgrade_512.

    PUT _snapshot/s3/upgrade_512
    {
      "indices": ".kibana",
      "include_global_state": false
    }
    

    Check snapshot status

    GET _snapshot/s3/_all
    {
      "snapshots": [
        {
          "snapshot": "upgrade_512",
          "uuid": "RPUl1FXuRzyt6_pRxEcgWw",
          "version_id": 5010299,
          "version": "5.1.2",
          "indices": [
            ".kibana"
          ],
          "state": "SUCCESS",
          "start_time": "2017-01-15T20:37:47.333Z",
          "start_time_in_millis": 1484512667333,
          "end_time": "2017-01-15T20:37:50.838Z",
          "end_time_in_millis": 1484512670838,
          "duration_in_millis": 3505,
          "failures": [],
          "shards": {
            "total": 1,
            "failed": 0,
            "successful": 1
          }
        }
      ]
    }
    

    Looking in S3 I notice that elasticsearch did the upgrade in the root of the bucket.

    Wrong backup

    This was never intended. We can easily delete it with:

    DELETE _snapshot/s3/upgrade_512
    

    The right definition is to put a base_path to it. This will place the data into the folder elasticsearch.

    PUT _snapshot/s3
    {
      "type": "s3",
      "settings": {
        "bucket": "my-s3-bucket",
        "compress": true,
        "base_path": "elasticsearch",
        "region": "eu-central-1"
      }
    }