Clean up AMI and related snapshots

Playbook that removes old AMI's and the related snapshots that were created using the ec2_ami module with machines tagged MakeImage=true
2018-06-09 16:01:40 +02:00 · 2018-06-09 16:01:40 +02:00 · 3fa8629123
parent 38b5df4372
commit 3fa8629123
4 changed files with 276 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@ -1,3 +1,4 @@
+*.retry
 .vagrant
 vagrant-ssh-config
 package_updates/roles/debian-update
--- a/cleanup_ami_snapshots/README.md
+++ b/cleanup_ami_snapshots/README.md
@ -0,0 +1,140 @@
+# Ansible playbook to remove oldest AMI and attached snapshots
+
+Tested with Ansible version: 2.5.4
+
+## Requirements
+This playbook needs the Python AWS SDK installed (both boto & boto3)
+
+    ```
+    sudo pip install boto boto3
+    # or
+    pip install --user boto boto3
+    # or use pipenv for a virtualenv setup..
+    ```
+
+
+## Variables
+- enable_debug: True / False (prints some info about registered variables)
+- aws_profile: the AWS profile name
+- log_destination: path to the file where actions will be logged.
+- ami_tag: AMI's with given tag will be filtered out. The oldest is then removed
+
+## AWS authentication
+### AWS profiles
+In this example we create an aws profile with the name `aws-ansible`.
+
+Create an AWS profile by editing `~/.aws/credentials` and add the following
+stanza:
+```
+[aws-ansible]
+aws_access_key_id = ***
+aws_secret_access_key = ***
+region = eu-west-1
+```
+
+Set the Playbook variable `aws_profile` to the name of this profile
+
+example:
+```
+  vars:
+    aws_profile: aws-ansible
+    ...
+```
+
+
+## Testing
+In the `test` folder of this repository you can find an Ansible playbook that
+creates some test AMI's.
+
+- AMI-001 (oldest)
+- AMI-002
+- AMI-003 (newest)
+
+When running the cleanup playbook AMI-001 should be removed first. On subsequent runs AMI-002 should be removed and then AMI-003.
+If no AMI's with the tag `DestroyImage: true` are available. The playbook skips
+all tasks.
+
+When AMI's with multiple snapshots are deregistered, all the attached snapshots
+will be deleted.
+
+**NOTE:** The deregistration of AMI's takes some time. So subsequent
+runs of the cleanup playbook short after each other will result in the playbook
+trying to deregister and delete snapshots of the same AMI. This should be an
+idempotent action.
+
+This is not the case if AMI's are tagged
+
+## Cleanup Playbook
+### prerequisites
+The Playbook requires that you set the `AWS_REGION` environment variable because
+some modules used in this playbook require this even if it's set in the aws
+profile.
+
+Example:
+```
+ export AWS_REGION=eu-west-1
+```
+
+### Run the Playbook
+```
+$ ansible-playbook playbook.yaml
+
+PLAY [Remove the oldest AMI and associated snapshots] **************************************************************************************
+
+TASK [Gathering Facts] *********************************************************************************************************************
+ok: [localhost]
+
+TASK [Register the AWS_REGION environment variable.] ***************************************************************************************
+ok: [localhost]
+
+TASK [Fail if the AWs_REGION environemtn var is not set] ***********************************************************************************
+skipping: [localhost]
+
+TASK [Gather facts about all AMIs with given tag ] *****************************************************************************************
+ok: [localhost]
+
+TASK [Check if log file already exists] ****************************************************************************************************
+ok: [localhost]
+
+TASK [Create log file] *********************************************************************************************************************
+skipping: [localhost]
+
+TASK [debug] *******************************************************************************************************************************
+skipping: [localhost]
+
+TASK [oldest ami] **************************************************************************************************************************
+ok: [localhost]
+
+TASK [debug] *******************************************************************************************************************************
+skipping: [localhost]
+
+TASK [Deregister AMI.] *********************************************************************************************************************
+changed: [localhost]
+
+TASK [LOG action] **************************************************************************************************************************
+changed: [localhost]
+
+TASK [Cleanup AMI snapshots] ***************************************************************************************************************
+changed: [localhost] => (item={u'ebs': {u'encrypted': False, u'snapshot_id': u'snap-0b4d8ef6c1bc098d1', u'delete_on_termination': True, u'volume_type': u'gp2', u'volume_size': 8}, u'device_name': u'/dev/xvda'})
+changed: [localhost] => (item={u'ebs': {u'encrypted': False, u'snapshot_id': u'snap-09c25bbe838912ced', u'delete_on_termination': True, u'volume_type': u'standard', u'volume_size': 8}, u'device_name': u'/dev/sdb'})
+
+TASK [LOG action] **************************************************************************************************************************
+changed: [localhost] => (item={u'ebs': {u'encrypted': False, u'snapshot_id': u'snap-0b4d8ef6c1bc098d1', u'delete_on_termination': True, u'volume_type': u'gp2', u'volume_size': 8}, u'device_name': u'/dev/xvda'})
+changed: [localhost] => (item={u'ebs': {u'encrypted': False, u'snapshot_id': u'snap-09c25bbe838912ced', u'delete_on_termination': True, u'volume_type': u'standard', u'volume_size': 8}, u'device_name': u'/dev/sdb'})
+
+PLAY RECAP *********************************************************************************************************************************
+localhost                  : ok=9    changed=4    unreachable=0    failed=0
+```
+
+### Logging
+Info about the cleanup actions are kept in a log file (default:
+/tmp/ansible-ami-cleanup.log)
+
+```
+2018-06-09T07:52:44Z AMI CLEANUP - Deregistered AMI: ami-577e7d2e
+2018-06-09T07:52:44Z AMI CLEANUP - Removed snapshots: snap-00c90dc0cfa23ae75
+2018-06-09T07:52:44Z AMI CLEANUP - Removed snapshots: snap-0ebfc8f7defefc968
+2018-06-09T08:10:58Z AMI CLEANUP - Deregistered AMI: ami-f77f7c8e
+2018-06-09T08:10:58Z AMI CLEANUP - Removed snapshots: snap-0b4d8ef6c1bc098d1
+2018-06-09T08:10:58Z AMI CLEANUP - Removed snapshots: snap-09c25bbe838912ced
+```
--- a/cleanup_ami_snapshots/cleanup-ami-snapshots.yaml
+++ b/cleanup_ami_snapshots/cleanup-ami-snapshots.yaml
@ -0,0 +1,85 @@
+- name: Remove the oldest AMI and associated snapshots
+  hosts: localhost
+  connection: local
+  gather_facts: True
+
+  vars:
+    enable_debug: False
+    aws_profile: aws-ansible
+    log_destination: /tmp/ansible-ami-cleanup.log
+    ami_tag:
+      "tag:DestroyImage": 'true'
+
+  tasks:
+    - name: Register the AWS_REGION environment variable.
+      set_fact:
+        aws_region_env_var: "{{ lookup('env', 'AWS_REGION') }}"
+      register: aws_region_env_var
+
+    - name: Fail if the AWs_REGION environemtn var is not set
+      fail:
+        msg: "The AWS_REGION environment variable is not set"
+      when: not aws_region_env_var
+
+    - name: Gather facts about all AMIs with given tag.
+      ec2_ami_facts:
+        profile: "{{ aws_profile }}"
+        owners: self
+        filters: "{{ ami_tag }}"
+      register: ami_list
+
+    - name: Check if log file already exists
+      stat:
+        path: "{{ log_destination }}"
+      register: logfile
+
+    - name: Create log file
+      file:
+        state: touch
+        path: "{{ log_destination  }}"
+      when: logfile.stat.exists == False
+
+    - name: debug
+      debug:
+        msg: "{{ ami_list }}"
+      when: enable_debug
+
+    - name: oldest ami
+      set_fact:
+        oldest_ami: "{{ ami_list.images | sort(attribute='creation_date') | first}}"
+      when: ami_list.images
+
+    - name: debug
+      debug:
+        msg: "{{ oldest_ami }}"
+      when: enable_debug and ami_list.images
+
+    - name: Deregister AMI.
+      ec2_ami:
+        profile: "{{ aws_profile }}"
+        image_id: "{{ oldest_ami.image_id }}"
+        state: absent
+        # Bug in deleting snapshots : https://github.com/ansible/ansible/issues/39541
+        #delete_snapshot: yes
+      when: ami_list.images
+
+    - name: LOG action
+      lineinfile:
+        line: "{{ ansible_date_time.iso8601 }} AMI CLEANUP - Deregistered AMI: {{ oldest_ami.image_id  }}"
+        dest: "{{ log_destination }}"
+      when: ami_list.images
+
+    - name: Cleanup AMI snapshots
+      ec2_snapshot:
+        profile: "{{ aws_profile }}"
+        snapshot_id: "{{ item.ebs.snapshot_id }}"
+        state: absent
+      with_items: "{{ oldest_ami.block_device_mappings }}"
+      when: ami_list.images
+
+    - name: LOG action
+      lineinfile:
+        line: "{{ ansible_date_time.iso8601  }} AMI CLEANUP - Removed snapshots: {{ item.ebs.snapshot_id }}"
+        dest: "{{ log_destination }}"
+      with_items: "{{ oldest_ami.block_device_mappings  }}"
+      when: ami_list.images
--- a/cleanup_ami_snapshots/test/deploy_test.yaml
+++ b/cleanup_ami_snapshots/test/deploy_test.yaml
@ -0,0 +1,50 @@
+- name: Create test AMI's (this might take some time)
+  hosts: localhost
+  connection: local
+  gather_facts: True
+
+  vars:
+    aws_profile: aws-ansible
+
+  tasks:
+    - name: Register the AWS_REGION environment variable.
+      set_fact:
+        aws_region_env_var: "{{ lookup('env', 'AWS_REGION') }}"
+      register: aws_region_env_var
+
+    - name: Fail if the AWs_REGION environemtn var is not set
+      fail:
+        msg: "The AWS_REGION environment variable is not set"
+      when: not aws_region_env_var
+
+    - name: Create ec2 instance to create AMI's from
+      ec2_instance:
+        profile: "{{ aws_profile }}"
+        state: present
+        name: "ansible-test-instance"
+        image_id: ami-ca0135b3
+        volumes:
+          - device_name: "/dev/sdb"
+            ebs:
+              volume_size: 8
+        network:
+          assign_public_ip: false
+        tags:
+          MakeImage: 'true'
+      register: test_instance
+
+    - name: Create AMI's from instance
+      ec2_ami:
+        profile: "{{ aws_profile }}"
+        state: present
+        name: "AMI-00{{ item }}"
+        instance_id: "{{ test_instance.instance_ids[0] }}"
+        wait: yes
+      with_sequence: start=1 end=3
+
+    - name: Cleanup temp instance
+      ec2_instance:
+        profile: "{{ aws_profile }}"
+        state: terminated
+        filters:
+          instance-id: "{{ test_instance.instance_ids[0] }}"