stationgroup/ansible-experiments
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

Clean up AMI and related snapshots #2

New issue
Closed
opened 2018-04-09 06:08:58 +00:00 by reelsense · 16 comments
reelsense commented 2018-04-09 06:08:58 +00:00 (Migrated from github.com)
Copy link

I need an Ansible Role/Playbook that can remove old AMI's and the related snapshots that were created using this ec2_ami_module with machines tagged MakeImage=true. I will probably execute this playbook from a scheduled cron job.

  • Non-destructive test mode.(Possible?)
  • Logging.
  • Able to run from a cron job. (untested, but no reason it wont work.)
  • Removes oldest AMI + attached snapshots.
  • Only works if EC2 Instance is tagged MakeImage=true DestroyImage=true.

ℹ️ Maybe this method I'm imagining above isn't the correct method. Maybe it would be safer to create a Playbook that adds something like a DestroyImage=true tag. Then an additional removal Role that only removes things that are tagged DestroyImage=true. I think this sounds safer as I write this...

Need quote and your input on this also.

x-post: https://github.com/stationgroup/ansible-experiments/pull/8

I need an Ansible Role/Playbook that can remove old AMI's and the related snapshots that were created using this [`ec2_ami_module`](http://docs.ansible.com/ansible/latest/modules/ec2_ami_module.html) with machines tagged `MakeImage`=`true`. I will probably execute this playbook from a scheduled _cron_ job. - [x] Non-destructive test mode.(Possible?) - [x] Logging. - [x] Able to run from a _cron_ job. _(untested, but no reason it wont work.)_ - [x] Removes oldest AMI + attached snapshots. - [x] Only works if EC2 Instance is tagged ~~`MakeImage`=`true`~~ `DestroyImage`=`true`. **ℹ️ Maybe this method I'm imagining above isn't the correct method. Maybe it would be safer to create a Playbook that adds something like a `DestroyImage`=`true` tag. Then an additional removal Role that only removes things that are tagged `DestroyImage`=`true`. I think this sounds safer as I write this...** Need quote and your input on this also. --- x-post: https://github.com/stationgroup/ansible-experiments/pull/8
srgvg commented 2018-04-29 11:17:10 +00:00 (Migrated from github.com)
Copy link

I clearly recalled having commented on this issue, and now it seems that is lost, or was not posted.

Most important part of this, is I need a way to test scripts with real AWS AMI info of your infrastructure. Do you see a way how this can happen? At least I'd need an example of return data of the ec2_ami_module

I clearly recalled having commented on this issue, and now it seems that is lost, or was not posted. Most important part of this, is I need a way to test scripts with real AWS AMI info of your infrastructure. Do you see a way how this can happen? At least I'd need an example of return data of the ec2_ami_module
reelsense commented 2018-04-29 16:15:26 +00:00 (Migrated from github.com)
Copy link

I’ll setup a testing account that you can start up instances on.

--
PGP:5EC1494DC59141C0

On Apr 29, 2018, at 04:17, Serge van Ginderachter notifications@github.com wrote:

I clearly recalled having commented on this issue, and now it seems that is lost, or was not posted.

Most important part of this, is I need a way to test scripts with real AWS AMI info of your infrastructure. Do you see a way how this can happen? At least I'd need an example of return data of the ec2_ami_module

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I’ll setup a testing account that you can start up instances on. -- PGP:5EC1494DC59141C0 > On Apr 29, 2018, at 04:17, Serge van Ginderachter <notifications@github.com> wrote: > > I clearly recalled having commented on this issue, and now it seems that is lost, or was not posted. > > Most important part of this, is I need a way to test scripts with real AWS AMI info of your infrastructure. Do you see a way how this can happen? At least I'd need an example of return data of the ec2_ami_module > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub, or mute the thread. >
reelsense commented 2018-05-01 18:11:42 +00:00 (Migrated from github.com)
Copy link

I emailed the credentials for this experiment and setup a server in the Frankfurt region for you with the correct tags.

If you email a public key I can add you to the authorized_keys file or you can create an EC2 instance and AMI with the System Administrator IAM policy your account has. This way I can pay for any EC2 expenses incurred during this.

I emailed the credentials for this experiment and setup a server in the Frankfurt region for you with the correct tags. If you email a public key I can add you to the `authorized_keys` file or you can create an EC2 instance and AMI with the _System Administrator_ IAM policy your account has. This way I can pay for any EC2 expenses incurred during this.
srgvg commented 2018-05-11 17:03:25 +00:00 (Migrated from github.com)
Copy link

Do you have specific AMI's you manage, and a procedure or script when you make snapshots? Or is there a way to quickly simulate having "old" AMI's and snapshots that should be deleted?

Do you have specific AMI's you manage, and a procedure or script when you make snapshots? Or is there a way to quickly simulate having "old" AMI's and snapshots that should be deleted?
reelsense commented 2018-05-12 00:59:58 +00:00 (Migrated from github.com)
Copy link

I'll reply out of band regarding that question.

I'll reply out of band regarding that question.
reelsense commented 2018-06-12 18:18:09 +00:00 (Migrated from github.com)
Copy link

There seems to be a very particular issue that I only hit in production.

fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'ebs'\n\nThe error appears to have been in '/Users/USER/github/public-org-stationgroup/ansible-experiments/cleanup_ami_snapshots/cleanup-ami-snapshots.yaml': line 72, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - name: Cleanup AMI snapshots\n      ^ here\n"}

I've reproduced it using the test AMI playbook that you made and made a screen recording.

VIDEO LINK

https://www.youtube.com/watch?v=V-1W1o4otrI

Can you add what is needed so that I don't error out under the conditions in the video. It seems like a true value followed by an empty value causes it.

There seems to be a very particular issue that I only hit in production. ```yml fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'ebs'\n\nThe error appears to have been in '/Users/USER/github/public-org-stationgroup/ansible-experiments/cleanup_ami_snapshots/cleanup-ami-snapshots.yaml': line 72, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: Cleanup AMI snapshots\n ^ here\n"} ``` I've reproduced it using the test AMI playbook that you made and made a screen recording. # VIDEO LINK https://www.youtube.com/watch?v=V-1W1o4otrI Can you add what is needed so that I don't error out under the conditions in the video. It seems like a `true` value followed by an _empty_ value causes it.
srgvg commented 2018-06-12 19:53:27 +00:00 (Migrated from github.com)
Copy link

Can you confirm me the exact ansible version you use? (just to be sure)

Can you confirm me the exact ansible version you use? (just to be sure)
reelsense commented 2018-06-12 19:54:29 +00:00 (Migrated from github.com)
Copy link
ansible 2.5.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/USER/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
``` ansible 2.5.4 config file = /etc/ansible/ansible.cfg configured module search path = [u'/home/USER/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/dist-packages/ansible executable location = /usr/bin/ansible python version = 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609] ```
reelsense commented 2018-06-12 20:00:53 +00:00 (Migrated from github.com)
Copy link

The above is in production.

This is the version in the video:

ansible 2.5.4
  config file = None
  configured module search path = [u'/Users/USER/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.5.4/libexec/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.15 (default, May  1 2018, 16:44:08) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)]
The above is in production. This is the version in the video: ```bash ansible 2.5.4 config file = None configured module search path = [u'/Users/USER/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/Cellar/ansible/2.5.4/libexec/lib/python2.7/site-packages/ansible executable location = /usr/local/bin/ansible python version = 2.7.15 (default, May 1 2018, 16:44:08) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] ```
reelsense commented 2018-06-13 19:03:51 +00:00 (Migrated from github.com)
Copy link

If solving this issue is more trouble than it's worth, let me know.

It's a real scenario; But maybe it's easier for the playbook to print a message that there is a DestroyImage key with an empty value and fail. And then I would just manually go through and find the missing value and remove the key.

If solving this issue is more trouble than it's worth, **let me know**. It's a real scenario; But maybe it's easier for the playbook to print a message that there is a `DestroyImage` _key_ with an empty _value_ and fail. And then I would just _manually_ go through and find the missing value and remove the key.
vincentvdk commented 2018-06-13 20:42:12 +00:00 (Migrated from github.com)
Copy link

Hi, I'm helping @srgvg on this. Could you provide the following information:
sudo pip show boto
sudo pip show boto3
aws --version

Hi, I'm helping @srgvg on this. Could you provide the following information: `sudo pip show boto` `sudo pip show boto3` `aws --version`
reelsense commented 2018-06-13 21:06:33 +00:00 (Migrated from github.com)
Copy link

sudo pip show boto

---
Metadata-Version: 2.0
Name: boto
Version: 2.48.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
Installer: pip
License: MIT
Location: /home/USER/.local/lib/python2.7/site-packages
Requires: 
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  License :: OSI Approved :: MIT License
  Operating System :: OS Independent
  Topic :: Internet
  Programming Language :: Python :: 2
  Programming Language :: Python :: 2.6
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4

sudo pip show boto3

---
Metadata-Version: 2.0
Name: boto3
Version: 1.7.15
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: UNKNOWN
Installer: pip
License: Apache License 2.0
Location: /home/USER/.local/lib/python2.7/site-packages
Requires: jmespath, s3transfer, botocore
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  Natural Language :: English
  License :: OSI Approved :: Apache Software License
  Programming Language :: Python
  Programming Language :: Python :: 2.6
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4
  Programming Language :: Python :: 3.5
  Programming Language :: Python :: 3.6

aws --version

aws-cli/1.15.4 Python/2.7.12 Linux/4.4.0-104-generic botocore/1.10.15
`sudo pip show boto` ``` --- Metadata-Version: 2.0 Name: boto Version: 2.48.0 Summary: Amazon Web Services Library Home-page: https://github.com/boto/boto/ Author: Mitch Garnaat Author-email: mitch@garnaat.com Installer: pip License: MIT Location: /home/USER/.local/lib/python2.7/site-packages Requires: Classifiers: Development Status :: 5 - Production/Stable Intended Audience :: Developers License :: OSI Approved :: MIT License Operating System :: OS Independent Topic :: Internet Programming Language :: Python :: 2 Programming Language :: Python :: 2.6 Programming Language :: Python :: 2.7 Programming Language :: Python :: 3 Programming Language :: Python :: 3.3 Programming Language :: Python :: 3.4 ``` --- `sudo pip show boto3` ``` --- Metadata-Version: 2.0 Name: boto3 Version: 1.7.15 Summary: The AWS SDK for Python Home-page: https://github.com/boto/boto3 Author: Amazon Web Services Author-email: UNKNOWN Installer: pip License: Apache License 2.0 Location: /home/USER/.local/lib/python2.7/site-packages Requires: jmespath, s3transfer, botocore Classifiers: Development Status :: 5 - Production/Stable Intended Audience :: Developers Natural Language :: English License :: OSI Approved :: Apache Software License Programming Language :: Python Programming Language :: Python :: 2.6 Programming Language :: Python :: 2.7 Programming Language :: Python :: 3 Programming Language :: Python :: 3.3 Programming Language :: Python :: 3.4 Programming Language :: Python :: 3.5 Programming Language :: Python :: 3.6 ``` --- `aws --version` ``` aws-cli/1.15.4 Python/2.7.12 Linux/4.4.0-104-generic botocore/1.10.15 ```
vincentvdk commented 2018-06-14 05:22:52 +00:00 (Migrated from github.com)
Copy link

Could you update these and try to reproduce?
sudo pip install --upgrade botocore
sudo pip install --upgrade boto3
sudo pip install --upgrade boto

Could you update these and try to reproduce? `sudo pip install --upgrade botocore` `sudo pip install --upgrade boto3` `sudo pip install --upgrade boto`
reelsense commented 2018-06-14 17:48:55 +00:00 (Migrated from github.com)
Copy link

Nothing changed from what I showed in the video.

sudo pip show boto

---
Metadata-Version: 2.0
Name: boto
Version: 2.48.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
Installer: pip
License: MIT
Location: /home/USER/.local/lib/python2.7/site-packages
Requires: 
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  License :: OSI Approved :: MIT License
  Operating System :: OS Independent
  Topic :: Internet
  Programming Language :: Python :: 2
  Programming Language :: Python :: 2.6
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4

sudo pip show boto3

---
Metadata-Version: 2.0
Name: boto3
Version: 1.7.38
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: UNKNOWN
Installer: pip
License: Apache License 2.0
Location: /home/USER/.local/lib/python2.7/site-packages
Requires: s3transfer, jmespath, botocore
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  Natural Language :: English
  License :: OSI Approved :: Apache Software License
  Programming Language :: Python
  Programming Language :: Python :: 2.6
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4
  Programming Language :: Python :: 3.5
  Programming Language :: Python :: 3.6

aws --version

aws-cli/1.15.38 Python/2.7.12 Linux/4.4.0-104-generic botocore/1.10.38
Nothing changed from what I showed in the video. `sudo pip show boto` ``` --- Metadata-Version: 2.0 Name: boto Version: 2.48.0 Summary: Amazon Web Services Library Home-page: https://github.com/boto/boto/ Author: Mitch Garnaat Author-email: mitch@garnaat.com Installer: pip License: MIT Location: /home/USER/.local/lib/python2.7/site-packages Requires: Classifiers: Development Status :: 5 - Production/Stable Intended Audience :: Developers License :: OSI Approved :: MIT License Operating System :: OS Independent Topic :: Internet Programming Language :: Python :: 2 Programming Language :: Python :: 2.6 Programming Language :: Python :: 2.7 Programming Language :: Python :: 3 Programming Language :: Python :: 3.3 Programming Language :: Python :: 3.4 ``` --- `sudo pip show boto3` ``` --- Metadata-Version: 2.0 Name: boto3 Version: 1.7.38 Summary: The AWS SDK for Python Home-page: https://github.com/boto/boto3 Author: Amazon Web Services Author-email: UNKNOWN Installer: pip License: Apache License 2.0 Location: /home/USER/.local/lib/python2.7/site-packages Requires: s3transfer, jmespath, botocore Classifiers: Development Status :: 5 - Production/Stable Intended Audience :: Developers Natural Language :: English License :: OSI Approved :: Apache Software License Programming Language :: Python Programming Language :: Python :: 2.6 Programming Language :: Python :: 2.7 Programming Language :: Python :: 3 Programming Language :: Python :: 3.3 Programming Language :: Python :: 3.4 Programming Language :: Python :: 3.5 Programming Language :: Python :: 3.6 ``` --- `aws --version` ``` aws-cli/1.15.38 Python/2.7.12 Linux/4.4.0-104-generic botocore/1.10.38 ```
vincentvdk commented 2018-06-15 05:41:59 +00:00 (Migrated from github.com)
Copy link

Somehow I cannot reproduce this.
video capture: https://youtu.be/Fxm0RSXRLrY

Somehow I cannot reproduce this. video capture: https://youtu.be/Fxm0RSXRLrY
reelsense commented 2018-06-15 05:49:05 +00:00 (Migrated from github.com)
Copy link

Strange... You copied my method exactly in the video. I can reproduce it on 2 separate AWS accounts, on 2 different OS's, in 2 different regions. One with IAM Roles for permissions, and one with IAM access keys (Administrator Access).

Thank you for your help. I'll close this and email @srgvg now.

U-496024757E7019832

Strange... You copied my method exactly in the video. I can reproduce it on 2 separate AWS accounts, on 2 different OS's, in 2 different regions. One with IAM Roles for permissions, and one with IAM access keys _(Administrator Access)_. Thank you for your help. I'll close this and email @srgvg now. --- ` U-496024757E7019832`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
No results found.
Labels
Clear labels   
bug

Something isn't working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
invalid

This doesn't seem right

  
question

Further information is requested

  
todo

   
todo 🗒️

   
wontfix

This will not be worked on

No labels
bug
duplicate
enhancement
good first issue
help wanted
invalid
question
todo
todo 🗒️
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: stationgroup/ansible-experiments#2
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?