Showing posts with label Ansible. Show all posts
Showing posts with label Ansible. Show all posts

Launch EC2 Instances in Autoscaling group with Load balancing

I have been learning AWS Services and writing my experience throughout this blog. I have discussed how to launch an instance using IaaC. eventually, I forget to test out AWS Autoscaling with Loadbalancers.

In this tutorial, I am gonna show up how to Launch EC2 Instances in the Autoscaling group with Load balancing.

What is Load balancers and Autoscaling?
Load balancers: IT is referred to as an ELB which manage and control the flow of inbound requests to a group of targets by distributing these requests evenly across the targeted resource group

Autoscaling: auto-scaling is a mechanism that automatically allows us to scale up/down EC2 resources to meet the demand based on custom-defined metrics and thresholds.

Steps
1. Launch an instance
2. Deploy the application on the instance
3. Build the AMI
4. Create a Launch Configuration
5. Create an Elastic Load Balancer
6. Create and configure an Auto Scaling Group

Find the full code here

Let's dive in,
1. launch an instance
A prerequisite for autoscaling involves building an AMI containing your working application, which will be used to launch new instances. We'll start by launching a new instance onto which we can deploy our application. Create the following files:
---
# group_vars/all.yml

default_region: us-east-1
az1: us-east-1a
az2: us-east-1b
az3: us-east-1c
group_id: sg-abcd1234
instance_type: t2.micro
instance_name: template_ami
volumes:
  - device_name: /dev/sda1
    device_type: gp2
    volume_size: 20
    delete_on_termination: true
key_name: ec2_key
---
# site.yml

- hosts: localhost
  connection: local
  gather_facts: no
  roles:
    - luanch_ec2_instance
---
#luanch_ec2_instance/tasks/main.yml
---
# tasks file for luanch_ec2_instance
  - name: deploy ec2 instance to create ami template
    ec2_ami_facts:
      owners: 099720109477
      filters:
        name: "ubuntu/images/ubuntu-zesty-17.04-*"
    register: ami_result

  - name: launch an instances
    ec2:
      region: "{{ default_region }}"
      keypair: "{{ key_name }}"
      zone: "{{ az1 }}"
      group: "{{ group_id }}"
      image: "{{ ami_result.image_id }}"
      instance_type: "{{ instance_type }}"
      instance_tags:
      Name: "{{ instance_name }}"
      volumes: "{{ volumes }}"
      wait: yes
    register: ec2

  - name: Add new instances to host group
    add_host:
      name: "{{ item.public_dns_name }}"
      groups: "{{ instance_name }}"
      ec2_id: "{{ item.id }}"
    with_items: ec2.instances

  - name: Wait for instance to boot
    wait_for:
      host: "{{ item.public_dns_name }}"
      port: 22
      delay: 30
      timeout: 300
      state: started
    with_items: ec2.instances

2. Deploy the application on the instance
let's use Ansible to deploy our application and start it on our instance.
---
# site.yml

#install nginx on my luanched instances
  - hosts: template_ami
    roles:
      - nginx

---
# nginx/tasks/main.yml
  - name: Install Nginx
    apt:
      pkg: nginx
      state: present
    sudo: yes

  - name: Configure Nginx
    copy:
      src: nginx.conf
      dest: /etc/sites-enabled/default
    sudo: yes

  - name: Enable and start Nginx
    service:
      name: nginx
      enabled: yes
      state: restarted
    sudo: yes

# nginx/files/nginx.conf

server {
  listen 80 default_server;
  location / {
    proxy_pass http://127.0.0.1:8000;
  }
}

Running the playbook each time will launch another instance, deploy our application, and set up Nginx as our web server. If you browse to the newest instance at its hostname, you should see an "Nginx Home" page.
3. Build the AMI
Now that the application is deployed and running, we can use the newly launched instance to build an AMI.
---
# site.yml
  - hosts: control
    connection: local
    gather_facts: no
    roles:
      - create_ami

 
---
# create_ami/tasks/main.yml
# tasks file for create_ami
- name: Create AMI
  ec2_ami:
    region: "{{ default_region }}"
    instance_id: "{{ ec2_id }}"
    name: "webapp-{{ ansible_date_time.iso8601 | regex_replace('[^a-zA-Z0-9]', '-') }}"
    wait: yes
    state: present
  register: ami
Note: Each time the playbook is run, Ansible launches a new instance. At this rate, we'll keep accumulating instances that we don't need, so we will add another role and a new task to locate these instances and terminate them. It will terminate any existing instances immediately afterward.
Terminate old instances
---
# site.yml
 ---
  - name: Find existing instance(s)
    hosts: "tag_Name_ami-build"
    gather_facts: false
    tags: find
    tasks:
      - name: Add to old-ami-build group
        group_by:
          key: old-ami-build
  - hosts: old-ami-build
    roles:
      - terminate_instance


---
# create_ami/tasks/main.yml
- name: Terminate old instance(s)
  ec2:
    instance_ids: "{{ ec2_id }}"
    region: "{{ default_region }}"
    state: absent
    wait: yes

4. Create a Launch Configuration
Now we want to create a new Launch Configuration to describe the new instances that should be launched from this AMI.
---
# site.yml
 - hosts: control
    connection: local
    gather_facts: no
    roles:
      - create_ami
      - luanch_config

---
# luanch_config/tasks/main.yml- name: Create Launch Configuration
  ec2_lc:
    region: "{{ defaul_region }}"
    name: "webapp-{{ ansible_date_time.iso8601 | regex_replace('[^a-zA-Z0-9]', '-') }}"
    image_id: "{{ ami.image_id }}"
    key_name: "{{ key_name }}"
    instance_type: "{{ instance_type }}"
    security_groups: "{{ group_id }}"
    volumes: "{{ volumes }}"
    instance_monitoring: yes

Automate backup using Ansible

I have been reading Ansible and it is a configuration management tool that is widely used in IT industries for automation purposes. Which makes automate and CI?CD process easier.

 In this blog post, I will explain how to automate the backup on folders across a group of servers and storing them in the backup server using Ansible. The whole architecture looks like below,
Let's Dive into the Backup
1. To run, our playbook we need to enable shh-key authentication 
 from Ansible Master to All Hosts (Host A, Host B, Host C)
 from Ansible Master to Backup Server


Note- Run from Ansible mater-shh-keygenssh-copy-id <user name>@<ip address or name of the host>-ssh-copy-id fayasak@Backup Server-ssh-copy-id fayasak@Host A-ssh-copy-id fayasak@Host B-ssh-copy-id fayasak@Host C
2. Enable ssh-key authentication from the backup server to other remote Host
Note- Run from Backup server to host contain file to be backup-shh-keygenssh-copy-id <user name>@<ip address or name of the host>-ssh-copy-id fayasak@Host A-ssh-copy-id fayasak@Host B-ssh-copy-id fayasak@Host C
3. Add remote server names to “/etc/hosts”
It’s always easy to remember the name rather than remembering the IP address, therefore open the “/etc/hosts” file and add entries like below “<ip address> <name of server>”
[Ansible@controller]$ cat etc/hosts127.0.0.1 localhost192.168.8.102 Host A192.168.8.103 Host B192.168.8.104 Host C
Do this in both Ansible Master and Backup server

 4. Assume Host A and Host B belongs to Infra Team and Host C belongs to DB Team. Therefore, I created an inventory file like below
[Ansible@controller]$ cat ansible-backup/inventory[infra]192.168.8.102 Host A192.168.8.103 Host B[db]192.168.8.104 Host C[bk_server]backup server
5. Ansible Synchronize module is used to copy the contents from the host machine to the backup server using pull and push.
(Regardless of “Push” or “Pull” method, SSH authentication must be created between remote hosts and Backup server in order to enable secure transfer of backup files).
The playbook is executed with respect to a backup server, therefore the mode we are using is “Pull”. That means the backup server is pulling the folder contents from remote hosts. If the playbook is executed with respect to remote hosts, then “Push” mode can be utilized to push the folder contents from remote hosts to the backup server
6. To handle multiple folder locations from the same remote server and to handle other parameters relevant to hosts, create a host variable.
[Ansible@controller]$ cat ansible-backup/host_vars/hostAname: host AIp : 192.168.8.102folders: /home/downloads/xxxx/
7. 2 sets of YAML files were created. 1 set of a file is dedicated to each category (DB group and Infra group) and another YAML file is dedicated to taking backup on servers (This YAML file is called as an external YAML file with respect to earlier specified YAML files which were dedicated to each category)
[Ansible@controller]$ cat ansible-backup/db.yml--- - name: This playbook is dedicated to taking backup related to DB servers hosts: DB gather_facts: False tasks: - name: Executing tasks via external yml include_tasks: backup.yml with_items: - "{{ folders }}" #Obtained from the host_vars


[Ansible@controller]$ cat ansible-backup/backup.yml--- - name: Copying files from source server to destination synchronize: src: "{{item}}" dest: "/home/sonic/wordpress/" mode: pull delegate_to: "{{groups['Backup'][0]}}"


Find the full code here

Question guys!
How are you acknowledge that every folder is being backed up since ansible show its code execution?
----create CSV report on each execution against it source folders from remote host---------

Things to know about set_facts Ansible


Hello Everyone!

 I've been reading and learning about AWS automation using Ansible these days!
I have decided to deploy a LAMP stack on AWS and I had an Issue on how to provide different subnets for different stack (EC2 instance) while we create a single role for the subnet using Ansible!

 Let me explain which helps us to connect the dot before digging into the solution.

 Use case: We need to spin up 3 instances and each instance needs to be assigned in different subnets. this will be set up on the fly when instances spun up!

 Solution 
Register and set_facts go hand in hand

Register: registering the result of that command as a variable. When you execute a task and save the return value in a variable to use later tasks, In such case you create a registered variable.

set_facts:  on Ansible Document page, it says that set_facts for host specific and use to register variable against the playbook we are running!

 This is sucks and it provides very least explanation. It's not sufficient for our issue to solve it!

 I was thinking that how do I capture each subnet ID using that module when I create a common subnet role EC2 instance creation playbook.

 Explanation gives you nothing unless you don't look into the code!
Let's dig in,

 Myplaybook
 |---- EC2_webserver.yml 
 |---- EC2_application.yml
 |---- Subnet.yml
 |---- roles
     |--- subnet.yml
 |--- web_subnet.yml
 |--- app_subnet.yml

 Subnet.yml


- import_playbook web_subnet.yml- import_playbook app_subnet.yml
web_subnet.yml

 ---
- name: create subnet for webserver1
  hosts: controt
# ask input from users
  vars_prompt:
    - name: "cidr_block_subnet1"
      prompt: "Enter the CIDR block you want for web server 1 subnet"
      private: no
    - name: "subnet_name1"
      prompt: "Enter the name of the web server 1 subnet"
      private: no
    - name: "subnet_az1"
      prompt: "Enter the availability zone of web server 1 subnet"
      private: no

  tasks:
    - set_fact:
        info: {}

    - name: creating webserver1
      include roles:
        name: ./roles/subnet
      vars:
        cidr_block_subnet: "{{cidr_block_subnet1}}"
        subnet_name: "{{subnet_name1}}"
        az: "{{subnet_az1}}"
 # subnetinfo is used to store subnet infomation
 # when we executing subnet roles    subnetinfo: wb1_subnet

    - name:  print webserver1 output
      debug:
        var: info
app_subnet.yml

 ---
- name: create subnet for appserver
  hosts: controt
# ask input from users
  vars_prompt:
    - name: "cidr_block_subnet2"
      prompt: "Enter the CIDR block you want for app server 1 subnet"
      private: no
    - name: "subnet_name2"
      prompt: "Enter the name of the app server 1 subnet"
      private: no
    - name: "subnet_az1"
      prompt: "Enter the availability zone of app server 1 subnet"
      private: no

  tasks:
    - set_fact:
        info: {}

    - name: creating app1
      include roles:
        name: ./roles/subnet
      vars:
        cidr_block_subnet: "{{cidr_block_subnet1}}"
        subnet_name: "{{subnet_name1}}"
        az: "{{subnet_az1}}"
 # subnetinfo is used to store subnet infomation
# when we executing subnet roles
    subnetinfo: wb1_subnet

    - name:  print app1 output
      debug:
        var: info

 Since it is using the same subnet role, on each playbook we need to capture web server and app server subnet information.
I used subnetifor variable and its value replace with each subnet role output.
Don't worry!
 if you see roles/subnet.yml
you will understand how it replaces subnet information.

 roles/subnet.yml

 ---
# tasks file for subnet
# this can be use as common roles for each LAMP stack
# name provide which server we are creating subnets
- name: creating subnets "{{subnet_name}}"
    ec2_vpc_subnet:
      state: present
      vpc_id: "{{ vpc.vpc.id }}"
      region: "{{default_region}}"
      az: "{{subnet_az}}"
      cidr: "{{cidr_block}}"
      resource_tags:
        Name: "{{subnet_name}}"
register: output


# register stores all the output of each subnet information.
# So!
# I want store each Subnet ID for Each stack
# I found combined jina function which replace variable value
# if so, what if I combined this output to a variable of stack
# then luanch instanes on specific subnets

- name: get subnetid of particular Stack
  set_fact:
    info: ""{{ info | combine({subnetinfo: output}) }}"


# subnetinfo is dynamic variable and it is passed with each stack subnet.yml
# and I am overwriting the output

Note: You can cache a fact set from set_facts the module so that when you execute your playbook next time, it's retrieved from the cache. You can set cacheable to yes to store variables across your playbook executions using a fact cache. You may need to look into precedence strategies used by ansible to evaluate the cacheable facts mentioned in their documentation.

 Find the full code here

 That's pretty much it for today, PEACE!

Thank you!



AWS EC2 intance Automation using Ansible

Hi folks,
It has been a long time since I wrote the last blog. and I have gone through little emotional stress. Though it took me little time to overcome the stress!

 Like I've said in an older blog, I have been learning about automation and CI/CD. I learned terraform a bit and I did spin up EC2 instances, public subnet, a private subnet, Internet gateway, security group and deploy some shell script.

 you can find the blog here

 In this blog, I wanna write about Ansible (spin-up EC2 instances and other key things. same as the previous blog). It was a pretty good experience. you can create a simple YAML file and run playbooks.

 My Idea is,
1. Create VPC
2. Create public and private subnets for each Availability Zones
3. Create an internet gateway
4. Create a public gateway and make associations with public gateway
5. spin up instances on a specific subnet.

 find the full code here

Please refer the Architecture below to understand my idea

 Before moving onto the subject, Ansible requires some requirements to run ansbile module
  
Ansible : sudo pip install ansible
Boto : sudo pip install boto
here are many ways to set our AWS credentials, in this tutorial, we'll create a file under our user home folder (~/.boto):
[Credentials]
AWS_ACCESS_KEY_ID=KID...CWU
AWS_SECRET_ACCESS_EY=3qv...DSP
 AWS CLI : sudo pip install awscli
Note that when launching an EC2 instance with ansible via the ansible ec2 module, the hosts variable should point to localhost and gather_facts should be set to False.

 
- hosts: local  gather_facts: flase  roles:    - vpc
Create VPC

  I did include some other variables under group_vars directory to fetch some essential variables.
To create the VPC ec2_vpc _net module used.
- include_vars: ./group_vars/all.yml

- name: create vpc with 10.0.0.0/16
  ec2_vpc_net:
    name: ansibletest
    cidr_block: 10.0.0.0/16
    region: "{{default_region}}"
    tags:
      Name: ansibletest
    state: present
    aws_access_key: "{{ aws_access_key }}"
    aws_secret_key: "{{ aws_secret_key }}"
    tenancy: default
    dns_hostnames: yes
    dns_support: yes
  register: vpc_info
  #store output of ec2 infroamtion
likewise, we can use available modules to create AWS services. Some other modules are listed below.

 ec2_vpc_subnet (this modules used create subnets)
ec2_vpc_igw (create internet gateway
ec2_vpc_route_table (create routing table and make association for subnets)

The vpc role which I created to deploy those services here

 Create security groups

 security groups especially stand to allow certain traffic to instances.
when I created security groups, I had in mind to spin up instances for deploy LAMP stack. So, I wanted to allow ports related to the LAMP stack.
Later I developed different efficient codes to do this. 

 But here's the security group role which I created.

 

- include_vars: ./group_vars/all.yml

- name: security group with 22,80,443 port enable
  ec2_group:
    name: LAMP_Stack
    description: sg with 22,80,443 port enable
    #right now I'am using vpc_id from ./group_vars/all.yml
    vpc_id: "{{vpc_id}}"
    rules:
      - proto: tcp
        from_port: 80
        to_port: 80
        cidr_ip: 0.0.0.0/0
      - proto: tcp
        from_port: 22
        to_port: 22
        cidr_ip: 0.0.0.0/0

let's move onto final steps
Subscribe to: Posts (Atom)