Showing posts with label Lambda. Show all posts
Showing posts with label Lambda. Show all posts
Optimize the Cost by Shutting down EC2 Instances using Lambda
Recently, I have written an article about launching Instances on the AutoScaling Group in which I have configured CloudWatch alarms to trigger to launch EC2 instances based on aggregate CPU usage within the autoscaling group (used scaling policy).
find the previous article here
Then, I came across that if EC2 instances in AWS are managed through Auto Scaling Groups, it is easy to schedule startup and shutdown of those instances and It opens up a question that a cloud infrastructure always comes at a cost and how to Optimize the Cost of our AWS full Architecture?
Scenario: Developers working on a testing phase of their project and to reduce the cost. The manager decided to Shut down all the AWS Infrastructure services after office hours (shutting down the instances at nights and launch them back on the working mornings). So it becomes cumbersome for the cloud team to re-provision all the environment on a daily basis.
Though, I decided to automate this process using lambda and cloud watch rule.
Note: As usual, I used IaaC (terraform) to all the processes.
Let's dive in,
Steps:
1. Create AWS IAM Role and Policies for Lambda
2. Create AWS LAMBDA function to start and stop instances
3. Use Cron AWS Cloudwatch Rules to trigger Lambda function
Find the full code here
1. Create AWS IAM Role and Policies for Lambda
I created an IAM role for assuming lambda and then attached an inline policy to access autoscaling group
2. Create AWS LAMBDA function to start and stop instances
The below function creates AWS Lambda function and autoscaling.py lambda code has attached to the function.
Now lambda function has created and let's create both start and stop CloudWatch rules
Not please write down below scale-up-daown-lamda.tf file
To Stop the autoscaling EC2 Instance
As you can see from the above code, schedule_expression = "0 0 22 1/1 * ? *" is a Cron expression and which specify at which day(s) and time this event should trigger the event (in my case, 10 pm).
See this website for a handy Cron calculator.
To stop the instance, we have set the MinSize and DesiredSize to 0. So I passed min, desired value as 0 to my lambda function as Jason format below.
To Sart the autoscaling EC2 Instance
The code Exactly the same as the stop instances.schedule_expression = "0 0 7 1/1 * ? *" Cron expression specified to start the autoscaling group at 7am moring.
And MinSize and DesiredSize have given as we configured autoscaling from the beginning. Meaning that it brings back the EC2 instances as Exactly we created the autoscaling group earlier at launch first launch.
Now we can easily we can stop our EC2 instances at night to save money on our cloud bill and automate this process using lambda. To make sure your autoscaling is done properly, we should monitor the EC2 instance with Cloudwatch.
Find the full repo here
That's pretty much it guys! Thank you
find the previous article here
Then, I came across that if EC2 instances in AWS are managed through Auto Scaling Groups, it is easy to schedule startup and shutdown of those instances and It opens up a question that a cloud infrastructure always comes at a cost and how to Optimize the Cost of our AWS full Architecture?
Scenario: Developers working on a testing phase of their project and to reduce the cost. The manager decided to Shut down all the AWS Infrastructure services after office hours (shutting down the instances at nights and launch them back on the working mornings). So it becomes cumbersome for the cloud team to re-provision all the environment on a daily basis.
Though, I decided to automate this process using lambda and cloud watch rule.
Note: As usual, I used IaaC (terraform) to all the processes.
Let's dive in,
Steps:
1. Create AWS IAM Role and Policies for Lambda
2. Create AWS LAMBDA function to start and stop instances
3. Use Cron AWS Cloudwatch Rules to trigger Lambda function
Find the full code here
1. Create AWS IAM Role and Policies for Lambda
I created an IAM role for assuming lambda and then attached an inline policy to access autoscaling group
# /iam.tf
#create a role for lamda access aws resouces
resouce "aws_iam_role" "lamda-iam"{
#role name
name="lamda-asg"
#assume role to a services
assume_role_policy =<<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
# create policy for role to assume autoscaling
resource "aws_iam_role_policy" "autoscaling-lambda-policy"{
name = "autoscaling_lambda_policy"
role = "${aws_iam_role.lamda-iam.id}"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:*"
],
"Resource": "*"
}
]
}
EOF
}
2. Create AWS LAMBDA function to start and stop instances
The below function creates AWS Lambda function and autoscaling.py lambda code has attached to the function.
# /scale-up-daown-lamda.tf
# create lambda
# to invoke autoscaling group
resource "aws_lambda_function" "lambda_test"{
function_name = "lambda_stop_start_autoscaling_instances"
filename = "./autoscaling.zip/autoscaling.py"
role = "${aws_iam_role.lamda-iam.arn}"
handler = "autoscaling.handler"
runtime = "python2.7"
}
Lambda function# /autoscaling.py
import boto3
import os
# Boto Connection
# aws_region, autoscaling-group-name,min,max and desired capacity has to be passed
asg = boto3.client('autoscaling','os.environ[aws_region]')
def lambda_handler(event, context):
response = asg.update_auto_scaling_group(AutoScalingGroupName=os.environ['asg_name'],MinSize=os.environ['min'],DesiredCapacity=os.environ['desired'],MaxSize=os.environ['max'])
Not please write down below scale-up-daown-lamda.tf file
To Stop the autoscaling EC2 Instance
# /scale-up-daown-lamda.tf
#...
#_______stop instances at 10pm everyday_________________
resource "aws_cloudwatch_event_rule" "stop" {
name = "everyday-10"
description = "stop instances everyday 10 pm"
schedule_expression = "0 0 22 1/1 * ? *"
event_pattern = <<PATTERN
{
"aws_region" : "eu-west-1",
"asg_name" : "webapp",
"min" : "0",
"desired" : "0",
"max" : "10"
}
PATTERN
}
resource "aws_cloudwatch_event_target" "lambda-stop" {
rule = "${aws_cloudwatch_event_rule.stop.name}"
target_id = "lambda"
arn = "${aws_lambda_function.lambda_test.arn}"
}
resource "aws_lambda_permission" "lamda-permision" {
statement_id = "AllowExecutionbycloudwatchrule"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.lambda_test.function_name}"
principal = "events.amazonaws.com"
source_arn = "${aws_cloudwatch_event_rule.stop.arn}"
}
#_______end of stop instances at 10pm_________________
See this website for a handy Cron calculator.
To stop the instance, we have set the MinSize and DesiredSize to 0. So I passed min, desired value as 0 to my lambda function as Jason format below.
event_pattern = <<PATTERN
{
"aws_region" : "eu-west-1",
"asg_name" : "webapp",
"min" : "0",
"desired" : "0",
"max" : "10"
}
PATTERN
Now, the stop rule has been created. resource "aws_cloudwatch_event_target" "lambda-stop".. used to select a target to invoke when an event matches. (literally trigger the lambda function on cloud watch rule event).To Sart the autoscaling EC2 Instance
# /scale-up-daown-lamda.tf
#...
#_______start instances at 7am everyday_________________
resource "aws_cloudwatch_event_rule" "start" {
name = "everyday-7am"
description = "start instances everyday 7am"
schedule_expression = "0 0 7 1/1 * ? *"
event_pattern = <<PATTERN
{
"aws_region" : "eu-west-1",
"asg_name" : "webapp",
"min" : "2",
"desired" : "5",
"max" : "10"
}
PATTERN
}
resource "aws_cloudwatch_event_target" "lambda-start" {
rule = "${aws_cloudwatch_event_rule.start.name}"
target_id = "lambda1"
arn = "${aws_lambda_function.lambda_test.arn}"
}
resource "aws_lambda_permission" "lamda-permision" {
statement_id = "AllowExecutionbycloudwatchrule"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.lambda_test.function_name}"
principal = "events.amazonaws.com"
source_arn = "${aws_cloudwatch_event_rule.start.arn}"
}
And MinSize and DesiredSize have given as we configured autoscaling from the beginning. Meaning that it brings back the EC2 instances as Exactly we created the autoscaling group earlier at launch first launch.
#luanch instance as we configured autoscaling from the begining
event_pattern = <<PATTERN
{
"aws_region" : "eu-west-1",
"asg_name" : "webapp",
"min" : "2",
"desired" : "5",
"max" : "10"
}
PATTERN
Now we can easily we can stop our EC2 instances at night to save money on our cloud bill and automate this process using lambda. To make sure your autoscaling is done properly, we should monitor the EC2 instance with Cloudwatch.
Find the full repo here
That's pretty much it guys! Thank you
Deploy Sever-less Application using Terraform (AWS API Gateway, Lambda and DynamoDB)
Nowadays, A popular approach to running "serverless" web applications is to implement the application functionality as one or more functions in AWS Lambda and then expose these for public consumption using Amazon API Gateway without having to think about infrastructure.
With my style, I am going through a procedure to deploy such a web application using terraform.
Note: In order to follow this guide you will need an AWS account and to have Terraform installed. Configure your credentials so that Terraform is able to act on your behalf.
or
you can simply clone my repo and place your access key in terraform.tfvars file.
Before moving onto the deployment refer below figure to understand how AWS full architecture works.
Our ideas is simple, User fille the feedback form and click submit, on submission write to the DynamoDB (post).
Refer the form below
Steps involved in the deployment
1. Create DynamoDB
2. Create AWS IAM Role to assume on behalf you
3. Create a Lamda function
3.1. Create put Item function
4. Create Api Gateway
4.1. Create api Resourse
4.3 Create api method
4.4. Create invoke function to invoke lamda
5. Deploy API and Allowing API Gateway to Access Lambda.
6. Copy the API ARN and paste to the form post URL to test (POSTMAN can also be used)
6. Copy the API ARN and paste to the form post URL to test (POSTMAN can also be used)
Note find the full repo here
1. Create DynamoDB
Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. – from AWS
Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. – from AWS
Create DynamoDB with primary as email
[fayasak@controller]$ cat dynamo.tfresource "aws_dynamodb_table" "dynamodb"{
name="mywebsite"
hash_key="email"
attribute=[
{
name="email"
type="s"
}
]
}
2. Create AWS IAM Role to assume on behalf of you
Before specifying the Lambda Functions we have to create permissions for our functions to use. This makes sure that our lambda function has permission to access DynamoDB resources.
Note: I have created a policy full access policy for dynamoDB. if you want you can change policy as per you want.
EX: "dynamodb:Scan","dynamodb:BatchWriteItem","dynamodb:PutItem"
Before specifying the Lambda Functions we have to create permissions for our functions to use. This makes sure that our lambda function has permission to access DynamoDB resources.
[fayasak@controller]$ cat iam.tf#create a role for lamda access aws resoucesresouce "aws_iam_role" "lamda-iam"{ #role name name="lamda-dynamodb" #assume role to a services assume_role_policy =<<EOF { "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Principal": { "Service": "lambda.amazonaws.com" }, "Effect": "Allow", "Sid": "" } ] }EOF}# create policy for role to assume dynamodbresource "aws_iam_role_policy" "dynamodb-lambda-policy"{ name = "dynamodb_lambda_policy" role = "${aws_iam_role.lamda-iam.id}"
policy = <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "dynamodb:*" ], "Resource": "${aws_dynamodb_table.dynamodb.arn}" } ] } EOF}Note: I have created a policy full access policy for dynamoDB. if you want you can change policy as per you want.
EX: "dynamodb:Scan","dynamodb:BatchWriteItem","dynamodb:PutItem"
Subscribe to:
Posts (Atom)