Squid Proxy for Home Network Using Raspberry Pi
Hi Everyone!
Since the WHO declared the Covid-19 pandemic. Hundreds of millions of people have lived through lockdowns. Just like everyone I have made the abrupt shift to working from home.
When I move back to my home, My family started using the same network and we ended up finishing the internet data allocated to my Home Network. Then, an idea comes in place to use Squid proxy with Raspberry Pi to my home network so that we can reduce the data outbound to the internet.
Scenario: Assume my family members have their own devices and they are watching the same video multiple times. (Multiple people watch the same youtube video).
Solution: Build a Squid proxy server and make a network connection through the proxy (Enable Caching).
**** The Fact is Building our own server cost us more *****
**** I had an unused raspberry pi at my home and decided to use as a proxy to cache content locally****
1. Install Raspberry Pi OS.
2. Mount External Hard Drive to Cache Content
3. Install Squid Proxy
4. Configure Squid Proxy for Content Caching
5. Verify and Connect Home Devices to Proxy
1. Install Raspberry Pi OS
In this blog, I will be talking much more about the Squid proxy server installation and content cache configuration. Steps for Install Raspberry Pi Os Here
2. Mount External Hard Drive to Cache Content
Issue asudo blkid to list the connected storage. You will find the partition for an external drive like /dev/sda
pi@rasberry:~$ sudo blkid
/dev/ramzswap0: TYPE="swap"
/dev/mmcblk0: PTUUID="000995aa" PTTYPE="dos"
/dev/mmcblk0p1: LABEL="RECOVERY" UUID="33EE-A3D4" TYPE="vfat" PARTUUID="000995aa-01"
/dev/mmcblk0p5: LABEL="SETTINGS" UUID="69023f3-1576-4881-9c89-5abdhgc8b271d" TYPE="ext4" PARTUUID="000995aa-05"
/dev/mmcblk0p6: LABEL="boot" UUID="6532-E279" TYPE="vfat" PARTUUID="000995aa-06"
/dev/mmcblk0p7: LABEL="root" UUID="ea79458b-8505-433d-b8da-cceb8d05016c" TYPE="ext4" PARTUUID="000995aa-07"
/dev/sda1: LABEL="WINDOWS" UUID="4CA7-E543" TYPE="ntfs"
Then, we need to mount a folder(Folder to cache content) to our External Drive. Enter the following command to your terminal.
pi@rasberry:~$ mkdir /SquidCache
pi@rasberry:~$ sudo chmod 755 /SquidCache
pi@rasberry:~$ sudo mount /dev/sda1 /SquidCachemkdir /SquidCache Used to create a directory /SquidCache chmod 755 /SquidCache Provide Read/Write Permission to the directory.mount /dev/sda1 /SquidCache Used to mount the created directory to the External drive.
Whenever Rasberry-pi rebooted, It disconnect the mounted external drive. Therefore, The drive needs to be added to fastabSo that, The volume will be persisted to the raspberry-pi on reboot.
Open the fstab and add the below-given line.
pi@rasberry:~$ sudo vim /etc/fstab
UUID=enter_uuid_here /SquidCache auto nofail 0 0
3. Install Squid Proxy
pi@rasberry:~$ sudo apt update
pi@rasberry:~$ sudo apt install squid
4. Configure Squid Proxy for Content Caching
After successful installation navigates to /etc/squid/squid.conf configuration directory and Configure Squid Proxy for Content Caching.
Note: squid.conf file contains lots of configauration. but we only going to change some line which contains acl and cache_dirin the squid.conf file. find those and edit like below. Modifed squid.conf file can be found here. You can replace this file for cache configuration.
pi@rasberry:~$ sudo cp /etc/squid/squid.conf squid.conf.bk
pi@rasberry:~$ sudo vim /etc/squid/squid.conf
A. Squid Proxy has Access Control Lists (acl) that restrict access to the matched the IP range. Allow your home network to use the proxy. (my local ip — 192.168.8.0/24).
pi@rasberry:~$ sudo vim /etc/squid/squid.conf acl localnet src 192.168.8.0/24
By default, the /etc/squid/squid.conf file contains the http_access allow localnet rule that allows using the proxy from all IP ranges specified in localnet ACLs. Note that you must specify all localnet ACLs before the http_access allow localnet rule.
B. The following ACL exists in the default configuration and defines 443 as a port that uses the HTTPS protocol
acl SSL_ports port 443
C. Update the list of acl Safe_ports rules to configure to which ports Squid can establish a connection. For my use case, I configure that clients using the proxy can only access resources on port 21 (FTP), 80 (HTTP), and 443 (HTTPS), keep only the following acl Safe_ports statements in the configuration
acl Safe_ports port 21
acl Safe_ports port 80
acl Safe_ports port 443Note: By default, the configuration contains the http_access deny !Safe_ports rule that defines access denial to ports that are not defined in Safe_ports ACLs.
D. Configure the cache type, the path to the cache directory, the cache size, and further cache type-specific settings in the cache_dir parameter
cache_dir ufs /SquidCache 20000 16 256
Squid uses the ufs cache type.
Squid stores its cache in the /SquidCache directory.
The cache grows up to 20 GB.
Squid creates 16 level-1 sub-directories in the /var/spool/squid/ directory.
Squid creates 256 sub-directories in each level-1 directory.
If you do not set a cache_dir directive, Squid stores the cache in memory.
Now save and quit the editor and reload the squid services
pi@rasberry:~$ sudo systemctl restart squid
5. Verify and Connect Home Devices to Proxy
To verify that the proxy works correctly, download a web page using the curl utility
pi@rasberry:~$ curl -O -L "https://www.redhat.com/index.html" --proxy "127.0.0.1:8080"If curl does not display any error and the index.html file was downloaded to the current directory, the proxy works.
Now connect your home device to the proxy or raspberry pi and enjoy speedily internet.
That's pretty much it guys and PEACE
Things to know about set_facts Ansible
Hello Everyone!
I've been reading and learning about AWS automation using Ansible these days!
I have decided to deploy a LAMP stack on AWS and I had an Issue on how to provide different subnets for different stack (EC2 instance) while we create a single role for the subnet using Ansible!
Let me explain which helps us to connect the dot before digging into the solution.
Use case: We need to spin up 3 instances and each instance needs to be assigned in different subnets. this will be set up on the fly when instances spun up!
Solution
Register and set_facts go hand in hand
Register: registering the result of that command as a variable. When you execute a task and save the return value in a variable to use later tasks, In such case you create a registered variable.
set_facts: on Ansible Document page, it says that set_facts for host specific and use to register variable against the playbook we are running!
This is sucks and it provides very least explanation. It's not sufficient for our issue to solve it!
I was thinking that how do I capture each subnet ID using that module when I create a common subnet role EC2 instance creation playbook.
Explanation gives you nothing unless you don't look into the code!
Let's dig in,
Myplaybook
|---- EC2_webserver.yml
|---- EC2_application.yml
|---- Subnet.yml
|---- roles
|--- subnet.yml
|--- web_subnet.yml
|--- app_subnet.yml
Subnet.yml
- import_playbook web_subnet.yml- import_playbook app_subnet.yml
web_subnet.yml---
- name: create subnet for webserver1
hosts: controt
# ask input from users
vars_prompt:
- name: "cidr_block_subnet1"
prompt: "Enter the CIDR block you want for web server 1 subnet"
private: no
- name: "subnet_name1"
prompt: "Enter the name of the web server 1 subnet"
private: no
- name: "subnet_az1"
prompt: "Enter the availability zone of web server 1 subnet"
private: no
tasks:
- set_fact:
info: {}
- name: creating webserver1
include roles:
name: ./roles/subnet
vars:
cidr_block_subnet: "{{cidr_block_subnet1}}"
subnet_name: "{{subnet_name1}}"
az: "{{subnet_az1}}"
# subnetinfo is used to store subnet infomation
# when we executing subnet roles subnetinfo: wb1_subnet
- name: print webserver1 output
debug:
var: infoapp_subnet.yml
---
- name: create subnet for appserver
hosts: controt
# ask input from users
vars_prompt:
- name: "cidr_block_subnet2"
prompt: "Enter the CIDR block you want for app server 1 subnet"
private: no
- name: "subnet_name2"
prompt: "Enter the name of the app server 1 subnet"
private: no
- name: "subnet_az1"
prompt: "Enter the availability zone of app server 1 subnet"
private: no
tasks:
- set_fact:
info: {}
- name: creating app1
include roles:
name: ./roles/subnet
vars:
cidr_block_subnet: "{{cidr_block_subnet1}}"
subnet_name: "{{subnet_name1}}"
az: "{{subnet_az1}}"
# subnetinfo is used to store subnet infomation
# when we executing subnet roles
subnetinfo: wb1_subnet
- name: print app1 output
debug:
var: infoSince it is using the same subnet role, on each playbook we need to capture web server and app server subnet information.
I used subnetifor variable and its value replace with each subnet role output.
Don't worry!
if you see roles/subnet.yml
you will understand how it replaces subnet information.
roles/subnet.yml
---
# tasks file for subnet
# this can be use as common roles for each LAMP stack
# name provide which server we are creating subnets
- name: creating subnets "{{subnet_name}}"
ec2_vpc_subnet:
state: present
vpc_id: "{{ vpc.vpc.id }}"
region: "{{default_region}}"
az: "{{subnet_az}}"
cidr: "{{cidr_block}}"
resource_tags:
Name: "{{subnet_name}}"
register: output
# register stores all the output of each subnet information.
# So!
# I want store each Subnet ID for Each stack
# I found combined jina function which replace variable value
# if so, what if I combined this output to a variable of stack
# then luanch instanes on specific subnets
- name: get subnetid of particular Stack
set_fact:
info: ""{{ info | combine({subnetinfo: output}) }}"
# subnetinfo is dynamic variable and it is passed with each stack subnet.yml
# and I am overwriting the output Note: You can cache a fact set from set_facts the module so that when you execute your playbook next time, it's retrieved from the cache. You can set cacheable to yes to store variables across your playbook executions using a fact cache. You may need to look into precedence strategies used by ansible to evaluate the cacheable facts mentioned in their documentation.
Find the full code here
That's pretty much it for today, PEACE!
Thank you!
AWS EC2 intance Automation using Ansible
It has been a long time since I wrote the last blog. and I have gone through little emotional stress. Though it took me little time to overcome the stress!
Like I've said in an older blog, I have been learning about automation and CI/CD. I learned terraform a bit and I did spin up EC2 instances, public subnet, a private subnet, Internet gateway, security group and deploy some shell script.
you can find the blog here
In this blog, I wanna write about Ansible (spin-up EC2 instances and other key things. same as the previous blog). It was a pretty good experience. you can create a simple YAML file and run playbooks.
My Idea is,
1. Create VPC
2. Create public and private subnets for each Availability Zones
3. Create an internet gateway
4. Create a public gateway and make associations with public gateway
5. spin up instances on a specific subnet.
find the full code here
Please refer the Architecture below to understand my idea
Before moving onto the subject, Ansible requires some requirements to run ansbile module
Ansible :
sudo pip install ansibleBoto :
sudo pip install botohere are many ways to set our AWS credentials, in this tutorial, we'll create a file under our user home folder (~/.boto):
[Credentials]
AWS_ACCESS_KEY_ID=KID...CWU
AWS_SECRET_ACCESS_EY=3qv...DSP
AWS CLI : sudo pip install awscliNote that when launching an EC2 instance with ansible via the ansible ec2 module, the hosts variable should point to localhost and gather_facts should be set to False.
- hosts: local gather_facts: flase roles: - vpc
Create VPCI did include some other variables under group_vars directory to fetch some essential variables.
To create the VPC ec2_vpc _net module used.
- include_vars: ./group_vars/all.yml
- name: create vpc with 10.0.0.0/16
ec2_vpc_net:
name: ansibletest
cidr_block: 10.0.0.0/16
region: "{{default_region}}"
tags:
Name: ansibletest
state: present
aws_access_key: "{{ aws_access_key }}"
aws_secret_key: "{{ aws_secret_key }}"
tenancy: default
dns_hostnames: yes
dns_support: yes
register: vpc_info
#store output of ec2 infroamtion
likewise, we can use available modules to create AWS services. Some other modules are listed below.ec2_vpc_subnet (this modules used create subnets)
ec2_vpc_igw (create internet gateway
ec2_vpc_route_table (create routing table and make association for subnets)
The vpc role which I created to deploy those services here
security groups especially stand to allow certain traffic to instances.
when I created security groups, I had in mind to spin up instances for deploy LAMP stack. So, I wanted to allow ports related to the LAMP stack.
Later I developed different efficient codes to do this.
But here's the security group role which I created.
- include_vars: ./group_vars/all.yml
- name: security group with 22,80,443 port enable
ec2_group:
name: LAMP_Stack
description: sg with 22,80,443 port enable
#right now I'am using vpc_id from ./group_vars/all.yml
vpc_id: "{{vpc_id}}"
rules:
- proto: tcp
from_port: 80
to_port: 80
cidr_ip: 0.0.0.0/0
- proto: tcp
from_port: 22
to_port: 22
cidr_ip: 0.0.0.0/0
| let's move onto final steps |
How to Use Powershell Script to Execute Linux Shell -POSH
This article demonstrates how to use Powershell POSH-SSH module to login Linux Machine and Execute Shell Commands
Posh-SSH is a PowerShell module for Windows, which allows you to establish SSH connections as well as SCP functionality to remote computers. Let's dig in...
WHY?
I've been learning about AWS Linux instances and automation. I used Terraform for a spin up AWS instances and execute scripts. But, Also often think about what if create a web console to do such a task from local pc.
I will write how to automate AWS using Terraform Later. Now I'll show how to create a web console to execute Linux shell using PowerShell script.
To achieve this,
1. Create Powershell Script to SSH AWS Linux instances
2. Create ASP.NET web console
Create a PowerShell script - POSH
Step 01: To verify whether POSH-SSH installed
Find-Module Posh-SSH
Since I did not create any web console yet, though, I'll add a Simple Powershell Script.
The below code will be changed as per the web console. So, Please follow up the tutorial as I go
Write-Host "Please Enter your parameter to ping"
$cloud=Read-Host
#connect to controller
$nopasswd = new-object System.Security.SecureString
$Crendential= New-Object System.Management.Automation.PSCredential ("ubuntu", $nopasswd)
#SSH Session is created connect aws instance using private key
$ssh=New-SSHSession –ComputerName ec2-13-232-138-59.ap-south-1.compute.amazonaws.com -KeyFile 'D:\w\tom.pem' -Credential $Crendential
#Script which deployed in aws instance and which also take parameters
$command="sh new.sh $cloud"
Write-Host "Pinging....."
#This is Write the output to powershell
$result=Invoke-SSHCommand -SSHSession $ssh -Command $command | select Output -ExpandProperty Output | Out-String
Write-Host " Ping is: $result"
#End ssh session
$end_result=Remove-SSHSession -SSHSession $ssh
All you Have to do is insert parameter and click right away
Here's the ASP.NET code which was written in..
find this in repo https://github.com/5sfayas/POSH-SHH/blob/master/POSH-SSH/Ping_Tool.aspx.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Management.Automation;
using System.Text;
namespace Final_CloudApp
{
public partial class Ping_Tool : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
}
//****Function start*****************
public void ping(string cloud, string parameter)
{
//Location of the script
string script = @cloud;
TextBox3.Text = string.Empty;
// Initialize PowerShell engine
var shell = PowerShell.Create();
// Add the script to the PowerShell object
shell.Commands.AddCommand(script, false);
//Name of the parameter as in the script
shell.Commands.AddParameter("com1", source); //Parameter-0
shell.Commands.AddParameter("com2", dest);//Parameter-1
// Execute the script
var results = shell.Invoke();
// Note : use |out-string for console-like output
if (results.Count > 0)
{
// We use a string builder ton create our result text
var builder = new StringBuilder();
foreach (var psObject in results)
{
// Convert the Base Object to a string and append it to the string builder.
// Add \r\n for line breaks
builder.Append(psObject.BaseObject.ToString() + "\r\n");
}
// Encode the string in HTML (prevent security issue with 'dangerous' caracters like < >
// TextBox2.Text = Server.HtmlEncode(builder.ToString());
TextBox3.Text = (builder.ToString());
}
}
//Function Finish
protected void Button1_Click(object sender, EventArgs e)
{
ping("C:\\D drive\\Ransara\\aws_ping.ps1", TextBox2.Text);
}
}
}
then fully modify your PowerShell script like below
find this in repo https://github.com/5sfayas/POSH-SHH/blob/master/POSH-SSH/execute_shell1.ps1
[CmdletBinding()]
param(
[parameter(position=0)]
[string]$cloud,
)
#connect to controller
$nopasswd = new-object System.Security.SecureString
$Crendential= New-Object System.Management.Automation.PSCredential ("ubuntu", $nopasswd)
#SSH Session is created connect aws instance using private key
$ssh=New-SSHSession –ComputerName ec2-13-232-138-59.ap-south-1.compute.amazonaws.com -KeyFile 'D:\w\tom.pem' -Credential $Crendential
#Script which deployed in aws instance and which also take parameters
$command="sh new.sh $cloud"
Write-Host "Pinging....."
#This is Write the output to powershell
$result=Invoke-SSHCommand -SSHSession $ssh -Command $command | select Output -ExpandProperty Output | Out-String
Write-Host " Ping is: $result"
#End ssh session
$end_result=Remove-SSHSession -SSHSession $ssh
Docker Cheats
This article will be your one-stop-shop for Docker, going over some of the best practices and must-know commands that any user should know.
Why Docker Instead of VMs? What’s the Big Deal?
let's say, applications you’ve built. If you want each app to be isolated, you will need to run each one inside of its own guest operating system. because of the binaries and libraries needs to run the application is not isolated. Though, it leads to the major issue.
if I want to run 30 applications and its need 30 virtual machines, you’ve got to boot 30 operating systems with at least minimum resource requirements available before factoring the hypervisor for them to run on with the base OS.
If you have Docker then you can have 30 containers that you want to run, you can run them all on a single virtual machine because its package all the dependencies and also make it portable.
So, Containers are a better choice than VMs when our biggest priority is maximizing the number of applications running.
Docker Commands and Practices
Here’s a quick overview of the vocabulary you should know.
Docker Image: Docker image is created at build time.
Docker Container: The Docker container is created at run time.
Docker File: The Dockerfile is at the heart of Docker. The Dockerfile tells Docker how to build the image that will be used to make containers.
I'll do another blog on how to create a docker image using a docker file.
Docker Commands
Find the version
docker --version or docker -v
Login to docker hub
docker login
Pull an image from the Docker Hub repository
docker pull <image>
Push an image to the Docker Hub repository
docker push <username/image>
Search the Docker Hub repository for a particular term
docker search <term>
Create a target tag or alias that refers to a source image
docker tag <source> <target>
Build an image from working directory
docker build -t <imagename> .
. refers workdir
-t optionally a tag in the ‘name:tag’ format
Ex: docker build -t flask:latest .
Build image for any locations docker file
docker build -t <imagename> -f <docker file path>/<url>
Ex: docker built -t flask -f /path/to/docker/Dockerfile .
EX: docker built -t flask -f github.com/5sfayas/IGm
See list of created images
docker images
docker image ls
Run the docker image
docker run <imagename>
docker run -d -p 80:8080 flask
-d run in the background -p assing port to container flask is imagename
Specify an ip-address for the container when running (See Network Related Section Below)
docker run -i -d --name=rc --net br0 --ip 192.168.0.104
Assing network type and ip to container
docker run it -add-host=example.com:10.1.1.2
---add-host flag is used to add a single hots to ip withing the docker container. this make entry in /etc/hosts file to make fqdn for container.
Remove a container upon exit
docker run --rm <image_name>
Remove docker by image name
docker rmi <image_name>
List running containers
docker ps
List all exited containers
docker ps -aq -f status=exited
Get all low-level details of the container (including IP address)
docker inspect <containerId>
This throws lots of information and its useful to find IP of container.
Note: later I'll write a simple blog to find IP of the container
Get logs of the container
docker logs <containerId>
very useful to find out what happened to a container
Kill and Remove container
Kill Container
docker kill <containerId>
This will stop a running container
docker kill ${docker ps -q}
kill all container
Remove stopped containerdocker rm <containerId>
This remove and clean container(running, stopped)
Run container then remove it
docker run rm <imagename>
Create and start container with terminal interaction
docker run -ti <imagename>
The useful to run bash command to container
docker run -ti <>imagename> <command>
docker run -ti <>imagename> /bin/bash
Inspect network
Docker Newtwork ls
Verify network through the inspect commanddocker network inspect br0
Linux File System and LVM
In this post, I'll share my experience with the Linux file system with error and creating custom Linux partitions using LVM based on the user's request at my work.
Why?
LVM
What, Why LVM?
Logical Volume Management (LVM) makes it easier to manage disk space. If a file system needs more space, it can be added to its logical volumes from the free spaces in its volume group and the file system can be re-sized as we wish. If a disk starts to fail, replacement disk can be registered as a physical volume with the volume group and the logical volumes extents can be migrated to the new disk without data loss.
-n [To create new partition]
-p [To create new primary partition]
-w [To save changes]
To view details of physical volume- pvdisplay
lvcreate -L 50G -n point2 volgrup01
lvcreate -l 100%FREE -n point3 volgrup01
mkfs.xfs /dev/volgrup01/point2
mkfs.xfs /dev/volgrup01/point3
7. Mount the logical volume to mount points
mount /dev/volgrup01/point1 /data1
mount /dev/volgrup01/point2 /data2
mount /dev/volgrup01/point3 /data3
8. If you reboot the system, the mount points will be removed. In order to make them persistent, open “fstab” file and add the below entries.
/dev/volgrup01/point1 /data1 xfs defaults,nofail 0 0
/dev/volgrup01/point2 /data1 xfs defaults,nofail 0 0
/dev/volgrup01/point3 /data1 xfs defaults,nofail 0 0






