Monitor AWS EC2 Resource Utilization with Python: A Paramiko-Based Mini Project

If you have an EC2 instance running on Amazon Web Services (AWS), you may need to monitor its system resources remotely from your local machine. You can Do that just by running a single Python script. Let's do that!

In this blog post, we will walk through how to use Python and Paramiko to collect various system information from your EC2 instance, including CPU usage, disk usage, memory usage, and system logs.

Requirements before you proceed:

• An AWS EC2 instance in a running state

• AWS CLI configured for user login credentials to your AWS account

  (Set Up the AWS Command Line Interface (AWS CLI))

• Python installed on your local machine

  (Download Python | Python.org)

• Paramiko library installed

  ($ pip install paramiko)

Let's understand this mini project in steps:

Step 1: Establish SSH Connection to EC2 Instance

Establishing SSH Connection to EC2 Instance Before we can collect system information from our EC2 instance, we need to establish an SSH connection to it. We can use the Paramiko library to achieve this. Here's the code to establish an SSH connection:

In the above code,

• first prompt the user to enter the public IP address, username, and path to the key file for their EC2 instance.

• Create an SSH client object and load the private key file for authentication.

• Connect to the EC2 instance using the provided host, username, and private key.

Step 2: Collect System Information of EC2 Instance

Collecting System Information Once we have established an SSH connection, we can start collecting system information. Here's the code to collect CPU, disk, and memory usage:

In the above code,

• we use the exec_command method to execute various commands on the remote EC2 instance.

• We use the top command to get CPU usage, the df command to get disk usage, free command to get memory usage, and cat command to read the system logs from the /var/log/syslog file.

• We then parse the output of these commands to variables, such as logs and total, used, and free.

• We then closed the SSH connection

Step 3: Get resource information of EC2 Instance to the Local host.

Write the system logs of the EC2 instance to a local file in the current directory. Finally, print the system information on the console, including CPU, disk, and memory usage.

Now, Let's demonstrate it.

• EC2 instance named docker having base OS - Ubuntu 22.04 is running

• EC2 instance access key "ssh.pem" & "Python script" in the same directory.

• Run the script and see the terminal output.

• Check the syslog file generated at the local host.

You can use above log file for troubleshooting purposes!

Summary:

In this mini project, we learned how to use Python and the Paramiko library to monitor resource utilization on remote EC2 instances. We created a Python script that connects to a remote EC2 instance via SSH, retrieves its CPU, disk, and memory usage metrics, and saves its system logs to a file.

Overall, this mini project demonstrated how Python can be used to automate and simplify cloud management tasks, making it easier to monitor and manage EC2 instances without relying on the AWS console.

I hope you learnt something valuable today with me!

Stay tuned for my next blog on "Containerization of Applications". I will keep sharing my learnings and knowledge here with you.

Let's learn together! I appreciate any comments or suggestions you may have to improve my blog content.

Thank you,

Chaitannyaa Gaikwad