In the actual software system, performance problems do not come out of the blue. They tend to begin as minor resource leakages, bizarre spikes, or other inefficient processes that have not been noticed until they affect users.
It is important to know how the system performs at the system level in terms of CPU, memory, disk, and process behavior. These metrics assist us in distinguishing between application code performance problems and infrastructure problems, as well as outside dependence problems.
Goal of the Monitoring Tool
In this blog, we are going to develop a Python tool that:
- Collects system metrics:
- CPU usage
- Memory usage
- Disk usage
- Process-level metrics
- Stores metrics to analyze history.
- Presents data in the form of graphs.
- Helps detect performance in applications.
Real-Life Analogy: Consider this tool as the health monitor of a server or an application, such as a smart watch monitoring heartbeat, activity, and sleep patterns.
The importance of monitoring in Real Projects
Monitoring in real-life projects is necessary in:
- Identifying performance bottlenecks.
- Identifying memory leaks
- Learning how the system behaves when under load.
- Troubleshooting slow applications or API.
- Performance and scalability testing Support.
Example: E-commerce Application.
Users complain that a site is slow when it is busy.
Monitoring reveals:
- CPU usage: 95% → Server overloaded
- Memory usage: continuously increasing → Possible memory leak
- Disk usage: 98% → Logs consuming storage
- A specific Java/Python process consuming high CPU → Code inefficiency
This shows that monitoring helps pinpoint the root cause instead of guessing.
Architecture of the Monitoring Tool
Our monitoring tool is designed with three logical layers:
Data Collection Layer
- Gathers system data using Python libraries like psutil
- Periodic execution (e.g., every 5 seconds)
Data Storage Layer
- Stores data in CSV files (can be extended to databases)
- Facilitates trend identification and analysis of history.
Visualization & Analysis Layer
- Produces graphs using matplotlib
- Helps detect trends like spikes and an increase in resources.
Architecture Flow
This layered design makes the tool scalable and extensible.
Understanding Key System Components
1. CPU
The CPU is the one that executes instructions, and that has a direct effect on the application’s performance.
Examples:
- Running backend services
- Processing API requests
- Running automation scripts.
The importance of CPU monitoring: An elevated CPU usage may lead to low response time or the crashing of the server.
Memory (RAM)
RAM is a memory used to save the temporary data that is utilized by the running applications.
Examples:
- User sessions
- Cached data
- Application runtime objects
The reason why it is important to monitor memory: A constant increase in memory can be a sign of a memory leak. Lack of sufficient memory has the ability to crash.
Disk (Storage)
The disk holds the enduring data in the form of logs, databases, and files.
Examples:
- Application logs
- Database files
- Uploaded documents
The use of disk monitoring is important because: Full disk space is able to prevent applications and logging mechanisms.
Process-Level Monitoring
A running program (e.g., Python, Java, or a database service) is a process.
Examples:
- Python app using 60% CPU
- 40% Memory used by database service.
The significance of process monitoring is that it helps determine the application that is using resources and is the key to performance debugging.
Defining Thresholds and Alerts
In real systems, thresholds are defined based on SLAs:
- CPU > 75% for 5 minutes → Warning
- CPU > 90% for 2 minutes → Critical alert
- Memory growth > 10% per hour → Potential memory leak
- Disk usage > 85% → Risk of failure
Monitoring is not just about collecting metrics—it is about detecting abnormal patterns.
Step-by-Step Implementation Guide
We will build a Python-based system monitoring tool.
It continuously checks:
- CPU usage
- Memory usage
- Disk usage
- Process-level metrics (specific service/app)
- Stores data
- Helps detect performance problems
Pre-requisites: Install Python and PyCharm.
Step 1: Verify Python is installed and install pip to install libraries
- Navigate to the terminal
- Run command: python3 –version
- Run command: sudo apt update
- Run command: sudo apt install python3-pip –y
Step 2: Create a project folder to keep project files in one place
- Navigate to the terminal
- Run command: mkdir python_monitor
- Run command: cd python_monitor
Step 3: Open project in PyCharm
- Open PyCharm
- Select python_monitor folder
Step 4: Create a virtual environment to isolate project libraries.
- Open PyCharm terminal
- Run command: python3 -m venv venv
- Run command: source venv/bin/activate
Step 5: Install Libraries psutil and matplotlib: psutil to get system metrics and matplotlib to draw graphs.
- Open PyCharm terminal
- Run command: pip install psutil matplotlib
Why we installed psutil and matplotlib:
Psutil -> System monitoring library
It gives us access to system information like:
- CPU usage
- RAM usage
- Disk usage
- Process details
Example: psutil.cpu_percent()
Matplotlib -> Graph plotting library
It helps us draw graphs from collected data.
Example: CPU usage graph over time, Memory usage graph over time.
Step 6: Create Folders in the Project to store monitoring data and graphs.
- Open PyCharm terminal
- Run command: mkdir data graphs
Step 7: Create Monitor.py file
- In PyCharm → Right click project → New → Python File → name it Monitor.py
Paste the code below:
Step 8: Run Monitor.py
- Open PyCharm terminal
- Run command: python3 Monitor.py
What happens: Every 5 seconds, data is collected.
You will see: [2026-02-03 13:01:27] Metrics captured
Step 9: Check the collected data
- Open PyCharm terminal
- Run command: cat data/metrics.csv
Example output: 2026-01-29 17:10:10,40,55,70,5,120,8
Meaning:
- CPU = 40%
- Memory = 55%
- Disk = 70%
- Python process CPU = 5%
- Python process memory = 120 MB
- Threads = 8
Step 10: Create Graphs.py
- PyCharm Editor → New → Python File → Graphs.py
- Paste the code below
Step 11: Run Graphs.py
- Open PyCharm terminal
- Run command: python3 Graphs.py
Result: A graph image is created in: graphs/performance.png
Conclusion
This blog went ahead to show how to create and develop a Python-based system monitoring tool and the application of these tools in real-life testing and production processes.
System monitoring is not an option but rather a vital skill that makes automation testing, performance engineering, and system reliability interrelated to an SDET.
With databases, dashboards, and alerting mechanisms added to this tool, it can be expanded to become an able solution for production-grade monitoring.
