Grafana — Docker Resource Consumption Monitoring
Monitoring resource consumption for any running container is equally important than building application to run under docker container.
“There are many more tool available in the market to monitor docker resource consumption, however I always prefer to create our own solution as that gives complete control on what we want to do & how….If you are like minded to build your own solution not only to have complete control but to uncover potential of your knowledge then go ahead follow the rest”
Whenever we build any application for production use, it is always required to share how much resources are required for that application under maximum load. To define resource consumption for any dockerized application, we need to collect resource consumption under maximum load (or mimic production like environment) for any dockerized application. Once we have all data points collected then it becomes easy for us to define how much resource application is going to need under maximum load so it won’t result in any down time just because of resource constraints. Let’s first find out what docker offers us to collect resource consumption then we will analyze why docker commands are not good enough for us to calculate resources required for any container in production environment. Here is complete youtube videos for end to end Docker resource consumption monitoring through Grafana -
Docker Resource Consumption — How to collect resource consumption for any running docker container With Shell Script (Part-1) — This video explains how can you write Shell script to collect resource consumption for all running container for any defined period of time.
1. How to Collect resource consumption for any running container
2. How to collect resource consumption for docker in file
3. Shell script to collect resource consumption for any defined period of time
4. Docker stats command
docker stats
docker stats — no-stream
5. docker stats command limitation
Docker- How to convert docker resource consumption into Time Series data points (Part-2) — This video explains how can you write Shell script to collect resource consumption for all running container for any defined period of time.
1. How to convert docker stats output into time series data points
2. How to format docker stats output by using --format option
3. Enhance script to accept time duration from command line as an argument
4. Modify Shell scripts to
i. Cleanup docker stats command output
ii. Append timestamp to the output of docker stats command to make time series data
5. How to extract resource consumption stats into csv file format
6. how to create time series data for CPU utilization to define how many CPUs are needed in production to run container
Docker Resource Consumption Time Series Data Points — How to import CSV data into PostgresDB (Part-3) — This video explains how can you write Shell script to import collected resource consumption into CSV to postgres database.
1. How to create table with PSQL
2. How to import CSV through PSQL
3. How to create Shell script for combining Step1 & Step2
Grafana — How to monitor docker resource consumption (Part-4) —
To monitor docker resource consumption, we can divide complete process into following phases (kind of summary what we have covered so far in last 3 videos) -
1. Capture Time series data points for any running container for CPU utilization
2. Import CSV into PostgreSQL
3. Create datasource for PostgreSQL on Grafana
4. Create dashboard by using datasource on Grafana
5. Create Graph with SQL for monitoring Docker Resource Consumption on Grafana
All the shell scripts can be found on my git repository Please share your feedback & suggestion so I can improve on my knowledge sharing for upcoming topics…