This is the third part of our series about Sysdig. The first tutorial walked you through the process of capturing and analyzing data with the Sysdig command-line utility. The second tutorial showed how to use Csysdig, the ncurses based GUI. Now, we’ll focus on explaining how you can troubleshoot your containers with Sysdig Inspect.
Before starting this tutorial, you should follow the first part of the series. We assume that Docker Compose and Sysdig are installed on your system. Also, you should have a running WordPress installation. Enter the following command to verify that everything is properly set up:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f2a5bf941eba wordpress:latest "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:8000->80/tcp wordpress-sysdig_wordpress_1 95c44dd63ba9 mysql:5.7 "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp, 33060/tcp wordpress-sysdig_db_1 8405c2c87f59 sysdig/sysdig "/docker-entrypoint.…" About a minute ago Up About a minute sysdig
Make sure the following three Docker containers are running on your system:
Install Sysdig Inspect
There are two ways in which you can install Sysdig inspect:
- Download the pre-built binary for your operating system from the project’s GitHub repository.
- Run it as a Docker container.
For this tutorial, you will install Sysdig Inspect inside of a Docker container.
In this section, you’ll install Sysdig Inspect and make it so that the trace files captured with
sysdig can be accessed from the Sysdig Inspect container.
- In a new terminal window (on the host), run the following command to create a directory called
- You can install Sysdig Inspect in a container with the following
docker run -d -v /home/vagrant/captures/:/captures -p8080:3000 sysdig/sysdig-inspect:latest
Unable to find image 'sysdig/sysdig-inspect:latest' locally latest: Pulling from sysdig/sysdig-inspect f661dfa5eb7d: Pull complete 572ec58ac70f: Pull complete c1d2c84e7140: Pull complete 1ebe1ad7056b: Pull complete e5e1416d36d0: Pull complete 9b379815df57: Pull complete 9034f4cfeb96: Pull complete 91638da20eef: Pull complete a4105e889626: Pull complete b7359900512b: Pull complete 2daad7eb7645: Pull complete d0fb8981639d: Pull complete ed8bc4c05912: Pull complete 29d1d8034bab: Pull complete fead5f9a4603: Pull complete Digest: sha256:0c6270a09cfb28f2c248761f4783d0bc53d3dfd9b83d27f03b98039d5366cf45 Status: Downloaded newer image for sysdig/sysdig-inspect:latest ee88efbc56205164a269102d266509ce5a677d024071d29774b12cd3414a7a69
A few things to note about the above command:
-vflag specifies the files on the host that Sysdig Inspect can access. In this example, the files placed inside of the
/home/vagrant/captures/directory will be available in the
capturesfolder in the container running Sysdig Inspect.
-dflag starts the container in detached mode
-pflag publishes the
3000port to the
8000port on the host.
- At this point you can inspect the status of the Sysdig Inspect container with:
Note that the status is currently set to
starting. Wait a bit until it changes to
- Now you can point your browser to http://localhost:8080:
As you can see in the screenshot above, you can specify the path to a capture file taken with Sysdig. In this section, you’ll create a new trace file, simulate an issue by deleting a file, and then load the data into Csysdig Inspect for inspection.
- In a new terminal window (on the host), display the list of your active containers:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7da3fee501db sysdig/sysdig-inspect:latest "/docker-entrypoint.…" 2 minutes ago Up 8 minutes (healthy) 0.0.0.0:8080->3000/tcp agitated_leakey f2a5bf941eba wordpress:latest "docker-entrypoint.s…" 8 minutes ago Up 8 minutes 0.0.0.0:8000->80/tcp wordpress-sysdig_wordpress_1 95c44dd63ba9 mysql:5.7 "docker-entrypoint.s…" 8 minutes ago Up 8 minutes 3306/tcp, 33060/tcp wordpress-sysdig_db_1 8405c2c87f59 sysdig/sysdig "/docker-entrypoint.…" 9 minutes ago Up 9 minutes sysdig
- Start a shell inside of the
wordpresscontainer by entering the following
docker execcommand and passing the identifier of the container:
docker exec -ti f2a5bf941eba /bin/bash
☞ In the above example, the identifier of our container is
f2a5bf941ebabut yours will be different.
- Similarly, start a shell inside of the
docker exec -ti 8405c2c87f59 /bin/bash
- Initiate a new capture by entering the following command inside of the
sysdig -w monitoring-wordpress-error.scap
- Open a new terminal window (on the host), and use ApacheBench to make 10000 requests with a maximum of 100 requests running concurrently:
ab -n 10000 -c 100 http://localhost:8000/
- While ApacheBench is running, move back to the
wordpresscontainer and delete the
rm -rf index.php
- When the benchmark is finished, move to the
sysdigcontainer and stop capturing data by entering “CTRL+C”.
- If everything went well, a file named monitoring-wordpress-error.scap should have been created:
ls -la monitoring-wordpress-error.scap
-rw-r--r--. 1 root root 146795720 Jan 16 20:56 monitoring-wordpress-error.scap
- Switch back to the host, and use the
docker cpcommand to copy the capture file to the
docker cp 8405c2c87f59:/monitoring-wordpress-error.scap captures/
- Verify whether the file has been successfully copied to the host with:
ls -la captures/
total 143356 drwxrwxr-x. 2 vagrant vagrant 45 Jan 16 21:02 . drwx------. 8 vagrant vagrant 208 Jan 16 18:27 .. -rw-r--r--. 1 vagrant vagrant 146795720 Jan 16 20:56 monitoring-wordpress-error.scap
- Point your browser to http://localhost:8080 and type the path to the trace file (that is
/captures/monitoring-wordpress-error.scap) in the “capture file path” text area:
- Press the
Enterkey and then wait a bit while Sysdig Inspect loads the trace file:
- When everything is ready, you will see the overview page:
How to Navigate the Interface
In this section, we’ll explain the basic concepts behind Sysdig’s Inspect interface. Once the trace file is loaded, you’ll be presented with a screen similar to the one below (the annotations are ours):
There’s a lot of information packed in here. Let’s break this down:
(A) At the top of the screen, Sysdig Inspect shows the name of the currently open trace file.
(B) The drill-down section shows the list of active filters. Select the “Containers” view and then navigate to the “Processes view”:
☞ You can customize what data is displayed by manually editing the “Sysdig Filter” textbox.
(C) Similar to how Csysdig works, Sysdig Inspect lets you filter the data using several built-in views.
(D) The main window contains several tiles. Each tile shows the values of a particular metric and its evolution in the period of time in which the capture was taken. Tiles are grouped together to make navigation easier.
(E) The timeline allows you to use two sliders to narrow down the period during which the data is analyzed. Continuing our previous example in which we selected the “Container” view and then “Navigated” to the process view, we used the sliders to narrow down the period of time for which the data is displayed:
We’ve covered the basics of using the interface! In the next section, we’ll show how you can use Sysdig Inspect to troubleshoot your containers.
Put It to the Test
Remember that, in the “Capture Data” section, we’ve purposely introduced an error. Now, we’ll make use of that to showcase how you can use Sysdig Inspect to analyze data and troubleshoot your containers.
- First, navigate to the overview page:
- From the “Views” section, select “Errors”. You’ll see a high number of
ENOENT(which stands for Error NO ENTry) errors:
- Highlight the
ENOENTline by clicking on it. This will display three round buttons:
- I/O Streams
- Select “Drill-down’ and Sysdig Inspect will show the list of processes that raised this error:
- Pick the one from top of the list. Then, select “Syscalls”. Since we’ve deleted the
index.phpfile a bit later in the capture, you must scroll down a few lines until you see lots of error messages similar to the ones below:
☞ Note that the
Sysdig Filter checkbox has been automatically updated.
Now, this sheds some light, but it doesn’t tell you what has happened.
- Let’s clear all the filters and navigate to the default view by selecting “Overview”:
- See the “Security” section. It shows that two commands were executed, and one file was deleted.
- To get more details, select the
Note that the
Drill-down button is now displayed. Also, the moment when the file was deleted is shown at the bottom of the page.
- Select the
Drill-downbutton, and Sysdig Inspect will show that the
index.phpfile was deleted:
- If you select the
Executed commands -> Drill-downtile, then you’ll see that the
rootuser deleted the
index.phpfile by running the
As you can see, Sysdig Inspect displays a host of useful information you can rely on to troubleshoot your containers.
In this example, it wasn’t overly difficult to figure out what caused the issue, but there may be cases in which you’ll have to dig a bit deeper. The knowledge you acquired is a complete tool belt that’ll help you get through in any situation that requires troubleshooting your container-based applications.
Thanks for reading!