Media & Entertainment
This is the third part of our series about Sysdig. The first tutorial walked you through the process of capturing and analyzing data with the Sysdig command-line utility. The second tutorial showed how to use Csysdig, the ncurses based GUI. Now, we’ll focus on explaining how you can troubleshoot your containers with Sysdig Inspect.
Before starting this tutorial, you should follow the first part of the series. We assume that Docker Compose and Sysdig are installed on your system. Also, you should have a running WordPress installation. Enter the following command to verify that everything is properly set up:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f2a5bf941eba wordpress:latest "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:8000->80/tcp wordpress-sysdig_wordpress_1 95c44dd63ba9 mysql:5.7 "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp, 33060/tcp wordpress-sysdig_db_1 8405c2c87f59 sysdig/sysdig "/docker-entrypoint.…" About a minute ago Up About a minute sysdig
Make sure the following three Docker containers are running on your system:
There are two ways in which you can install Sysdig inspect:
For this tutorial, you will install Sysdig Inspect inside of a Docker container.
In this section, you’ll install Sysdig Inspect and make it so that the trace files captured with
sysdig can be accessed from the Sysdig Inspect container.
docker run -d -v /home/vagrant/captures/:/captures -p8080:3000 sysdig/sysdig-inspect:latest
Unable to find image 'sysdig/sysdig-inspect:latest' locally latest: Pulling from sysdig/sysdig-inspect f661dfa5eb7d: Pull complete 572ec58ac70f: Pull complete c1d2c84e7140: Pull complete 1ebe1ad7056b: Pull complete e5e1416d36d0: Pull complete 9b379815df57: Pull complete 9034f4cfeb96: Pull complete 91638da20eef: Pull complete a4105e889626: Pull complete b7359900512b: Pull complete 2daad7eb7645: Pull complete d0fb8981639d: Pull complete ed8bc4c05912: Pull complete 29d1d8034bab: Pull complete fead5f9a4603: Pull complete Digest: sha256:0c6270a09cfb28f2c248761f4783d0bc53d3dfd9b83d27f03b98039d5366cf45 Status: Downloaded newer image for sysdig/sysdig-inspect:latest ee88efbc56205164a269102d266509ce5a677d024071d29774b12cd3414a7a69
A few things to note about the above command:
-vflag specifies the files on the host that Sysdig Inspect can access. In this example, the files placed inside of the
/home/vagrant/captures/directory will be available in the
capturesfolder in the container running Sysdig Inspect.
-dflag starts the container in detached mode
-pflag publishes the
3000port to the
8000port on the host.
Note that the status is currently set to
starting. Wait a bit until it changes to
As you can see in the screenshot above, you can specify the path to a capture file taken with Sysdig. In this section, you’ll create a new trace file, simulate an issue by deleting a file, and then load the data into Csysdig Inspect for inspection.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7da3fee501db sysdig/sysdig-inspect:latest "/docker-entrypoint.…" 2 minutes ago Up 8 minutes (healthy) 0.0.0.0:8080->3000/tcp agitated_leakey f2a5bf941eba wordpress:latest "docker-entrypoint.s…" 8 minutes ago Up 8 minutes 0.0.0.0:8000->80/tcp wordpress-sysdig_wordpress_1 95c44dd63ba9 mysql:5.7 "docker-entrypoint.s…" 8 minutes ago Up 8 minutes 3306/tcp, 33060/tcp wordpress-sysdig_db_1 8405c2c87f59 sysdig/sysdig "/docker-entrypoint.…" 9 minutes ago Up 9 minutes sysdig
wordpresscontainer by entering the following
docker execcommand and passing the identifier of the container:
docker exec -ti f2a5bf941eba /bin/bash
☞ In the above example, the identifier of our container is
f2a5bf941ebabut yours will be different.
docker exec -ti 8405c2c87f59 /bin/bash
sysdig -w monitoring-wordpress-error.scap
ab -n 10000 -c 100 http://localhost:8000/
wordpresscontainer and delete the
rm -rf index.php
sysdigcontainer and stop capturing data by entering “CTRL+C”.
ls -la monitoring-wordpress-error.scap
-rw-r--r--. 1 root root 146795720 Jan 16 20:56 monitoring-wordpress-error.scap
docker cpcommand to copy the capture file to the
docker cp 8405c2c87f59:/monitoring-wordpress-error.scap captures/
ls -la captures/
total 143356 drwxrwxr-x. 2 vagrant vagrant 45 Jan 16 21:02 . drwx------. 8 vagrant vagrant 208 Jan 16 18:27 .. -rw-r--r--. 1 vagrant vagrant 146795720 Jan 16 20:56 monitoring-wordpress-error.scap
/captures/monitoring-wordpress-error.scap) in the “capture file path” text area:
Enterkey and then wait a bit while Sysdig Inspect loads the trace file:
In this section, we’ll explain the basic concepts behind Sysdig’s Inspect interface. Once the trace file is loaded, you’ll be presented with a screen similar to the one below (the annotations are ours):
There’s a lot of information packed in here. Let’s break this down:
(A) At the top of the screen, Sysdig Inspect shows the name of the currently open trace file.
(B) The drill-down section shows the list of active filters. Select the “Containers” view and then navigate to the “Processes view”:
☞ You can customize what data is displayed by manually editing the “Sysdig Filter” textbox.
(C) Similar to how Csysdig works, Sysdig Inspect lets you filter the data using several built-in views.
(D) The main window contains several tiles. Each tile shows the values of a particular metric and its evolution in the period of time in which the capture was taken. Tiles are grouped together to make navigation easier.
(E) The timeline allows you to use two sliders to narrow down the period during which the data is analyzed. Continuing our previous example in which we selected the “Container” view and then “Navigated” to the process view, we used the sliders to narrow down the period of time for which the data is displayed:
We’ve covered the basics of using the interface! In the next section, we’ll show how you can use Sysdig Inspect to troubleshoot your containers.
Remember that, in the “Capture Data” section, we’ve purposely introduced an error. Now, we’ll make use of that to showcase how you can use Sysdig Inspect to analyze data and troubleshoot your containers.
ENOENT(which stands for Error NO ENTry) errors:
ENOENTline by clicking on it. This will display three round buttons:
index.phpfile a bit later in the capture, you must scroll down a few lines until you see lots of error messages similar to the ones below:
☞ Note that the
Sysdig Filter checkbox has been automatically updated.
Now, this sheds some light, but it doesn’t tell you what has happened.
Note that the
Drill-down button is now displayed. Also, the moment when the file was deleted is shown at the bottom of the page.
Drill-downbutton, and Sysdig Inspect will show that the
index.phpfile was deleted:
Executed commands -> Drill-downtile, then you’ll see that the
rootuser deleted the
index.phpfile by running the
As you can see, Sysdig Inspect displays a host of useful information you can rely on to troubleshoot your containers.
In this example, it wasn’t overly difficult to figure out what caused the issue, but there may be cases in which you’ll have to dig a bit deeper. The knowledge you acquired is a complete tool belt that’ll help you get through in any situation that requires troubleshooting your container-based applications.
Thanks for reading!