Most programmers prefer to write code over debugging it. Unfortunately, code breaks a lot more often than we would like and it often breaks in situations that are hard to debug. Therefore, an essential skill as a programmer is to know how to debug your code (and that of others).

When facing our first bug we all start out with what is called printf debugging. That means we add lines of code that call printf (or any other print function) at relevant places in our code and output values of variables or just print a message that indicates that the execution of the program has reached this particular line in the code. Then we recompile the program, run it, reproduce the problem, and add more printf calls until we find the bug.

There is nothing wrong with printf debugging. It is the most basic method of debugging and it works quite well in many situations and is available to the programmer in nearly any environment. Lots of programs even write a log file during normal operation to help track down problems that happened in production. Log files are nothing else than built-in, glorified printf debugging.

printf debugging is great but it has its limitations. For example, you can’t step through a program line by line or just jump into a specific location and look around exploring the variables and the state of the program at the time.

Whenever you want to check a new variable or data structure that wasn’t on your radar before you have to add new printf statements to the code, recompile it, run it, and get the program into the desired error state. Also, there is no way to halt the program every time a certain variable or memory address is read or modified and see from which line of code the memory access happened.

All these things and more can be done with a debugger. And that is why it is crucial to know a good debugger and know it well.

One of the most popular and powerful debuggers is gdb. And it is also available on more platforms than probably any other debugger. This article will show you everything you need to know to debug your own programs with gdb.

Fixing a simple Crash

Let’s say you write a program in C or C++ and it crashes. While in languages like Java or Python, you will get a full-fledged stack trace all you get in our case is the message “Segmentation fault” in the terminal from which you started your program.

Figuring out where your program crashed is probably the most simple problem you can solve with a debugger like gdb.

Save the following program to a file named crash.c:

#include <stdio.h>

void do_something()
{
    char *str = NULL;
    puts(str);
}   
 
int main()
{
    do_something();
    return 0;
}

Compile it with the following command:

$ gcc -o crash crash.c

When you run it you will see that it actually crashes:

Of course, it is pretty obvious why this program crashes and how to fix it. But let’s pretend for a moment that this program is much bigger and we have no idea where and why it crashes.

So let’s start gdb and load the program with:

$ gdb crash

You will now see a gdb prompt similar to this one:

$ gdb crash 
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from crash...
(No debugging symbols found in crash)
(gdb)

Start your program with the command run (if this program had any parameters you could just put them behind the run command):

(gdb) run

The program will crash immediately and gdb informs us that a crash happened:

(gdb) run
Starting program: /home/user/crash/crash 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
__strlen_sse2 () at ../sysdeps/x86_64/multiarch/strlen-vec.S:126
126	../sysdeps/x86_64/multiarch/strlen-vec.S: No such file or directory.
(gdb)

Enter the command backtrace (or its shorthand bt) to get a backtrace:

(gdb) backtrace
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/strlen-vec.S:126
#1  0x00007ffff7df1ee8 in __GI__IO_puts (str=0x0) at ./libio/ioputs.c:35
#2  0x0000555555555169 in do_something ()
#3  0x000055555555517e in main ()

You have to read this backtrace backward. The program started in main. It then called do_something which in turn called __GI__IO_puts (looks like puts is actually a macro which expands to this awkward function name). The puts function then called a SSE2 optimized version of strlen and this is where the actual crash happened.

For us, this doesn’t matter much, though. It is good enough to know that we triggered the crash in our function do_something. At least as long as we assume that the functions of the C standard library are correct. And most of the time this is a good assumption.

So there is only one problem left. The backtrace shows the line numbers of the standard library functions but it doesn’t show the line numbers where things happen in our program. The reason for this is that we didn’t compile our program with debug symbols. Of course, this is not a problem with our toy program. If all your functions have only 2 lines of code it is good enough to know the function where something fails.

Since real programs aren’t that short usually, let’s fix this.

First quit gdb with:

(gdb) quit

Now recompile your program with this command:

$ gcc -ggdb -o crash crash.c

The flag -g adds debug symbols to our program. Now run it in gdb again and print a new backtrace after it crashes:

(gdb) backtrace
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/strlen-vec.S:126
#1  0x00007ffff7df1ee8 in __GI__IO_puts (str=0x0) at ./libio/ioputs.c:35
#2  0x0000555555555169 in do_something () at crash.c:6
#3  0x000055555555517e in main () at crash.c:11

And now we have line numbers and can see that the crash is triggered in crash.c at line 6. It is not a good idea to call puts with a NULL pointer I suppose. So initialize str with “Hello World”, recompile and run the program and it no longer crashes.

Command Abbreviation: Commands can be abbreviated by only entering as many characters of the command as are needed for the command name to be non-ambiguous.

So instead of breakpoint for example you can only enter b. Also, break would work as would bre. This is why often you will see people use c instead of continue, n instead of next, s instead of step etc.

With those line numbers, we could now find out where a crash happens even in a huge program with dozens of nested function calls and hundreds of thousands of lines of code.

Stepping through a Program

It’s great to be able to inspect a program after it crashes. But there is much more you can do with a debugger. Most of the time we actually want to execute a program step by step to learn how it works or to figure out why some of its behavior is not as we expect.

But there is one problem. If we start a program with gdb like we did in the previous example gdb will not start the program until we enter the run command. Once we do that though, the program runs from start to finish or until it crashes. A long-running program like a server will just go about its business and run infinitely.

To be able to step through a program we need to tell gdb that it has to halt the program whenever certain lines in the code are about to be executed. We do this by setting a so-called breakpoint.

The two most common ways to set a breakpoint are by setting a breakpoint at the start of a function or at a certain line of code in a certain code file.

In gdb breakpoints can be set at the beginning of a function with this command:

(gdb) breakpoint function_name

If you want gdb to halt the execution of the program at a certain line of code (e.g. at line 123 of filename.c) you can use the following command:

(gdb) breakpoint filename.c:123

Let’s demonstrate the full process of stepping through a program with the following code:

#include <stdio.h>

void print_loop()
{
    for (int i = 0; i < 100; i++)
    {
        printf("Loop counter: %d\n", i);
    }
}

int main(int argc, char **argv)
{
    int a = 0;
    int b = 0;
    int sum = 0;

    a = 3;
    b = 5;
    sum = a + b;

    printf("Sum: %d\n\n", sum);

    print_loop();

    return 0;
}

This program adds two numbers in a very verbose way and prints the sum. Then it calls a function that counts from 0 to 99 and prints the current loop counter after every increment. After that, the function returns and the program exits to the operating system with status code 0.

First, we save our program as sum.c and compile it with:

$ gcc -ggdb -o sum sum.c

Now we can start gdb and tell it that we want to debug our freshly compiled program:

$ gdb sum

It greets us with a prompt like this:

GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from sum...
(gdb) 

At this point, our program is not running. gdb tells us that it has read the symbols from our binary before dropping us on its command prompt.

If we started our program now it would just run to completion and we would have no way to interact with it before it terminates. Therefore, before we start it we set a breakpoint on the main function:

(gdb) break main

gdb will give us a reply similar to:

Breakpoint 1 at 0x1198: file sum.c, line 13.
(gdb)

Now we can start the program with the run command:

(gdb) run

If our program required any parameters we could just put them behind the run command and they would be passed to the program.

After starting our program gdb will tell us immediately that our breakpoint was hit and it stopped our program. We are now at the beginning of the main function:

Starting program: /home/user/gdb/step 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main (argc=1, argv=0x7fffffffdc98) at sum:13
13	    int a = 0;
(gdb)

This tells us that we are at line 13 in the source file sum.c.

To get a better idea where we are in the code we can use the list command at any time during our debugging session and it will print out the code surrounding our current location:

(gdb) list
8	    }
9	}
10	
11	int main(int argc, char **argv)
12	{
13	    int a = 0;
14	    int b = 0;
15	    int sum = 0;
16	
17	    a = 3;
(gdb)

We can now go to the next line with the next command:

(gdb) next
14	    int b = 0;
(gdb)

We are now at line 14. Now execute the next command until we are at line 21:

(gdb) n
21 printf("Sum: %d\n\n", sum);
(gdb)

Command Repetition: When you just press enter without entering a new command, gdb will execute the previous command again. This is especially useful when using the next command. You enter next to go to the next line and then can just press enter again and again to go through the code.

At this point, the printf function hasn’t been executed, yet. The perfect time to check what is in the result variable above:

(gdb) print sum
$2 = 8
(gdb)

With the print command, we can print the value of every variable that is accessible from the current line of code. In case your variable is a pointer you might want to dereference it by using a command like print *variable_name to get the value instead of the pointer address.

The value of sum is 8 which is what we would expect from adding 3 and 5.

$2 = 8
(gdb) next
Sum: 8

23	    print_loop();
(gdb)

What we see here is that the printf function was executed and printed its message. We are now in line 23 directly before the execution of the print_loop function.

If we used the next command again gdb would let the program run the function to completion like it did with the printf function. But that is not what we want to do. Instead, we want to jump into the function and inspect what it is doing.

To do this we have to use the step command instead of next:

(gdb) step
print_loop () at sum.c:5
5	    for (int i = 0; i < 100; i++)
(gdb)

We are now at the beginning of the print_loop function and therefore also at the beginning of the for loop.

If we use next to advance to the next line we will get to the call to printf:

(gdb) next
7	        printf("Loop counter: %d\n", i);
(gdb)

Enter next once more and we are back at the beginning of the loop:

(gdb) next
Loop counter: 0
5	    for (int i = 0; i < 100; i++)
(gdb)

To get out of the loop we need to go through all 100 iterations. You might feel a little bit trapped at this point.

But don’t despair. There is a faster way to break out of the loop. Just set a breakpoint to line 9 which is the first line after the loop:

(gdb) break sum.c:9

Instead of break you could also use the tbreak command. It creates a temporary breakpoint which is automatically deleted after it gets hit for the first time while a breakpoint created with break stays active until it is manually deleted.

This makes tbreak great for when creating breakpoints to navigate through a program. But since this part of the code is executed only once it doesn’t really matter here.

Even easier is to use the until command which stops at the target line automatically.

Then you can use the continue command to make gdb run the program until it hits our new breakpoint:

(gdb) continue

The program will then print the loop counter for all the remaining values up to 99 and gdb will stop the program after the loop:

Breakpoint 2, print_loop () at sum.c:9
9	}
(gdb)

If we use the next command we will drop out of the function:

(gdb) next
main (argc=1, argv=0x7fffffffdc98) at sum.c:25
25	    return 0;
(gdb)

Three more next commands and the program terminates:

(gdb) next
26	}
(gdb) next
__libc_start_call_main (main=main@entry=0x555555555185 <main>, argc=argc@entry=1, argv=argv@entry=0x7fffffffdc98) at ../sysdeps/nptl/libc_start_call_main.h:74
74	../sysdeps/nptl/libc_start_call_main.h: No such file or directory.
(gdb) next
[Inferior 1 (process 16159) exited normally]
(gdb)

This little tour through a simple example program should have given you a good idea about how to do basic debugging with gdb. The following list of commands will help you with your first steps with gdb:

Saving your Breakpoints for later Use

When you debug a complex program for hours or even days the relevant breakpoints you come up with during your debugging session are among the most important knowledge you gain. And you also need to set them again and again whenever you need to restart gdb.

So instead of remembering them or writing them down, you can just save them to a file with this command:

(gdb) save breakpoints [filename]

From then on you can use the following command to restore them:

(gdb) source [filename]

Working more comfortably using the Text User Interface (TUI)

The default user interface of gdb is functional but it feels a bit like it was made for computers that display their output not onto a modern flat screen but print it onto continuous paper via a dot matrix printer.

Fortunately, gdb supports an alternate user interface that uses the block graphics capabilities of your terminal emulator to its advantage. The greatest benefit of this interface is that it shows the source code in a window separate from your command prompt. Therefore you no longer have to type the source command to see where you are in the source code of your program but see it all the time and you can just scroll through the code like with less or a text editor.

You can enable TUI with:

(gdb) tui enable

Now you see gdb’s tui mode in all its glory:

In the top window, you see your source code and the bottom window is the window where you enter your commands and where variables show up when you print them.

With two windows only one window can have the focus at any point in time. This sounds obvious but at first, it can be a bit confusing if you press up or down on your cursor keys but instead of cycling through your command history you see your source code move up and down.

Another reason why TUI can be confusing at first is that the mental model of windows and a window focus is not exactly true. It is true for the cursor keys but not for the letters on your keyboard. The text you enter always goes to the command window. And why should it go elsewhere? You can only watch the source code, not change it.

When you enter TUI the focus is on the source code window. So if you press the cursor up and down keys you can look around in the source code. You can also enter gdb commands with enter and execute them by pressing enter. Only if you want to use the command history you have to change the focus to the command window:

(gdb) focus cmd

Now you can use the command history with the cursor keys but are no longer able to scroll the source code.

To change the focus back to the source command enter:

(gdb) focus src

Instead of changing the focus back and forth between the source window and the command window, you could also use the shortcuts Ctrl-p and Ctrl-n to browse through the command history, regardless of which window has the focus.

Of course, TUI mode is still pretty bare-bones. So why is it a good idea to use it? There are fully graphical gdb frontends that are based on modern UI toolkits like Qt and Gtk, after all.

There are two reasons why gdb’s tui is probably the best UI for gdb:

1. It is built into gdb
2. It is as cross-platform as it gets

With TUI you have a reasonable level of comfort while at the same time being able to debug in nearly every environment out there without needing any extra time to get started. Now imagine you are used to this super fancy graphical gdb frontend, instead. And out of the blue, you need to debug a problem on somebody else’s computer, on a box in the cloud, or even via a serial link cable.

If you are used to tui you can switch environments with ease and just get the job done.

Sometimes less is actually more.

Breaking on changing Variables with Watchpoints

Sometimes you don’t know at which location in the program you want the program to break. You only know that you want the program to break whenever a certain variable is changed or read.

Let’s say we have a program that prints (reads) and changes a global integer variable:

#include <stdio.h>

int global_int = 0;

int main()
{
    printf("global_int before: %d\n", global_int);
    global_int = 42;
    printf("global_int after: %d\n", global_int);
}

Save the program as watch.c and compile it with:

$ gcc -ggdb -o watch watch.c

Start the debugger with:

$ gdb watch

To break the program and drop into the debugger when the variable is changed we can create a watchpoint with this command:

(gdb) watch global_int

To create a watchpoint that breaks the program whenever global_int is read use this command:

(gdb) rwatch global_int

In case you want to break whenever a variable is read OR written, you can use this command:

(gdb) awatch global_int

Note that those commands don’t take only a variable name but a full-fledged expression as a parameter. That means you can also set watchpoints on members of arrays or structs, pointers, as well as deeply nested data members.

All watchpoints currently in existence can be listed with the following command:

(gdb) info watch

While not very useful in such a simple program watchpoints are incredibly useful in complex programs where it can be very hard to find all the places in the code where a variable is used.

Attaching gdb to a running Program

There are situations when a program is already running and it already got into an error state that is hard to reproduce. Sometimes you don’t even know how to reproduce it. In such cases starting the program with gdb is not an option.

Fortunately, gdb allows you to attach itself to an already running process.

You just need to find the PID (process id) of your program, e.g. by entering:

ps aux|grep PROGRAMNAME

Once you know the PID you can then attach gdb to your running process:

gdb -p PID

On some newer Linux systems, you might get an error like “Operation not permitted” although the target process and gdb run as the same user. In that case, you need to execute the command sudo bash -c "echo 0 > /proc/sys/kernel/yama/ptrace_scope" before you can attach gdb to the process.

As soon as it starts gdb will halt the execution of the program. This means you will end up at a random position in the code. To find out where you are just use the backtrace command:

(gdb) backtrace

You can then either step through the program beginning from this position (which often doesn’t work because the program might be several function calls deep into a system library) or set some breakpoints at promising locations in your program and continue execution to (hopefully) end up in one of those locations.

Customizing GDB with your own Startup File

Changing gdb settings to your favorite values whenever you start gdb is quite annoying. But no reason to despair. gdb has a user-specific startup file so you can just configure gdb once.

To use this feature you just have to create a file .gdbinit in your home directory. In this file you can configure gdb like you do in the interactive gdb command prompt:

tui enable
set print pretty on

This example file enables the tui, as well as pretty printing of variables (important when printing big structures).

Conclusion

Gdb is a great and very versatile debugger and it allows you to get more insight into the execution of your programs than printf debugging alone could ever give you.

Hopefully, this article has shown you why it is beneficial to know gdb and how to use it for basic debugging. Of course, there is so much more it can do, but the commands and features described in this article should be enough to get you started.