They Did Surgery on a Process

This memo provides information for the curious and the on-call. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited, and comments are welcome by email (saksh.am [at] outlook [dot] in). I don’t trust comment boxes online.

A long deployment script got stuck in a loop that could never finish, and restarting it was not an option. This memo describes how the loop was unstuck by attaching a debugger to the live process and changing an environment variable in its memory.

Unlike the first two memos, this one is not theory. I just faced this, and it was too good not to write down while it is still fresh.

The short version: a deployment script was stuck, I did not want to restart it, and a debugger let me reach into the running process and fix the one value that was wrong. Let’s go through it.

A massive, long-running production deployment script (cron.sh) was stuck in an infinite until loop.

Somewhere upstream, a file came out empty. Because of that, the environment variable $MODULE_ID resolved to an empty string. The part that matters is this:

MODULE_ID=`cat module_internal_cid_qa`
cat $MODULE_ID
sudo docker start `cat module_internal_cid_qa`
HEALTH_STATUS=`sudo docker inspect --format='{{json .State.Health.Status}}' $MODULE_ID`
echo $HEALTH_STATUS
until [ "`sudo docker inspect --format='{{json .State.Health.Status}}' $MODULE_ID`" == "\"healthy\"" ]
do
    echo "waiting for container to become healthy";
    sleep 5;
    HEALTH_STATUS=`sudo docker inspect --format='{{json .State.Health.Status}}' $MODULE_ID`
    echo $HEALTH_STATUS
done;
echo "Healthy"

The container ID is read from a file. That file was empty, so MODULE_ID became an empty string. Notice that $MODULE_ID is not quoted, so an empty value does not even become an empty argument. It vanishes, and the loop runs docker inspect with no container named at all. That fails every time, the status is never "healthy", and the loop never ends. It is not a slow wait. It is a wait for something that cannot happen.

Fine, just kill it and run it again?

That was the catch. The pipeline was long, and restarting it from the top was not an option.

First, find the process that is looping:

ps aux | grep bash

That pointed at process ID 1036, the bash running the script. The container I actually wanted it to watch had the ID c0105eaf12df.

How did you know the ID, if the file was empty?

The file was empty, but the container itself was fine. docker ps -a lists every container on the host, running or not, with its ID, image, status and name. The container I was after was right there in the list, so I could read its ID off the CONTAINER ID column.

sudo docker ps -a

So the situation was: a live process, one variable with the wrong value, and the right value known. If only there were a way to change a variable inside a process without stopping it for good.

Wait, can’t I just export MODULE_ID=c0105eaf12df in another terminal?

No, and the reason is worth a minute. Every process gets its own private view of memory, called its virtual address space. The kernel (with help from the CPU) translates the addresses a process uses into real memory, separately for each process. An address like 0x404034 in one process and the same number in another point at completely different things. Process A cannot even name process B’s variables, let alone change them.

Here is a tiny program that shows it. It has one global variable, counter, and prints that variable’s address and value:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int counter = 0; /* one global variable, one fixed address */

int main(int argc, char **argv) {
    counter = atoi(argv[1]);
    printf("pid %d: &counter = %p, counter = %d\n", getpid(), (void *)&counter, counter);
    sleep(2); /* stay alive so both processes overlap */
    printf("pid %d: &counter = %p, counter = %d\n", getpid(), (void *)&counter, counter);
    return 0;
}

Build it and run two copies at the same time, giving each a different number:

gcc -no-pie -o same_addr same_addr.c
./same_addr 111 & ./same_addr 222 & wait

(-no-pie gives the program a fixed load address, so the numbers line up. Without it, address randomisation would move counter around on every run.) The output:

pid 891887: &counter = 0x404034, counter = 111
pid 891887: &counter = 0x404034, counter = 111
pid 891888: &counter = 0x404034, counter = 222
pid 891888: &counter = 0x404034, counter = 222

Same address, 0x404034, in both processes. Yet one sees 111 and the other sees 222, and neither one’s value changes because of the other. (The process IDs will differ on your machine.) The address is not a place in physical memory. It is a label that the kernel translates, per process, to a different spot in real memory. So from process A’s point of view, there is no address that means “process B’s counter”.

Environment variables make this clearer. When a process starts another one, the child gets a copy of the parent’s environment, once, at that moment. After that the two are independent. Your export in a new terminal changes that shell’s environment and that of its future children. It does not reach back into a script that is already running.

So the normal ways to influence a running process are all polite ones: send it a signal, write a file it reads, talk to a socket it listens on. The process has to be built to listen. Our script was not.

There is exactly one sanctioned way around this wall, a system call named ptrace. It is what gdb and strace are built on. A process that has attached with ptrace can pause the target, read and write its memory and registers, and let it continue. The kernel guards it: you need to own the process (or be root), and many distributions further restrict it with ptrace_scope so that you can only attach to your own children. That is the reason for the sudo in the next sections.

Linux lets you look at a process’s memory from the outside through the /proc filesystem. Two files are enough for a tour:

cat /proc/1036/maps

This lists the process’s address space: the program’s code, the heap, the stack, and the shared libraries such as the C library. Each line is a range of addresses with its permissions. The variables of a running program live in the heap and stack ranges.

sudo cat /proc/1036/environ | tr '\0' '\n'

This prints the process’s environment, one NAME=value per line (the entries are separated by NUL bytes, hence the tr).

So I could have just edited that file?

No. /proc/PID/environ is a snapshot of the environment as it was when the process started. The kernel is showing you the bytes that were placed at the top of the stack at launch. If the program changes its environment later, those original bytes are not updated, so this file will disagree with what the process really sees. It is not a cache you can flush; it is simply a record of the beginning.

Where is the live value, then? Inside the process, in memory the program manages itself. In a typical C program the C library keeps an array of pointers, environ, and getenv and setenv walk and edit it, pointing an entry at a new string and leaving the original bytes alone.

Bash is a twist. It keeps its own table of shell variables, and it also ships its own copies of getenv and setenv that read and write that table. You can see this from gdb: info symbol setenv answers setenv in section .text of /usr/bin/bash, not libc. That is why this trick works on a bash script even though MODULE_ID was a plain shell variable that was never exported. (I checked this on bash 5.2.21 with a small test script. Other shells, such as dash, may behave differently.)

You can read the live value the same way the program does, by asking it, from inside gdb:

(gdb) call (char *) getenv("MODULE_ID")

There is: the GNU Debugger, gdb. It can attach to a process that is already running, without killing it.

Step 1: Attach to the live process.

sudo gdb -p 1036

The sudo is there because attaching to a process you don’t own, or on a system that restricts it, needs the privilege. While attached, the process is paused, and nothing about the script is lost.

Step 2: Call a C function inside the process.

A process can be asked to run its own functions from inside the debugger. The C library function setenv overwrites an environment variable, so at the (gdb) prompt:

call (int) setenv("MODULE_ID", "c0105eaf12df", 1)

The (int) cast tells gdb what the function returns, since it has no debug info to say so. The last argument, 1, means “overwrite if it already exists”. Without it, setenv leaves an existing variable alone, and an empty string still counts as existing.

Step 3: Let go.

detach
quit

detach releases the process and lets it carry on from exactly where it was paused.

So what did we actually bypass?

Not a cache, and not the kernel’s protections. Nothing from outside wrote into the script’s memory. What gdb did is pause the process, save its registers, set up a call to setenv as if the program itself had made it, run it, and restore the registers. The process edited its own variables, using its own setenv, in its own address space. We never had to find the variable’s address or touch /proc/1036/environ. We borrowed the process and asked it to do the job.

You can check the result before letting go, with the getenv call from the previous section. Run it before and after the setenv and you should see the empty string turn into the container ID.

The moment I detached, the script resumed. On the next iteration it evaluated the real container ID, saw the healthy status, broke out of the loop, and carried on to finish the deployment.

No restart, no re-running the long part of the pipeline, and no lost progress.

This is a scalpel, not a habit.

• You are writing into the memory of a live process. A wrong call can crash it, and then you have lost the thing you were trying to save.
• Know how the script reads the value before you change it. Here the loop looked the variable up again on each iteration, so the new value was picked up. If the script had copied it somewhere earlier, this would not have helped.
• Do it only when restarting is truly worse. If a restart is cheap, restart.
• Your machine may restrict attaching to other processes (see Section 4 and ptrace_scope in the references).

And the real fix lives upstream, in the script itself:

• Fail loudly on an empty value: MODULE_ID="${MODULE_ID:?MODULE_ID is empty}", or set -u for unset ones.
• Never wait forever. Put a limit on the loop, for example with timeout or an attempt counter, so a stuck wait turns into an error.

A running program is not a sealed box. With gdb -p you can attach, change one value, detach, and the program carries on none the wiser.

It also loops back to RFC 0001: an empty string quietly became an infinite wait, and nothing said so. A timeout and a loud failure would have told me in seconds instead of leaving me to notice.

[RFC0001] Saksham, “Observability, My Dear Watson”, blogs.saksh.fyi/observability-my-dear-watson.

[GDB] GNU Project, “Debugging with GDB”, sourceware.org/gdb/current/onlinedocs/gdb.

[PTRACE] Linux kernel documentation, “Yama”, kernel.org/doc/html/latest/admin-guide/LSM/Yama.