Proc will not be killed

Posted by Zeno on Sat 15 Jan 2005 06:08 PM — 12 posts, 28,194 views.

Zeno USA Sat 15 Jan 2005 06:08 PM #0

There are (for some unknown reason) 3 gdb proccesses running on my MUD proccess. I did not start them... But they will not KILL, even with -SIGKILL. There is only one user/connect on, and thats me, so its not hanging on any other users. I've tried exiting, but I log back on, and the 3 are still there.


zeno     11322  0.0  0.8 10784 6300 ?        T    Jan10   0:00 gdb inuyasha 11234

This is one, for example. Since this is Smaug, I figured I'd post here.

David Haley USA Sat 15 Jan 2005 07:08 PM #1

You mean you have one normal process for the MUD, and then three gdb processes hanging around? Have you tried actually shutting down the one you have access to, and seeing what's there?

Could you show a full set of the processes you're running? (e.g. ps -u username or however you normally get those.)

Zeno USA Sat 15 Jan 2005 07:18 PM #2


USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
zeno     11322  0.0  0.8 10784 6300 ?        T    Jan10   0:00 gdb inuyasha 11234
zeno     27737  0.0  0.8 10272 6252 ?        T    11:03   0:00 gdb inuyasha 27677
zeno     27783  0.0  0.8 11148 6252 ?        T    11:04   0:00 gdb inuyasha 27677
zeno     28029  0.0  0.2  8696 2052 ?        S    11:11   0:00 sshd: zeno@pts/5
zeno     28030  0.0  0.1  5856 1348 pts/5    S    11:11   0:00 -bash
zeno     28084  0.0  0.1  5580 1216 pts/5    S    11:13   0:00 /bin/csh -f ./startup
zeno     28591  0.0  0.5  7824 4628 pts/5    S    12:05   0:00 inuyasha 1801 hotboot 5 m ? ë? -1
zeno     28761  0.0  0.0  3536  736 pts/5    R    12:19   0:00 ps ux

I've tried shutting the MUD down, kill all other processes, etc.

David Haley USA Sat 15 Jan 2005 07:50 PM #3

OK... so if you shut down the MUD process, you should be left with:


USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
zeno     11322  0.0  0.8 10784 6300 ?        T    Jan10   0:00 gdb inuyasha 11234
zeno     27737  0.0  0.8 10272 6252 ?        T    11:03   0:00 gdb inuyasha 27677
zeno     27783  0.0  0.8 11148 6252 ?        T    11:04   0:00 gdb inuyasha 27677
zeno     28029  0.0  0.2  8696 2052 ?        S    11:11   0:00 sshd: zeno@pts/5
zeno     28030  0.0  0.1  5856 1348 pts/5    S    11:11   0:00 -bash
zeno     28761  0.0  0.0  3536  736 pts/5    R    12:19   0:00 ps ux

So you're saying that typing:
kill -9 11322
kill -9 27737
kill -9 27783
doesn't work?

Interestingly enough, the 'T' in the process status means that the process is: "stopped, either by a job control signal or because it is being traced". (man page) Is either of these a possibility? Did you perhaps get out of one of the gdbs by typing ctrl-z? Even so, you'd think that a forced kill would work...

Zeno USA Sat 15 Jan 2005 08:00 PM #4

Like I said, I never started those gdbs. I went to go and try to gdb the MUD process, but it said one already existed. The only thing I can think of is my startup auto-debug script.

    if ( -e core ) then
      set index = 1000
      while ( 1 )
        set crashfile = ../crash/$index.crash
            if ( ! -e $crashfile ) break
            @ index++
      end
      mv core ../src
      cd ../src
      date > $crashfile
      gdb -batch -x commands inuyasha core >> $crashfile
      mv core ../crash/core.$index
#     rm -f core
      cd ../area
    endif

But why the gdb would never stop I'm not sure. But I still cannot kill those processes even with -9 when there was nothing else running.

David Haley USA Sat 15 Jan 2005 08:51 PM #5

What are the contents of the file 'commands'?

I suppose you might have to ask your sysadmin to force shutdown the process. I'm still a little surprised that kill -9 isn't working.

Zeno USA Sat 15 Jan 2005 09:08 PM #6

[zeno@boralis iyg]$ ps ux
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
zeno     11322  0.0  0.8 10784 6300 ?        T    Jan10   0:00 gdb inuyasha 11234
zeno     27737  0.0  0.8 10272 6252 ?        T    11:03   0:00 gdb inuyasha 27677
zeno     27783  0.0  0.8 11148 6252 ?        T    11:04   0:00 gdb inuyasha 27677
zeno     29550  0.0  0.2  7156 2048 ?        S    14:07   0:00 sshd: zeno@pts/6
zeno     29551  0.0  0.1  5648 1336 pts/6    S    14:07   0:00 -bash
zeno     29582  0.0  0.0  3788  740 pts/6    R    14:07   0:00 ps ux
[zeno@boralis iyg]$ kill -9 11322
[zeno@boralis iyg]$ ps ux
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
zeno     11322  0.0  0.8 10784 6300 ?        T    Jan10   0:00 gdb inuyasha 11234
zeno     27737  0.0  0.8 10272 6252 ?        T    11:03   0:00 gdb inuyasha 27677
zeno     27783  0.0  0.8 11148 6252 ?        T    11:04   0:00 gdb inuyasha 27677
zeno     29550  0.0  0.2  7156 2048 ?        S    14:07   0:00 sshd: zeno@pts/6
zeno     29551  0.0  0.1  5648 1336 pts/6    S    14:07   0:00 -bash
zeno     29583  0.0  0.0  2324  740 pts/6    R    14:07   0:00 ps ux

All are the same. They will not kill. Although where did "commands" come from... I never put it in the script before. -x means to exclude right? The file doesn't exist.

[EDIT] Perhaps not. It means to execute. Odd... But the script has always worked.

Samson USA Sat 15 Jan 2005 09:39 PM #7

Mighty strange situation this is.

Even as root I am unable to SIGKILL the processes.
There is no reason they should be being traced. I never asked the system to do that.

I'd rather avoid having to reboot the server over something like this. It's not really affecting things much just yet.

I had the same thing happen with a Valgrind process that died and was not able to be stopped. That one I had little choice but to reboot because it eventually overran all the resources and refused to be put down.

If anyone has any advice on what to do about this I'd appreciate it. Would hate to see this start ballooning up into something nasty.

Samson USA Sat 15 Jan 2005 09:52 PM #8

Oooo.... solved it.

For future reference to others - if this ever happens to you, try this:

gdb attach <PID#>

When I did that, I got the following back ( as root ):

Quote:

Attaching to process 11322
Reading symbols from /usr/bin/gdb...(no debugging symbols found)...done.
Using host libthread_db library "/lib/tls/libthread_db.so.1".
Reading symbols from /usr/lib/libncurses.so.5...done.
Loaded symbols for /usr/lib/libncurses.so.5
Reading symbols from /lib/libtermcap.so.2...done.
Loaded symbols for /lib/libtermcap.so.2
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/tls/libthread_db.so.1...done.
Loaded symbols for /lib/tls/libthread_db.so.1
linux_test_for_tracefork: waitpid: No child processes.
A program is being debugged already. Kill it? (y or n) y

When I answered yes, the system shut the rogue process down.

[EDIT]

Or... not. The processes were removed from his account and transferred to root. So I'm not sure what the deal was. Since I had a kernel update pending I just rebooted the server.

Zeno USA Sat 15 Jan 2005 10:44 PM #9

Odd enough. When you explained how you solved it, I didn't see how that would work since it's basically killing the process the same way, one way or another. Although transfering to the root, even more odd. I've commented out the script for now...

Zeno USA Sat 26 Feb 2005 03:33 AM #10

Going to attempt to add the script back in since I found the commands file. But I want to make sure this is "safe" to add before I do, so this problem doesn't happen again.

In startup script:

    if ( -e core ) then
      set index = 1000
      while ( 1 )
        set crashfile = ../crash/core.$index
            if ( ! -e $crashfile ) break
            @ index++
      end
      mv core ../src
      cd ../src
      date > $crashfile
      gdb -batch -x commands inuyasha core >> $crashfile
      mv core $crashfile
      cd ../area
    endif

Commands file:

bt
frame 0
list
info locals
frame 1
list
info locals
frame 2
list
info locals
frame 3
list
info locals
frame 4
list
info locals
frame 5
list
info locals

Samson USA Sat 26 Feb 2005 06:45 AM #11

There should be nothing unsafe about that commands file. Or the startup script. I'm still not entirely sure how the situation came about, will just need to keep an eye on it and try to narrow it down if it does happen.