jvm hang and kill -3 & jmap failed

Question

I have a tomcat process that is:

1. all the threads is not running, all incoming connections is stuck on SYNC_RECV status
2. not responding to kill -3

3. jmap and jstack failed to attach to it

   sudo -u tomcat /usr/java/jdk1.6.0_45/bin/jmap -heap 19938
   Attaching to process ID 19938, please wait...
   Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process
   
   sudo -u tomcat /usr/java/jdk1.6.0_45/bin/jmap -heap -F 19938
   
   Attaching to process ID 19938, please wait...
   Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process
   

4. the output of gc stat is not changing, even the timestamp

   sudo -u tomcat /usr/java/jdk1.6.0_45/bin/jstat -gc -t 19938 1000 5
   
   Timestamp        S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
          370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
          370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
          370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
          370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
          370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
   

Environment information;

1. linux

   Linux xxxx 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
   

2. jdk

   java version "1.6.0_45"
   Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
   Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
   

3. JAVA_OPTS

   -server -Xms9g -Xmx9g -Xss256k -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=24 -XX:+UseParallelOldGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/tomcatdump  
   -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/tmp/tomcatlog.log -XX:NewSize=3g -XX:MaxNewSize=3g -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
   

UPDATES

The SYN_RECV status of connections is a problem of LVS, I think LVS detected the abnormal of tomcat and switch the service to other server, causing the last ACK delivered to another server, then the connections on this server will stuck on SYN_RECV status, but I don't see how that relevant to the jvm hang, huge thanks to @Stephen C

UPDATES2

This process has been stuck on this status for over a week, and the CPU usage is very low

Answer 1

For 1) ... that could be a networking problem; e.g. https://serverfault.com/questions/273807/all-connections-from-this-network-get-stuck-in-syn-recv-state-connections-from.

For 2) not sure. However note that the standard launch script for Tomcat on Linux redirects the standard output and standard error to a log file. Look for the thread dumps in the "catalina.out" log file.

For 3) ... according to https://stackoverflow.com/a/2943651/139985 you need to run jmap / jstack as the same user that launched the JVM. It is probably not root.

For 4) ... that could be explained by the fact that the GC hasn't needed to run.

---

UPDATES

It is also possible that SELinux may be getting in the way.

I'm afraid that while it might appear to you that the network is properly configured, there is definitely something wrong at that level. The SYN_RECV state is a state in TCP "3-way handshake" that happens when a connection is being established in the network stack. Java is not involved in that processes. Basically you have lots of Java threads that have attempted to initiate Socket connections, but the connects have all jammed up.