Java_openVZ_Futex_issue, version 1
Anthony Callegaro
parent
800cd923b1
commit
4c6593d00c
122
Java_openVZ_Futex_issue.md
Normal file
122
Java_openVZ_Futex_issue.md
Normal file
@ -0,0 +1,122 @@
|
||||
Java and openVZ 2.6.32 - Futex issue
|
||||
====================================
|
||||
|
||||
1. The context
|
||||
--------------
|
||||
|
||||
We are currently in the process of upgrading all our servers.
|
||||
|
||||
Thanks to our [HAVEN High Availability
|
||||
architecture](http://www.personalized-software.ie/Services#Hosting) we
|
||||
can migrate virtual servers from one host to its slave with virtually no
|
||||
down time. So once the new servers were installed we started testing the
|
||||
migration of containers.
|
||||
|
||||
HAVEN being based on openVZ we were using kernels from Proxmox as they
|
||||
backport all fixes from RedHat 6. The only difference between the hosts
|
||||
was that the new servers had to use the 2.6.32 kernel, whereas the old
|
||||
ones were deployed with 2.6.24.
|
||||
|
||||
2. The issue
|
||||
------------
|
||||
|
||||
During the migration testing we noticed that one of our Java application
|
||||
(BigBlueButton) was crashing. It looks as if it was running fine but it
|
||||
wasn’t opening its ports and you’d have to do a `kill -9` to stop it.
|
||||
|
||||
Some quick investigation with strace showed that one of BigBlueButton
|
||||
requirements (ActiveMQ) was waiting on one of its children which was
|
||||
stuck in a infinite loop :
|
||||
|
||||
- strace of the parent process
|
||||
strace -p PID
|
||||
Process PID attached - interrupt to quit
|
||||
futex(0xb77dfbd8, FUTEX_WAIT, FIRST_CHILD_PID, NULL^C <unfinished ...>
|
||||
Process PID detached
|
||||
|
||||
- Get all child processes
|
||||
ps -efL | grep PID
|
||||
|
||||
- strace-ing through the children list I found one that was looping
|
||||
infinitely :
|
||||
strace -p CHILD_PID
|
||||
futex(0x998e028, FUTEX_WAKE_PRIVATE, 1) = 0
|
||||
gettimeofday({1338478277, 139155}, NULL) = 0
|
||||
clock_gettime(CLOCK_REALTIME, {1338478277, 139659879}) = 0
|
||||
futex(0x998e044, FUTEX_WAIT_PRIVATE, 1, {0, 999495121}) = -1 ETIMEDOUT (Connection timed out)
|
||||
|
||||
A quick search on the openVZ bugzilla showed that it was indeed a [known
|
||||
issue](http://bugzilla.openvz.org/show_bug.cgi?id=2206) affecting Java
|
||||
application with openVZ 2.6.32 kernels BUT there is no bugfix, nor
|
||||
workaround to the problem. Comments from other bug-posters were also
|
||||
less than encouraging.
|
||||
|
||||
Receiving no answer from the openVZ devs I decided to do more
|
||||
investigations on the issue.
|
||||
|
||||
3. Investigation and workaround
|
||||
-------------------------------
|
||||
|
||||
I first tried to reproduce the problem on a local virtual machine by
|
||||
installing Proxmox 1.9 and creating a container with
|
||||
[BigBlueButton](opensource:BigBlueButton\_Debian\_Squeeze) in it.
|
||||
|
||||
At first I couldn’t reproduce the issue, even though I was reproducing
|
||||
it on a test container on a remote server where it would fail on any
|
||||
threaded Java program.
|
||||
|
||||
Because it is a futex related problem I wondered if it was due to SMP so
|
||||
I added a new processor to my virtual machine and this time I reproduced
|
||||
it.
|
||||
|
||||
All our containers in production have the CPUS= parameter set but for
|
||||
some reasons containers running on 2.6.24 where still seeing all the
|
||||
hosts CPUs even if only 1 was in the configuration file. This seems to
|
||||
have been corrected in 2.6.32 and this is probably the reason why Java
|
||||
is now crashing.
|
||||
|
||||
Java is already suffering of very annoying memory issues when running
|
||||
inside containers that oblige us to run everything with the -Xmx,-Xms,
|
||||
XX:MaxPermSize etc. parameters and it seems that even if the container
|
||||
has only 1 CPU it tries to use more.
|
||||
|
||||
Java does not provide any CPU affinity options as the process scheduler
|
||||
is part of the OS. Fortunately openVZ has a very handy settings called
|
||||
CPUMASK that allows you to force a Container to run on only one specific
|
||||
CPU.
|
||||
|
||||
After trying a `vzctl set XXX --cpumask 0 --save` on my test environment
|
||||
the issue disappeared !
|
||||
|
||||
A quick test show that it also work for containers that requires
|
||||
multiple CPUs like this :
|
||||
|
||||
vzctl set XXX --cpus 2 --cpumask 0 --save
|
||||
|
||||
Also assigning more than 1 CPU to a container work around the problem.
|
||||
|
||||
I cannot guarantee that it will work for you but at least for us we have
|
||||
no more problem since implementing either of these tricks. We can now
|
||||
continue our migration testing.
|
||||
|
||||
4. Summary and Conclusion
|
||||
-------------------------
|
||||
|
||||
The issue is apparently affecting only :
|
||||
|
||||
- hosts with several cores/CPUs running any version of the
|
||||
2.6.32-openvz kernel (tested with debian squeeze, proxmox 1.9,
|
||||
proxmox 2.0 and vanilla patched kernel).
|
||||
- (debian) guests with only one CPU
|
||||
|
||||
Solution/workaround :
|
||||
|
||||
- Affect more than 1 CPU to the guest
|
||||
- Give CPU affinity (—cpumask) to the guest
|
||||
|
||||
This was quite tricky to debug so I hope this might help other people
|
||||
stuck with the same problem. Unfortunately once you know what the
|
||||
solution is you always find people who [found the
|
||||
same](http://forum.openvz.org/index.php?t=msg&th=10025&goto=43571&#msg_43571)
|
||||
In any case it cannot hurt to have more documentation about this :o)
|
||||
|
||||
Loading…
Reference in New Issue
Block a user