Details
-
Type:
Bug
-
Status:
Resolved
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: 3.6.4
-
Fix Version/s: 3.7.0 beta
-
Component/s: Core
-
Labels:None
-
Ignite Forum URL:
-
Acceptance Test - Add?:No
Description
Several users report that Openfire runs out of memory, after a user switches to the Empathy client.
-
- pep-rewrite-01.diff
- 12/31/09 10:55 PM
- 45 kB
- Guus der Kinderen
-
- pep-rewrite-02.diff
- 01/01/10 12:52 AM
- 45 kB
- Guus der Kinderen
Activity
Another discussion at http://www.igniterealtime.org/community/message/198163#198163
Telepathy-python comes with a set of example scripts which might be useful for testing. The following script runs the roster example, but could be used to run any of the examples:
- Install the required components, if they aren't already.
sudo aptitude install telepathy-python telepathy-gabble
- This isolates the rest of the script from any instances of empathy you may have running.
- It is also necessary if you are not running in an X session.
$(dbus-launch)
- Write your configuration somewhere that the example script can read it.
- Replace account and password with the correct ones.
- If your server requires any other funky configurations like server and port, add them in here.
cat > test_account.account <<EOF
manager: gabble
protocol: jabber
account: YOURACCOUNT@YOURDOMAIN.COM
password: WHATEVERYOURPASSWORDIS
EOF
for i in `seq 10` # Increase this number as high as you want once you know it works with 10.
do
python /usr/share/doc/python-telepathy/examples/roster.py test_account.account
done
Note that you will need to kill it using ^c each time it says 'waiting for changes'. If you want to speed it up, you can edit /usr/share/doc/python-telepathy/examples/roster.py and replace the line "print 'waiting for changes'" with "self.quit()"
Alternatively, you can run the roster scripts in parallel, using:
python /usr/share/doc/python-telepathy/examples/roster.py test_account.account &
inside the loop and then clean up at the end by doing:
kill -INT $(pgrep -f examples/roster.py)
I hope this helps.
- Install the required components, if they aren't already. sudo aptitude install telepathy-python telepathy-gabble
- This isolates the rest of the script from any instances of empathy you may have running.
- It is also necessary if you are not running in an X session. $(dbus-launch)
- Write your configuration somewhere that the example script can read it.
- Replace account and password with the correct ones.
- If your server requires any other funky configurations like server and port, add them in here. cat > test_account.account <<EOF manager: gabble protocol: jabber account: YOURACCOUNT@YOURDOMAIN.COM password: WHATEVERYOURPASSWORDIS EOF
erm... so it seems to have mangled my comments. I also can't seem to see an edit or preview button, so I can only hope that when I post it again here it will work.
# Install the required components, if they aren't already. sudo aptitude install telepathy-python telepathy-gabble # This isolates the rest of the script from any instances of empathy you may have running. # It is also necessary if you are not running in an X session. $(dbus-launch) # Write your configuration somewhere that the example script can read it. # Replace account and password with the correct ones. # If your server requires any other funky configurations like server and port, add them in here. cat > test_account.account <<EOF manager: gabble protocol: jabber account: YOURACCOUNT@YOURDOMAIN.COM password: WHATEVERYOURPASSWORDIS EOF for i in `seq 10` # Increase this number as high as you want once you know it works with 10. do python /usr/share/doc/python-telepathy/examples/roster.py test_account.account done
# Install the required components, if they aren't already. sudo aptitude install telepathy-python telepathy-gabble # This isolates the rest of the script from any instances of empathy you may have running. # It is also necessary if you are not running in an X session. $(dbus-launch) # Write your configuration somewhere that the example script can read it. # Replace account and password with the correct ones. # If your server requires any other funky configurations like server and port, add them in here. cat > test_account.account <<EOF manager: gabble protocol: jabber account: YOURACCOUNT@YOURDOMAIN.COM password: WHATEVERYOURPASSWORDIS EOF for i in `seq 10` # Increase this number as high as you want once you know it works with 10. do python /usr/share/doc/python-telepathy/examples/roster.py test_account.account done
This issue appears to be caused by a bug in the PEP (XEP-0163: Personal Eventing Protocol) implementation.
In a local setup using Openfire 3.6.4 and Empathy 2.28.1, I have observed the following:
- The Empathy client reconnects to Openfire regularly (roughly once every five minutes).
- After each restart, the Empathy client publishes an avatar.
- After each restart, the instance count of org.jivesoftware.openfire.pep.PEPService grows with one. Each instance references (indirectly) avatar data.
- The pepServices instance field (a Map) of IQPEPHandler does not appear to contain multiple instances for the same user.
In effect, Openfire keeps another avatar in memory, every time that the Empathy client is reconnected. Two copies of the avatar data are stored (raw data and XML based), which adds to the impact of this bug.
Workaround: Disable PEP by setting the property xmpp.pep.enabled to false
- The Empathy client reconnects to Openfire regularly (roughly once every five minutes).
- After each restart, the Empathy client publishes an avatar.
- After each restart, the instance count of org.jivesoftware.openfire.pep.PEPService grows with one. Each instance references (indirectly) avatar data.
- The pepServices instance field (a Map) of IQPEPHandler does not appear to contain multiple instances for the same user.
The Empathy client gets disconnected, because it is detected to be idle. Its keep-alives apear to be TCP based, which does not get picked up by Openfire. Instead, Empathy should either send whitespace pings (Smack works this way) or valid XMPP data (such as XEP-0199: XMPP Ping, what Pidgin does).
Note that the cause of the OOM lies with Openfire, not Empathy. The original bug should be fixed in Openfire. Empathy could be improved to reduce the impact of this bug (additionally, not being reconnected every few minutes of idle should be a welcome general improvement for Empathy users).
Another user reports memory-related issues which are worked around by disabling PEP: http://www.igniterealtime.org/community/message/198610#198610
I have attached a first attempt to rewrite significant parts of the PEP routines. The memory leak should now be gone (things are cached in a limited cache now). Care to test, anyone?
Hi Guus,
Thanks for efforting this! I tried to apply your patch against a current trunk and got two failures:
patching file src/java/org/jivesoftware/util/cache/CacheFactory.java
Hunk #1 FAILED at 128.
Hunk #2 FAILED at 200.
2 out of 2 hunks FAILED – saving rejects to file src/java/org/jivesoftware/util/cache/CacheFactory.java.rej
patching file src/plugins/clustering/coherence-cache-config.xml
Hunk #1 FAILED at 883.
1 out of 1 hunk FAILED – saving rejects to file src/plugins/clustering/coherence-cache-config.xml.rej
Hmm, odd. I recreated the diff - could you try with the 02 one?
Sorry for the noise, those two files have CRLF on subversion and was confusing patch
The patch has been committed. If any problems arise, we'll reopen the issue.
I'm getting errors like:
2010.01.12 22:19:52 Internal server error. Triggered by packet: <presence id="Cck2L-3" from="alice@domain/Smack"/> java.lang.NullPointerException at org.jivesoftware.openfire.pep.IQPEPHandler.availableSession(IQPEPHandler.java:586) at org.jivesoftware.openfire.user.PresenceEventDispatcher.availableSession(PresenceEventDispatcher.java:81) at org.jivesoftware.openfire.session.LocalClientSession.setPresence(LocalClientSession.java:754) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:149) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:135) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:199) at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:149) at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:85) at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:84) at org.jivesoftware.openfire.SessionPacketRouter.route(SessionPacketRouter.java:120) at org.jivesoftware.openfire.SessionPacketRouter.route(SessionPacketRouter.java:75) at org.jivesoftware.openfire.http.HttpSession.sendPendingPackets(HttpSession.java:631) at org.jivesoftware.openfire.http.HttpSessionManager$HttpPacketSender.run(HttpSessionManager.java:389) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at java.lang.Thread.run(Thread.java:595)
and
2010.01.13 21:10:07 org.jivesoftware.openfire.pep.PEPService java.io.NotSerializableException: org.jivesoftware.openfire.pep.PEPService at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1081) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302) at org.jivesoftware.util.cache.DefaultCache.calculateSize(DefaultCache.java:583) at org.jivesoftware.util.cache.DefaultCache.put(DefaultCache.java:141) at org.jivesoftware.util.cache.CacheWrapper.put(CacheWrapper.java:129) at org.jivesoftware.openfire.pep.PEPServiceManager.create(PEPServiceManager.java:110) at org.jivesoftware.openfire.pep.IQPEPHandler.handleIQ(IQPEPHandler.java:337) at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:65) at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:371) at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:121) at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76) at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:336) at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler.java:87) at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:301) at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:193) at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandler.java:144) at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived(AbstractIoFilterChain.java:570) at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299) at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53) at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648) at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80) at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299) at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53) at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648) at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimpleProtocolDecoderOutput.java:58) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:185) at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299) at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53) at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648) at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:239) at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:283) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51) at java.lang.Thread.run(Thread.java:595)
in the error log.
2010.01.12 22:19:52 Internal server error. Triggered by packet: <presence id="Cck2L-3" from="alice@domain/Smack"/> java.lang.NullPointerException at org.jivesoftware.openfire.pep.IQPEPHandler.availableSession(IQPEPHandler.java:586) at org.jivesoftware.openfire.user.PresenceEventDispatcher.availableSession(PresenceEventDispatcher.java:81) at org.jivesoftware.openfire.session.LocalClientSession.setPresence(LocalClientSession.java:754) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:149) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:135) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:199) at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:149) at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:85) at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:84) at org.jivesoftware.openfire.SessionPacketRouter.route(SessionPacketRouter.java:120) at org.jivesoftware.openfire.SessionPacketRouter.route(SessionPacketRouter.java:75) at org.jivesoftware.openfire.http.HttpSession.sendPendingPackets(HttpSession.java:631) at org.jivesoftware.openfire.http.HttpSessionManager$HttpPacketSender.run(HttpSessionManager.java:389) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at java.lang.Thread.run(Thread.java:595)
2010.01.13 21:10:07 org.jivesoftware.openfire.pep.PEPService java.io.NotSerializableException: org.jivesoftware.openfire.pep.PEPService at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1081) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302) at org.jivesoftware.util.cache.DefaultCache.calculateSize(DefaultCache.java:583) at org.jivesoftware.util.cache.DefaultCache.put(DefaultCache.java:141) at org.jivesoftware.util.cache.CacheWrapper.put(CacheWrapper.java:129) at org.jivesoftware.openfire.pep.PEPServiceManager.create(PEPServiceManager.java:110) at org.jivesoftware.openfire.pep.IQPEPHandler.handleIQ(IQPEPHandler.java:337) at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:65) at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:371) at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:121) at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76) at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:336) at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler.java:87) at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:301) at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:193) at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandler.java:144) at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived(AbstractIoFilterChain.java:570) at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299) at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53) at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648) at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80) at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299) at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53) at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648) at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimpleProtocolDecoderOutput.java:58) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:185) at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299) at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53) at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648) at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:239) at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:283) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51) at java.lang.Thread.run(Thread.java:595)
Hi Guenther,
I believe I corrected the first bug you mentioned with r11535
The second one I am seeing as well and will need Guus' help on.
daryl
I've added a workaround, by having PEPService implement the Cacheable interface. I've opted to not dynamically calculate the cached size (that would be rather intensive). Instead, I've put a rather arbitrary number. I've update the javadoc with a warning not to cache instances in caches that use size-based eviction policies (use time-based ones instead).
This workaround feels a bit like a hack - but that goes for the entire PEP implementation. We should redo this properly in a next release.
Stacktraces are gone, PEP events are echoed back to the user that initiates them and to contacts.
The bug is not fixed as it should in 3.7.0 beta. I have written a post (http://community.igniterealtime.org/thread/43054) and proposed a patch. The post has been moderated and the patch propositions removed...
Hello vinz,
Sorry that the patch was moderated out, I am not sure why that was done. Please feel free to try to attach the patch again or email it directly to me if you wish: akrherz@iastate.edu
daryl
Community discussion at http://www.igniterealtime.org/community/message/198145#198145