Skip to content

Session listeners are not called back in redis cluster #478

Closed
@sanketmeghani

Description

@sanketmeghani

@rwinch We are using spring-session 1.2.0.RC1 and spring-data-redis 1.7.0.RC1. Somehow we are not getting the listener call back if we use redis cluster. It works perfect if we use a stand-alone redis server. I have uploaded a sample springSessionContext.xml.

On one of the nodes in redis cluster, we see "SUBSCRIBE" "__keyevent@:expired" "spring:session:event:created:" "__keyevent@*:del" command. So it does subscribe on one node. Should it subscribe to all nodes in the cluster? Or this is expected behavior?

Sample application code is available here.

Activity

rwinch

rwinch commented on Apr 11, 2016

@rwinch
Member

Thanks for the report!

cc @christophstrobl

leehanwool

leehanwool commented on Apr 12, 2016

@leehanwool

I experienced same situation at tomcat 8 with spring-web-application(spring-data-redis 1.7.1.RELEASE) and redis-cluster(3.0.7).
When I stop tomcat, I received WARNING message like below, then tomcat going to hanging status, so I have to kill tomcat myself.(sometimes stop tomcat oneself after few minutes.)
would you please tell me about release plan with this issue?

[WARN] [RedisMessageListenerContainer$SubscriptionTask:864] : Unable to close connection. Subscription task still running
 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [] appears to have started a thread named [redisMessageListenerContainer-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.net.SocketInputStream.socketRead0(Native Method)
 java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
 java.net.SocketInputStream.read(SocketInputStream.java:170)
 java.net.SocketInputStream.read(SocketInputStream.java:141)
 java.net.SocketInputStream.read(SocketInputStream.java:127)
 redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:195)
 redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40)
 redis.clients.jedis.Protocol.process(Protocol.java:141)
 redis.clients.jedis.Protocol.read(Protocol.java:205)
 redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)
 redis.clients.jedis.Connection.getRawObjectMultiBulkReply(Connection.java:242)
 redis.clients.jedis.BinaryJedisPubSub.process(BinaryJedisPubSub.java:87)
 redis.clients.jedis.BinaryJedisPubSub.proceed(BinaryJedisPubSub.java:82)
 redis.clients.jedis.BinaryJedis.subscribe(BinaryJedis.java:2980)
 redis.clients.jedis.BinaryJedisCluster$150.execute(BinaryJedisCluster.java:1607)
 redis.clients.jedis.BinaryJedisCluster$150.execute(BinaryJedisCluster.java:1604)
 redis.clients.jedis.JedisClusterCommand.runWithAnyNode(JedisClusterCommand.java:87)
 redis.clients.jedis.BinaryJedisCluster.psubscribe(BinaryJedisCluster.java:1610)
 org.springframework.data.redis.connection.jedis.JedisClusterConnection.pSubscribe(JedisClusterConnection.java:2508)
 org.springframework.data.redis.listener.RedisMessageListenerContainer$SubscriptionTask.eventuallyPerformSubscription(RedisMessageListenerContainer.java:779)
 org.springframework.data.redis.listener.RedisMessageListenerContainer$SubscriptionTask.run(RedisMessageListenerContainer.java:746)
 java.lang.Thread.run(Thread.java:745)
rwinch

rwinch commented on Apr 12, 2016

@rwinch
Member

@leehanwool I haven't had a chance to look into this issue yet. Note that we also have a ticket for official Redis Cluster support that we need to get to. See #422

As for your StackTrace, could you please create a separate ticket for the unclosed thread?

christophstrobl

christophstrobl commented on Apr 12, 2016

@christophstrobl
Member

@rwinch @leehanwool created DATAREDIS-493 to check that.

leehanwool

leehanwool commented on Apr 13, 2016

@leehanwool

@rwinch
Thank for reply.
I created new #482 ticket for Stacktrace.

nirav-patel

nirav-patel commented on Apr 15, 2016

@nirav-patel

@rwinch @leehanwool @christophstrobl
I have done spring-session configuration with Redis by using Jedis Client (2.8.1 version) as mentioned in issue description by @sanketmeghani above. I found that internally when cluster is created, each node gets subscribed to patterns for event notifications (basically those are session creation, session expiration and session deletion events) via method

void redis.clients.jedis.BinaryJedisCluster.psubscribe(BinaryJedisPubSub jedisPubSub, byte[]... patterns)

Actual implementation of the method in BinaryJedisCluster class is as shown below:
Note in execute method, Jedis Connection is subscribed to patterns using subscribe method [i.e. connection.subscribe(jedisPubSub, patterns);].

@Override
  public void psubscribe(final BinaryJedisPubSub jedisPubSub, final byte[]... patterns) {
    new JedisClusterCommand<Integer>(connectionHandler, maxRedirections) {
      @Override
      public Integer execute(Jedis connection) {
        connection.subscribe(jedisPubSub, patterns);
        return 0;
      }
    }.runWithAnyNode();
  }

Here If I override this method with custom implementation and subscribe each Jedis Connection in execute method with psubscribe method instead of subscribe method [i.e., connection.psubscribe(jedisPubSub, patterns)], then I`m able to retrieve session notifications for creation, expiration and deletion activities performed by Redis Cluster.

@Override
  public void psubscribe(final BinaryJedisPubSub jedisPubSub, final byte[]... patterns) {
    new JedisClusterCommand<Integer>(connectionHandler, maxRedirections) {
      @Override
      public Integer execute(Jedis connection) {
        connection.psubscribe(jedisPubSub, patterns);
        return 0;
      }
    }.runWithAnyNode();
  }

In case of single Redis node implementation, Jedis Connection is subscribed to patterns using psubscribe method and not the subscribe method. So I applied the same logic for Redis Cluster use-case too and things are working fine.

I don`t know whether it is a Jedis issue or current implementation is correct for some purpose.
Is this a correct way to fix above issue and get session listeners call backs from Redis Cluster?

@rwinch @leehanwool @christophstrobl Please provide your feedback for this solution.

rwinch

rwinch commented on Apr 15, 2016

@rwinch
Member

@nirav-patel Thanks for the detailed writeup!

I think @christophstrobl will be able to provide better feedback on your solution

leehanwool

leehanwool commented on Apr 18, 2016

@leehanwool

@nirav-patel , Thank you for advise!
I tried to use your solution in my case #482
It works perfectly. Tomcat no more going to hanging status, So I don't need kill tomcat anymore.
Thank you!!!

christophstrobl

christophstrobl commented on Apr 18, 2016

@christophstrobl
Member

thanks @nirav-patel for detailed information.

HeartSaVioR

HeartSaVioR commented on Apr 18, 2016

@HeartSaVioR

Thanks all for finding missed spot for Jedis.

self-assigned this
on May 2, 2016
rwinch

rwinch commented on May 2, 2016

@rwinch
Member

Thanks for the reports and helping sort this out! I'm closing this as it appears the issue has been fixed in Jedis

horance

horance commented on Jan 11, 2018

@horance

@rwinch Sorry about commenting on closed issue. I ran into the issue that @sanketmeghani mentioned.

I've done a sample project SpringSessionTest with spring-session 1.3.0.RELEASE and spring-data-redis 1.8.3.RELEASE. I've found redis message listener container only subscribed to one node. According to https://github.com/antirez/redis/issues/2541, the keyspace notifications are only sent locally. As a result, the session-expired event will not be processed by listener, unless the key was created on the subscribed node.

Following is my test result with my test application, I send two http request to my test application with curl:

HoranceMBP:SpringSessionTest horance$ curl http://localhost:8080/hello 
The session ID is :bf182e9c-a35e-4ce4-a8c0-f5ddb68372cc, session expires key stored on node: 127.0.0.1:7001, psub pattern count is 3, will receive expired event.
HoranceMBP:SpringSessionTest horance$ curl http://localhost:8080/hello 
The session ID is :407240ff-7d09-41db-858b-fdc44186e6cb, session expires key stored on node: 127.0.0.1:7002, psub pattern count is 0, will not receive expired event.

And after couple minutes, the log shows the first session expired event have been received and processed, but the second session was not.

Session created: bf182e9c-a35e-4ce4-a8c0-f5ddb68372cc
Session created: 407240ff-7d09-41db-858b-fdc44186e6cb
2018-01-11 22:09:59.999 DEBUG 98525 --- [pool-1-thread-1] o.s.s.d.r.RedisSessionExpirationPolicy   : Cleaning up sessions expiring at Thu Jan 11 22:09:00 CST 2018
2018-01-11 22:10:20.204 DEBUG 98525 --- [enerContainer-4] s.s.d.r.RedisOperationsSessionRepository : Publishing SessionDestroyedEvent for session bf182e9c-a35e-4ce4-a8c0-f5ddb68372cc
Session destroyed: bf182e9c-a35e-4ce4-a8c0-f5ddb68372cc
2018-01-11 22:10:59.999 DEBUG 98525 --- [pool-1-thread-1] o.s.s.d.r.RedisSessionExpirationPolicy   : Cleaning up sessions expiring at Thu Jan 11 22:10:00 CST 2018

Please help to re-investigate this issue,
thanks a lot!!

christophstrobl

christophstrobl commented on Jan 11, 2018

@christophstrobl
Member

@horance we've got an open issue in Spring Data Redis to tackle this. Please see: DATAREDIS-534.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @rwinch@HeartSaVioR@christophstrobl@horance@nirav-patel

      Issue actions

        Session listeners are not called back in redis cluster · Issue #478 · spring-projects/spring-session