Session listeners are not called back in redis cluster #478

sanketmeghani · 2016-04-11T14:44:02Z

@rwinch We are using spring-session 1.2.0.RC1 and spring-data-redis 1.7.0.RC1. Somehow we are not getting the listener call back if we use redis cluster. It works perfect if we use a stand-alone redis server. I have uploaded a sample springSessionContext.xml.

On one of the nodes in redis cluster, we see "SUBSCRIBE" "__keyevent@:expired" "spring:session:event:created:" "__keyevent@*:del" command. So it does subscribe on one node. Should it subscribe to all nodes in the cluster? Or this is expected behavior?

Sample application code is available here.

rwinch · 2016-04-11T15:03:48Z

Thanks for the report!

cc @christophstrobl

leehanwool · 2016-04-12T08:50:48Z

I experienced same situation at tomcat 8 with spring-web-application(spring-data-redis 1.7.1.RELEASE) and redis-cluster(3.0.7).
When I stop tomcat, I received WARNING message like below, then tomcat going to hanging status, so I have to kill tomcat myself.(sometimes stop tomcat oneself after few minutes.)
would you please tell me about release plan with this issue?

[WARN] [RedisMessageListenerContainer$SubscriptionTask:864] : Unable to close connection. Subscription task still running
 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [] appears to have started a thread named [redisMessageListenerContainer-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.net.SocketInputStream.socketRead0(Native Method)
 java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
 java.net.SocketInputStream.read(SocketInputStream.java:170)
 java.net.SocketInputStream.read(SocketInputStream.java:141)
 java.net.SocketInputStream.read(SocketInputStream.java:127)
 redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:195)
 redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40)
 redis.clients.jedis.Protocol.process(Protocol.java:141)
 redis.clients.jedis.Protocol.read(Protocol.java:205)
 redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)
 redis.clients.jedis.Connection.getRawObjectMultiBulkReply(Connection.java:242)
 redis.clients.jedis.BinaryJedisPubSub.process(BinaryJedisPubSub.java:87)
 redis.clients.jedis.BinaryJedisPubSub.proceed(BinaryJedisPubSub.java:82)
 redis.clients.jedis.BinaryJedis.subscribe(BinaryJedis.java:2980)
 redis.clients.jedis.BinaryJedisCluster$150.execute(BinaryJedisCluster.java:1607)
 redis.clients.jedis.BinaryJedisCluster$150.execute(BinaryJedisCluster.java:1604)
 redis.clients.jedis.JedisClusterCommand.runWithAnyNode(JedisClusterCommand.java:87)
 redis.clients.jedis.BinaryJedisCluster.psubscribe(BinaryJedisCluster.java:1610)
 org.springframework.data.redis.connection.jedis.JedisClusterConnection.pSubscribe(JedisClusterConnection.java:2508)
 org.springframework.data.redis.listener.RedisMessageListenerContainer$SubscriptionTask.eventuallyPerformSubscription(RedisMessageListenerContainer.java:779)
 org.springframework.data.redis.listener.RedisMessageListenerContainer$SubscriptionTask.run(RedisMessageListenerContainer.java:746)
 java.lang.Thread.run(Thread.java:745)

rwinch · 2016-04-12T13:09:47Z

@leehanwool I haven't had a chance to look into this issue yet. Note that we also have a ticket for official Redis Cluster support that we need to get to. See #422

As for your StackTrace, could you please create a separate ticket for the unclosed thread?

christophstrobl · 2016-04-12T17:27:43Z

@rwinch @leehanwool created DATAREDIS-493 to check that.

leehanwool · 2016-04-13T05:34:06Z

@rwinch
Thank for reply.
I created new #482 ticket for Stacktrace.

nirav-patel · 2016-04-15T07:30:31Z

@rwinch @leehanwool @christophstrobl
I have done spring-session configuration with Redis by using Jedis Client (2.8.1 version) as mentioned in issue description by @sanketmeghani above. I found that internally when cluster is created, each node gets subscribed to patterns for event notifications (basically those are session creation, session expiration and session deletion events) via method

void redis.clients.jedis.BinaryJedisCluster.psubscribe(BinaryJedisPubSub jedisPubSub, byte[]... patterns)

Actual implementation of the method in BinaryJedisCluster class is as shown below:
Note in execute method, Jedis Connection is subscribed to patterns using subscribe method [i.e. connection.subscribe(jedisPubSub, patterns);].

@Override
  public void psubscribe(final BinaryJedisPubSub jedisPubSub, final byte[]... patterns) {
    new JedisClusterCommand<Integer>(connectionHandler, maxRedirections) {
      @Override
      public Integer execute(Jedis connection) {
        connection.subscribe(jedisPubSub, patterns);
        return 0;
      }
    }.runWithAnyNode();
  }

Here If I override this method with custom implementation and subscribe each Jedis Connection in execute method with psubscribe method instead of subscribe method [i.e., connection.psubscribe(jedisPubSub, patterns)], then I`m able to retrieve session notifications for creation, expiration and deletion activities performed by Redis Cluster.

@Override
  public void psubscribe(final BinaryJedisPubSub jedisPubSub, final byte[]... patterns) {
    new JedisClusterCommand<Integer>(connectionHandler, maxRedirections) {
      @Override
      public Integer execute(Jedis connection) {
        connection.psubscribe(jedisPubSub, patterns);
        return 0;
      }
    }.runWithAnyNode();
  }

In case of single Redis node implementation, Jedis Connection is subscribed to patterns using psubscribe method and not the subscribe method. So I applied the same logic for Redis Cluster use-case too and things are working fine.

I don`t know whether it is a Jedis issue or current implementation is correct for some purpose.
Is this a correct way to fix above issue and get session listeners call backs from Redis Cluster?

@rwinch @leehanwool @christophstrobl Please provide your feedback for this solution.

rwinch · 2016-04-15T13:43:06Z

@nirav-patel Thanks for the detailed writeup!

I think @christophstrobl will be able to provide better feedback on your solution

leehanwool · 2016-04-18T08:25:36Z

@nirav-patel , Thank you for advise!
I tried to use your solution in my case #482
It works perfectly. Tomcat no more going to hanging status, So I don't need kill tomcat anymore.
Thank you!!!

christophstrobl · 2016-04-18T08:31:05Z

thanks @nirav-patel for detailed information.

HeartSaVioR · 2016-04-18T09:36:12Z

Thanks all for finding missed spot for Jedis.

rwinch · 2016-05-02T20:19:03Z

Thanks for the reports and helping sort this out! I'm closing this as it appears the issue has been fixed in Jedis

horance · 2018-01-11T14:20:21Z

@rwinch Sorry about commenting on closed issue. I ran into the issue that @sanketmeghani mentioned.

I've done a sample project SpringSessionTest with spring-session 1.3.0.RELEASE and spring-data-redis 1.8.3.RELEASE. I've found redis message listener container only subscribed to one node. According to redis/redis#2541, the keyspace notifications are only sent locally. As a result, the session-expired event will not be processed by listener, unless the key was created on the subscribed node.

Following is my test result with my test application, I send two http request to my test application with curl:

HoranceMBP:SpringSessionTest horance$ curl http://localhost:8080/hello 
The session ID is :bf182e9c-a35e-4ce4-a8c0-f5ddb68372cc, session expires key stored on node: 127.0.0.1:7001, psub pattern count is 3, will receive expired event.
HoranceMBP:SpringSessionTest horance$ curl http://localhost:8080/hello 
The session ID is :407240ff-7d09-41db-858b-fdc44186e6cb, session expires key stored on node: 127.0.0.1:7002, psub pattern count is 0, will not receive expired event.

And after couple minutes, the log shows the first session expired event have been received and processed, but the second session was not.

Session created: bf182e9c-a35e-4ce4-a8c0-f5ddb68372cc
Session created: 407240ff-7d09-41db-858b-fdc44186e6cb
2018-01-11 22:09:59.999 DEBUG 98525 --- [pool-1-thread-1] o.s.s.d.r.RedisSessionExpirationPolicy   : Cleaning up sessions expiring at Thu Jan 11 22:09:00 CST 2018
2018-01-11 22:10:20.204 DEBUG 98525 --- [enerContainer-4] s.s.d.r.RedisOperationsSessionRepository : Publishing SessionDestroyedEvent for session bf182e9c-a35e-4ce4-a8c0-f5ddb68372cc
Session destroyed: bf182e9c-a35e-4ce4-a8c0-f5ddb68372cc
2018-01-11 22:10:59.999 DEBUG 98525 --- [pool-1-thread-1] o.s.s.d.r.RedisSessionExpirationPolicy   : Cleaning up sessions expiring at Thu Jan 11 22:10:00 CST 2018

Please help to re-investigate this issue,
thanks a lot!!

christophstrobl · 2018-01-11T14:29:08Z

@horance we've got an open issue in Spring Data Redis to tackle this. Please see: DATAREDIS-534.

modouxiansheng · 2019-04-28T02:58:18Z

you can configure many containers to monitor,For example

public void onApplicationEvent(ContextRefreshedEvent contextRefreshedEvent) {
        RedisClusterConnection redisClusterConnection = redisConnectionFactory.getClusterConnection();
        if (redisClusterConnection != null) {
            Iterable<RedisClusterNode> nodes = redisClusterConnection.clusterGetNodes();
            for (RedisClusterNode node : nodes) {
                if (node.isMaster()) {
                    String containerBeanName = "messageContainer" + node.hashCode();
                    if (beanFactory.containsBean(containerBeanName)) {
                        return;
                    }
                    JedisConnectionFactory factory = new JedisConnectionFactory(
                            new JedisShardInfo(node.getHost(), node.getPort()));
                    BeanDefinitionBuilder containerBeanDefinitionBuilder = BeanDefinitionBuilder
                            .genericBeanDefinition(RedisMessageListenerContainer.class);
                    containerBeanDefinitionBuilder.addPropertyValue("connectionFactory", factory);
                    containerBeanDefinitionBuilder.setScope(BeanDefinition.SCOPE_SINGLETON);
                    containerBeanDefinitionBuilder.setLazyInit(false);
                    beanFactory.registerBeanDefinition(containerBeanName,
                            containerBeanDefinitionBuilder.getRawBeanDefinition());

                    RedisMessageListenerContainer container = beanFactory
                            .getBean(containerBeanName, RedisMessageListenerContainer.class);
                    String listenerBeanName = "messageListener" + node.hashCode();
                    if (beanFactory.containsBean(listenerBeanName)) {
                        return;
                    }
                    container.addMessageListener(messageListener,new PatternTopic("__keyevent@0__:expire"));
                    container.start();
                }
            }
        }

sanketmeghani mentioned this issue Apr 11, 2016

Add HttpSessionListener Support #4

Closed

leehanwool mentioned this issue Apr 18, 2016

When stop tomcat, Unable to close connection because unclosed thread. #482

Closed

christophstrobl mentioned this issue Apr 18, 2016

BinaryJedisCluster.psubscribe calls subscribe instead of psubscribe. redis/jedis#1258

Closed

rwinch self-assigned this May 2, 2016

rwinch closed this as completed May 2, 2016

rwinch added the status: invalid label May 2, 2016

spring-projects-issues mentioned this issue Dec 30, 2020

RedisMessageListener does not receive events correctly in cluster [DATAREDIS-493] spring-projects/spring-data-redis#1069

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Session listeners are not called back in redis cluster #478

Session listeners are not called back in redis cluster #478

sanketmeghani commented Apr 11, 2016

rwinch commented Apr 11, 2016

leehanwool commented Apr 12, 2016

rwinch commented Apr 12, 2016

christophstrobl commented Apr 12, 2016

leehanwool commented Apr 13, 2016

nirav-patel commented Apr 15, 2016 •

edited

rwinch commented Apr 15, 2016

leehanwool commented Apr 18, 2016

christophstrobl commented Apr 18, 2016 •

edited

HeartSaVioR commented Apr 18, 2016

rwinch commented May 2, 2016

horance commented Jan 11, 2018

christophstrobl commented Jan 11, 2018

modouxiansheng commented Apr 28, 2019

Session listeners are not called back in redis cluster #478

Session listeners are not called back in redis cluster #478

Comments

sanketmeghani commented Apr 11, 2016

rwinch commented Apr 11, 2016

leehanwool commented Apr 12, 2016

rwinch commented Apr 12, 2016

christophstrobl commented Apr 12, 2016

leehanwool commented Apr 13, 2016

nirav-patel commented Apr 15, 2016 • edited

rwinch commented Apr 15, 2016

leehanwool commented Apr 18, 2016

christophstrobl commented Apr 18, 2016 • edited

HeartSaVioR commented Apr 18, 2016

rwinch commented May 2, 2016

horance commented Jan 11, 2018

christophstrobl commented Jan 11, 2018

modouxiansheng commented Apr 28, 2019

nirav-patel commented Apr 15, 2016 •

edited

christophstrobl commented Apr 18, 2016 •

edited