Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

@TimeLimiter times out slow method but does not cancel running future #905

Closed
georgeharley opened this issue Mar 6, 2020 · 15 comments
Closed
Labels

Comments

@georgeharley
Copy link

Resilience4j version:
1.3.1

Java version:
1.8.0_241

Spring Boot version:
2.2.5.RELEASE

If a method has the @TimeLimiter annotation added that references a time limiter configured with cancelRunningFuture: true then, when a timeout event successfully occurs, the currently running CompleteableFuture does not appear to be cancelled. Instead it just keeps on running.

In the below set up the slow method being called from the protected method just sleeps for 30 seconds and then writes a message to announce it has woken up. This is deliberately being done to try and simulate a very slow/hung method.

application.yml

  resilience4j:
    instances:
      mytimelimiter:
        baseConfig: default
        timeoutDuration: 5000
        cancelRunningFuture: true

    thread-pool-bulkhead:
      instances:
        mybulkhead:
          coreThreadPoolSize: 10
          maxThreadPoolSize: 10
          queueCapacity: 5

SlowService.java

@Service
public class SlowService {

    public String slowMethod() {
        LOGGER.info("slowMethod going to sleep for 30 seconds...");
        try {
            Thread.sleep(30000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        LOGGER.info("...slowMethod awake after 30 seconds");

        . . .

        return "Successful call to slowMethod";
    }
}

MyController.java

@RestController
public class MyController {

    . . .

    @Autowired
    private final SlowService slowService;

    . . .

    @Bulkhead(name = "mybulkhead", type = Type.THREADPOOL)
    @TimeLimiter(name = "mytimelimiter")
    @GetMapping("/slow")
    public CompletableFuture<String> slowHandler() {
        return CompletableFuture.supplyAsync(slowService::slowMethod);
    }

The presence of the @TimeLimiter annotation on the slowHandler() method above ensures that requests to it time out after 5000ms (just as configured for the mytimelimiter instance in the application.yml values). Perfect. However, after another 25 seconds or so the slowMethod() logs that it has woken up and then can presumably carry on going about its work. That seems to indicate that the future was not cancelled when the time out successfully occurred on the calling method.

Is the above working as expected? Perhaps it is the case that resilience4j.timelimiter.instances.{name}.cancelRunningFuture is not expected to have any effect when the time limiter is referenced from a @TimeLimiter annotation? Alternatively, maybe my calling method decorated with the annotation needs to be rewritten in order to enable the cancelling of the CompleteableFuture?

@RobWin
Copy link
Member

RobWin commented Mar 6, 2020

This blog post explains it quite well:
https://www.nurkiewicz.com/2015/03/completablefuture-cant-be-interrupted.html?m=1

This flag only works when you use TimeLimiter to decorate a method which returns a Future. But unfortunately Futures are not yet supported via annotations/aspects, but only via the functional style.

@carlbjor
Copy link

This flag only works when you use TimeLimiter to decorate a method which returns a Future.

Is there a code example for this? This did not work for me, the Thread was not interrupted:

public Callable<List<String>> getAll() {
    TimeLimiterConfig config = TimeLimiterConfig.custom().cancelRunningFuture(true).timeoutDuration(Duration.ofMillis(1000)).build();
    TimeLimiterRegistry timeLimiterRegistry = TimeLimiterRegistry.of(config);
    ExecutorService executor = Executors.newSingleThreadExecutor();
	
    return timeLimiterRegistry.timeLimiter("getAll").decorateFutureSupplier(() -> executor.submit(() -> repository.getAll()));
}

@SidekickJohn
Copy link

@RobWin I'm trying to understand this, but I struggle with the following:

  • I tried the @TimeLimiter-Annotation on a method simply returning an object/String/whatever --> I get an error that the TimeLimiter-Annotation needs a future return type --> fine with me

  • on the other hand you mention, that the @TimeLimiter-Annotation cannot cancel running CompletableFutures and therefore doesn't work as expected

So what return-type do I need for the @TimeLimiter-Annotation to work as expected (cancel running execution and throw TimeoutException)?

We had a running functional implementation, but we are trying to switch away from this to an annotation-based style. Simply because we want to get rid of the gluecode (decorating suppliers...).

@RobWin
Copy link
Member

RobWin commented Dec 2, 2020

Hi @SidekickJohn

You can use a TimeLimiter to decorate a method which returns a Future. A future can be canceled.
But unfortunately Futures are not yet supported via annotations/aspects, but only via the functional style.
We tried to implement it, but supporting Future for TimeLimiter, Retry, Bulkhead, CircuitBreaker via annotations caused us some headaches.

@SidekickJohn
Copy link

@RobWin thanks for clarifying.

Just to make sure I understood you correctly:
Doesnt't that actually mean that there is currently no way to make the @TimeLimiter-Annotation work? No matter what return-type I would choose, the annotation would not work --> correct?

@RobWin
Copy link
Member

RobWin commented Dec 2, 2020

No, TimeLimiter (TimeLimiterAspect) still throws a TimeoutException. See our Spring Boot 2 demo.
But TimeLimiter cannot cancel a CompletableFuture. Otherwise we would have to implement our own CompletableFuture, but that is too cumbersome to maintain.

The TimeLimiter also works for all reactive types from Spring Reactor or RxJava2 and RxJava3

@SidekickJohn
Copy link

Ah alright, now I get it. Thank you.

The TimeoutException is thrown, but the CompletableFuture is still running.

In this case: would you expect a @Retry-Annotation which has the TimeoutException configured as a retryException to take action?

Example:

@TimeLimiter(name="myName")
@Retry(name="myName)
public CompletableFuture<String> myFuncition() {...}

@RobWin
Copy link
Member

RobWin commented Dec 2, 2020

@RobWin RobWin closed this as completed Jan 7, 2021
@FedorSymkin
Copy link

FedorSymkin commented Jan 21, 2021

Hello! Thank you for explanation for this issue. I'd like to clarify my understanding.

I need to use a ThreadPoolBulkhead combined with CircuitBreaker and TimeLimiter, something like example from docs

CompletableFuture<String> future = Decorators.ofSupplier(supplier)
    .withThreadPoolBulkhead(threadPoolBulkhead)
    .withTimeLimiter(timeLimiter, scheduledExecutorService)
    .withCircuitBreaker(circuitBreaker)
    .withFallback(asList(TimeoutException.class, 
                         CallNotPermittedException.class, 
                         BulkheadFullException.class),  
                  throwable -> "Hello from Recovery")
    .get().toCompletableFuture();

So, in this case (because of CompletableFuture) I cannot use interrupting of ThreadPoolBulkhead's thread by timeout. Is there some way to use ThreadPoolBulkhead + TimeLimiter with ability to interrupt a thread, for example using ThreadPoolBulkhead with simple Future instead of CompletableFuture?

As I know I could do it in hystrix. It needed to prevent threads from freezing for a long time when many timeouts occur

@RobWin
Copy link
Member

RobWin commented Jan 21, 2021

Unfortunately the ThreadPoolBulkhead always returns a CompletionStage backed by a CompletableFuture.
You can only combine an ExecutorService and TimeLimniter:

        ExecutorService executor = Executors.newFixedThreadPool(10);
        timeLimiter.executeFutureSupplier(() -> executor.submit(callable)

@FedorSymkin
Copy link

OK, thank you. Are there some plans to support it in thread pool bulhkead in future for compatibility of migrating from hystrix?

@RobWin
Copy link
Member

RobWin commented Jan 22, 2021

No, currently there are no plans to support Future since blocking code should be prevented.
We also do not strive for compatibility to Hystrix.

@FedorSymkin
Copy link

FedorSymkin commented Jan 22, 2021

OK, if I understand correctly, to use TimeLimiter with interruptable thread pool I can use ExecutorService instead of ThreadPoolBulkhead with almost same features (core threads, max threads, keepAliveTime, intercepting cases when there are no free workers), and I have to implement getting thread pool by name using ConcurrentHashMap directly instead of ThreadPoolBulkheadRegistry?

@RobWin
Copy link
Member

RobWin commented Jan 22, 2021

If you like you can try to create a PR and add a <T> Future<T> submit(Callable<T> task) and Supplier<Future<T>> decorateCallable(...) method to the ThreadPoolBulkhead interface.

Shouldn't be too complicated and then we can support ThreadPoolBulkhead and Timelimiter.

@pandekartik007
Copy link

@RobWin
"@Timelimiter times out slow method but does not cancel running future" any additions made for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants