That's quite a dramatic performance penalty. We'll need to do some profiling to find out if the issue is in our TomcatEmbeddedWebappClassLoader or one of the higher Tomcat classloaders that we inherit from. @markt-asf have you seen any similar reports on Tomcat before?

Nothing that comes to mind based on this description.

Thanks for the sample, @cdelgado83. I have reproduced the problem, but only when the application is packaged and run using java -jar. The slowness does not occur when run directly in the IDE.

With trace level logging enabled, I can see the following when running using java -jar:

2019-04-15 15:08:50.953 DEBUG 29022 --- [nio-8080-exec-3] o.a.c.loader.WebappClassLoaderBase       :     findClass(com.example.1)
2019-04-15 15:08:50.953 DEBUG 29022 --- [nio-8080-exec-3] o.a.c.loader.WebappClassLoaderBase       :       findClassInternal(com.example.1)
2019-04-15 15:08:51.423 DEBUG 29022 --- [nio-8080-exec-3] o.a.c.loader.WebappClassLoaderBase       :     --> Returning ClassNotFoundException
2019-04-15 15:08:51.424 DEBUG 29022 --- [nio-8080-exec-3] o.a.c.loader.WebappClassLoaderBase       :     --> Passing on ClassNotFoundException

The processing performed for findClassInternal is taking almost 500ms.

If two requests that load the same non-existent class are made in quick succession the second request is much quicker:

2019-04-15 15:12:19.355 DEBUG 29022 --- [nio-8080-exec-9] o.a.c.loader.WebappClassLoaderBase       :     findClass(com.example.1)
2019-04-15 15:12:19.356 DEBUG 29022 --- [nio-8080-exec-9] o.a.c.loader.WebappClassLoaderBase       :       findClassInternal(com.example.1)
2019-04-15 15:12:19.830 DEBUG 29022 --- [nio-8080-exec-9] o.a.c.loader.WebappClassLoaderBase       :     --> Returning ClassNotFoundException
2019-04-15 15:12:19.830 DEBUG 29022 --- [nio-8080-exec-9] o.a.c.loader.WebappClassLoaderBase       :     --> Passing on ClassNotFoundException
2019-04-15 15:12:23.741 DEBUG 29022 --- [nio-8080-exec-1] o.a.c.loader.WebappClassLoaderBase       :     findClass(com.example.1)
2019-04-15 15:12:23.741 DEBUG 29022 --- [nio-8080-exec-1] o.a.c.loader.WebappClassLoaderBase       :       findClassInternal(com.example.1)
2019-04-15 15:12:23.741 DEBUG 29022 --- [nio-8080-exec-1] o.a.c.loader.WebappClassLoaderBase       :     --> Returning ClassNotFoundException
2019-04-15 15:12:23.741 DEBUG 29022 --- [nio-8080-exec-1] o.a.c.loader.WebappClassLoaderBase       :     --> Passing on ClassNotFoundException

The second request is also much quicker if two requests for different classes are made in quick succession:

2019-04-15 15:13:42.315 DEBUG 29022 --- [nio-8080-exec-5] o.a.c.loader.WebappClassLoaderBase       :     findClass(com.example.1)
2019-04-15 15:13:42.316 DEBUG 29022 --- [nio-8080-exec-5] o.a.c.loader.WebappClassLoaderBase       :       findClassInternal(com.example.1)
2019-04-15 15:13:42.783 DEBUG 29022 --- [nio-8080-exec-5] o.a.c.loader.WebappClassLoaderBase       :     --> Returning ClassNotFoundException
2019-04-15 15:13:42.784 DEBUG 29022 --- [nio-8080-exec-5] o.a.c.loader.WebappClassLoaderBase       :     --> Passing on ClassNotFoundException
2019-04-15 15:13:44.044 DEBUG 29022 --- [nio-8080-exec-7] o.a.c.loader.WebappClassLoaderBase       :     findClass(com.example.2)
2019-04-15 15:13:44.045 DEBUG 29022 --- [nio-8080-exec-7] o.a.c.loader.WebappClassLoaderBase       :       findClassInternal(com.example.2)
2019-04-15 15:13:44.045 DEBUG 29022 --- [nio-8080-exec-7] o.a.c.loader.WebappClassLoaderBase       :     --> Returning ClassNotFoundException
2019-04-15 15:13:44.045 DEBUG 29022 --- [nio-8080-exec-7] o.a.c.loader.WebappClassLoaderBase       :     --> Passing on ClassNotFoundException

The slowness appears to be caused by JarWarResourceSet which is only used when jar files are nested inside a packed war file. This is the case when running a .war file with java -jar and also when deploying to Tomcat configured with unpackWARs=false.

There is a JarWarResourceSet for every jar in WEB-INF/lib. When an attempt is made to load a non-existent class, each resource set produces a map containing every entry in its jar. This map is cached, but only briefly as it's cleared by the background processing thread. There is a method for looking up a single entry, AbstractArchiveResourceSet.getArchiveEntry(String), but its javadoc states that "for performance reasons, getArchiveEntries(boolean) should always be called first" and the implementation in JarWarResourceSet always throws an IllegalStateException. This behaviour seems inefficient to me. Making the same class loading attempt using Boot's LaunchedURLClassLoader is much quicker, and it has the same nested jars on its classpath.

@markt-asf Does the above give you enough to have another look at this?

Running from a packed WAR is always going to be slower because of the two layers of compression. Note that SpringBoot doesn't apply compression to the WAR. This enables access at a comparable speed to an unpacked WAR. Tomcat has to work with the more general case where the WAR may be compressed so Tomcat is slower.
Caching the entries speeds things up but that trades additional memory usage for speed. There is going to be a range of places where users want to draw that line.
I can see a couple if different solutions here:

• Use org.apache.catalina.webresources.ExtractingRoot. This will extract the JARs to the working directory on start allowing faster access.
• SpringBoot provides alternative Resource and ResourceSet implementations that are aware of the WAR not being compressed.
• Add an option to reduce rate of / disable the triggering of the gc() method to Tomcat. This should improve performance at the cost of a larger memory footprint. Note that if gc() is disabled then file locking will occur. This has the greatest impact on Windows.

Thanks, Mark. We're going to investigate implementing our own Resource and ResourceSet. In the meantime, and in general, we strongly recommend using jar rather than war packaging.

Thanks for the analysis !

When do you think this enhancement will be released officialy in Spring Boot ?

It's quite low priority as the problem can be avoided by using .jar packaging rather than .war packaging or, I believe, by switching to another embedded container. As things stand, we have no plans to tackle this in the 2.x timeframe.

Ok, thanks for the quick reply.

Unfortunatelly, I have not mentionned that we are stuck to the war packaging.

In fact, it seems that there is another issue while using the jar packaging: while handling an RPC call, GWT tries to load policy files (*.gwt.rpc) using:

dispatchServlet.getServletContext().getResourceAsStream(
          serializationPolicyFilePath))

Which returns null, and the process it then stopped.

it seems that there is another issue while using the jar packaging

Please open a separate issue for that. If you provide a small sample that reproduces the problem, we'll take a look and see if it can be fixed.

Thanks for the update, I just open a new issue regarding the problem of serialization policy file loading : #17240

Just have to say that this bug is still very much present in Spring-boot v. 2.1.5. We ran into it with old and large app migrated from .war

.jsp kept us on war but I was able to work around this using .jar with the help of this --> https://github.com/hengyunabc/spring-boot-fat-jar-jsp-sample

Ps for us performance was a DRASTIC drop since we perform at-runtime classloading for extensive plugin system

How would one configure Tomcat to use the mentioned ExtractingRoot in Spring Boot?

@Frettman You should be able to use a TomcatServletWebServerFactory sub-class that overrides postProcessContext(Context) and calls setResources(WebResourceRoot) with an instance of ExtractingRoot:

@Bean
TomcatServletWebServerFactory tomcatFactory() {
	return new TomcatServletWebServerFactory() {

		@Override
		protected void postProcessContext(Context context) {
			context.setResources(new ExtractingRoot());
		}

	};
}

Thank you, @wilkinsona, this has worked perfectly.

There is one caveat to using the ExtractingRoot that one should be aware of: By default, every start of the application creates a new folder like tomcat.1150352738695762234.443 in the temp directory. This happens with or without the ExtractingRoot. But: With the ExtractingRoot the Jars will be extracted to a application-jars subdirectory of this temporary tomcat folder. And even though the Tomcat documentation states that this folder is removed when the application stops, this does not happen; at least not for me. So every start of your application will use up more space (unless your application is e.g. in a Docker container). To avoid that set the application property server.tomcat.basedir to any static folder.
The Jars will be extracted on every start, so missing files will be recreated and others overwritten. However, obsolete files are not removed. So if Tomcat simply loads all Jars in that directory (not sure how to test that yet) that will most likely turn into a problem at some point. So it's probably a good idea to remove the application-jars folder e.g. in a shell script with every start.

The slowness appears to be caused by JarWarResourceSet which is only used when jar files are nested inside a packed war file. This is the case when running a .war file with java -jar and also when deploying to Tomcat configured with unpackWARs=false.

There is a JarWarResourceSet for every jar in WEB-INF/lib. When an attempt is made to load a non-existent class, each resource set produces a map containing every entry in its jar. This map is cached, but only briefly as it's cleared by the background processing thread. There is a method for looking up a single entry, AbstractArchiveResourceSet.getArchiveEntry(String), but its javadoc states that "for performance reasons, getArchiveEntries(boolean) should always be called first" and the implementation in JarWarResourceSet always throws an IllegalStateException. This behaviour seems inefficient to me. Making the same class loading attempt using Boot's LaunchedURLClassLoader is much quicker, and it has the same nested jars on its classpath.

@markt-asf Does the above give you enough to have another look at this?

when running with JarLauncher the classical class loader loades all the classes in BOOT-INF, when running with WarLauncher the classical class loader loades all the classes in WEB-INF, so why is TomcatEmbeddedWebAppClassLoader needed?

I believe that this has been addressed in Spring Boot 3.2 by #37452.

I have conducted some tests with Spring Boot 3.2.0 and also with 3.3.4 and I can pretty much assure it's not fixed : I still see slow performances when launching the War with "java -jar" outside the IDE. And this goes away when using this code (based on the github example from @cdelgado83):

package com.misc.tomcatslowness;

import org.apache.catalina.webresources.ExtractingRoot;
import org.apache.catalina.Context;
import org.springframework.boot.web.embedded.tomcat.TomcatContextCustomizer;
import org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory;
import org.springframework.boot.web.server.WebServerFactoryCustomizer;
import org.springframework.stereotype.Component;

@Component
public class MyTomcatWebServerCustomizer
        implements WebServerFactoryCustomizer<TomcatServletWebServerFactory> {

    @Override
    public void customize(TomcatServletWebServerFactory factory) {
        factory.addContextCustomizers((Context context) -> {
            ExtractingRoot extractingRoot = new ExtractingRoot();
            extractingRoot.setCachingAllowed(true); // Enable caching for better performance
            context.setResources(extractingRoot);
        });
    }

}

@jplandrain if you'd like us to investigate, please provide a minimal sample that reproduces the problem, some data that shows the slow performance in absolute terms, and some details of the hardware, JVM, etc that you have tested on.