Skip to content

OutOfMemoryError: Java heap space in GZIPContentDecoder #3373

New issue
New issue
Closed
Closed
OutOfMemoryError: Java heap space in GZIPContentDecoder#3373
@ltp217

Description

@ltp217
ltp217
opened on Feb 20, 2019 · edited by sbordet

I'am using jetty-9.4.14 to send request, when I have some big data to response, there is an OutOfMemory exception,I have see the issue about GZIPContentDecoder ,but I'am not found how I could solve the problem,could you help me.The stack is below:

java.lang.OutOfMemoryError: Java heap space
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[?:1.8.0_191]
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[?:1.8.0_191]
	at org.eclipse.jetty.util.BufferUtil.allocate(BufferUtil.java:116) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.io.ByteBufferPool.newByteBuffer(ByteBufferPool.java:61) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.io.MappedByteBufferPool.acquire(MappedByteBufferPool.java:65) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.http.GZIPContentDecoder.acquire(GZIPContentDecoder.java:408) ~[jetty-http-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.http.GZIPContentDecoder.decodedChunk(GZIPContentDecoder.java:108) ~[jetty-http-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.http.GZIPContentDecoder.decodeChunks(GZIPContentDecoder.java:189) ~[jetty-http-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.http.GZIPContentDecoder.decode(GZIPContentDecoder.java:71) ~[jetty-http-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:347) ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:283) ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1787) ~[jetty-http-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1517) ~[jetty-http-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:172) ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:135) ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:73) ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:133) ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:155) ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]

Activity

olamy

olamy commented on Feb 20, 2019

@olamy
olamy
on Feb 20, 2019
Member

what is the possible size of this some big data to response?
you are probably too close to your maximum heap size so the java.lang.OutOfMemoryError: Java heap space

sbordet

sbordet commented on Feb 20, 2019

@sbordet
sbordet
on Feb 20, 2019
Contributor

I concur with @olamy: from the information you gave you probably have a too small heap and you are unzipping a zip bomb or similar.

ltp217

ltp217 commented on Feb 23, 2019

@ltp217
ltp217
on Feb 23, 2019
Author

@olamy @sbordet first, I have checked the heap and the reponse stream, the heap is 256M, the response is about 1M, also I exculed the zip bomb reason.I have found the same problem in my three services .After I checked the environment, once I do the request, I have get the dump when the OOM happend, I found that there are 478 entries of java.util.concurrent.ConcurrenthashMap$Node, and in these Node ,it's all the response byte, the bytes are also the same and not complete, I catched these infomation by Memory Aanlyzer, one of the 478 entries is below. In the last line ,the byte is the response data but not complete . In the 478 entries, the entries are all the same , and the byte value is same too. I still think maybe when jetty handle the chunk, there maybe some problem. But I still can't get the really scenes。I guess is there any situation the pool may not clean the chunk data before, so it will accumulate ,for example: the chunk date is about 2k every time, first 2k, then 4k, then 6k, but the pool ,there maybe 2+4+6+...at last ,it will be OOM. BTW, I also used python to do the request, and I can get the response data sucessfully.

one of the 478 entries:

ClassName  | Shallow Heap | Retained Heap | Percentage
-------------------------------------------------------------------------------------------------------------------
java.util.concurrent.ConcurrentHashMap$Node[1024]@0xf22f1688 |   4,112 |   234,673,336 |     91.67%|
- java.util.concurrent.ConcurrentHashMap$Node@0xf43bb438 |          32 |           979,176 |      0.38%|
  |-org.eclipse.jetty.io.ByteBufferPool$Bucket@0xf43bb3d8   |           32 |       979,128 |      0.38%|
  |  |-java.util.concurrent.ConcurrentLinkedDeque@0xf43bb3f8  |           24 |       979,080 |      0.38%|
  |  |  '- java.util.concurrent.ConcurrentLinkedDeque$Node@0xf43bb410  |           24 |       979,056 |      0.38%|  
  |  |     '- java.util.concurrent.ConcurrentLinkedDeque$Node@0xf43bb458  |           24 |       979,032 |      0.38%|  
  |  |        '- java.nio.HeapByteBuffer@0xf41e5aa0  |           48 |       979,008 |      0.38%|  
  |  |           '- byte[978944] @ 0xf534b070  _**the response data...**_}]}.2019-02-20 11:50:21,767 INFO  ...
ltp217

ltp217 commented on Feb 23, 2019

@ltp217
ltp217
on Feb 23, 2019 · edited by sbordet
Author

@olamy @sbordet I have get the souce code and get some log, In Thread HttpClient@2424a36f-68 ,I see the size is larger and larger, And the bucket is about 482 ,The log where I set is below:

In GZIPContentDecoder.java

protected boolean decodedChunk(ByteBuffer chunk)
    {
        if (_inflated==null)
        {
            _inflated=chunk;
        }
        else
        {
            int size = _inflated.remaining() + chunk.remaining();
            LOG.warn("["+Thread.currentThread().getName()+"]-------TEST------infremain:"+_inflated.remaining()
            	+",chunkremain:"+chunk.remaining()+",size:"+size);
            if (size<=_inflated.capacity())
            {
                BufferUtil.append(_inflated,chunk);
                release(chunk);
            }
            else
            {
                ByteBuffer bigger=acquire(size);
                int pos=BufferUtil.flipToFill(bigger);
                BufferUtil.put(_inflated,bigger);
                BufferUtil.put(chunk,bigger);
                BufferUtil.flipToFlush(bigger,pos);
                release(_inflated);
                release(chunk);
                _inflated = bigger;
            }
        }
        return false;
    }

In MappedByteBufferPool.java

public ByteBuffer acquire(int size, boolean direct)
    {
        int b = bucketFor(size);
        ConcurrentMap<Integer, Bucket> buffers = bucketsFor(direct);
        LOG.warn("["+Thread.currentThread().getName()+"]--------test-------size:"+size
        		+",b:"+b+",factor:"+_factor);
        Bucket bucket = buffers.get(b);
        if (bucket==null)
            return newByteBuffer(b*_factor, direct);
        return bucket.acquire(direct);
    }

The log:

...

2019-02-23 16:49:30,260 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:952320,b:465,factor:2048
2019-02-23 16:49:30,260 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,260 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:952320,chunkremain:2048,size:954368
2019-02-23 16:49:30,260 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:954368,b:466,factor:2048
2019-02-23 16:49:30,260 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,261 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:954368,chunkremain:2048,size:956416
2019-02-23 16:49:30,261 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:956416,b:467,factor:2048
2019-02-23 16:49:30,261 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,261 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:956416,chunkremain:2048,size:958464
2019-02-23 16:49:30,261 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:958464,b:468,factor:2048
2019-02-23 16:49:30,262 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,262 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:958464,chunkremain:2048,size:960512
2019-02-23 16:49:30,262 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:960512,b:469,factor:2048
2019-02-23 16:49:30,262 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,263 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:960512,chunkremain:2048,size:962560
2019-02-23 16:49:30,263 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:962560,b:470,factor:2048
2019-02-23 16:49:30,263 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,263 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:962560,chunkremain:2048,size:964608
2019-02-23 16:49:30,263 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:964608,b:471,factor:2048
2019-02-23 16:49:30,264 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,264 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:964608,chunkremain:2048,size:966656
2019-02-23 16:49:30,264 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:966656,b:472,factor:2048
2019-02-23 16:49:30,264 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,264 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:966656,chunkremain:2048,size:968704
2019-02-23 16:49:30,265 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:968704,b:473,factor:2048
2019-02-23 16:49:30,265 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,265 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:968704,chunkremain:2048,size:970752
2019-02-23 16:49:30,265 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:970752,b:474,factor:2048
2019-02-23 16:49:30,266 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,266 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:970752,chunkremain:2048,size:972800
2019-02-23 16:49:30,266 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:972800,b:475,factor:2048
2019-02-23 16:49:30,267 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,267 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:972800,chunkremain:2048,size:974848
2019-02-23 16:49:30,267 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:974848,b:476,factor:2048
2019-02-23 16:49:30,268 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,268 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:974848,chunkremain:2048,size:976896
2019-02-23 16:49:30,268 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:976896,b:477,factor:2048
2019-02-23 16:49:30,268 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,269 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:976896,chunkremain:2048,size:978944
2019-02-23 16:49:30,269 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:978944,b:478,factor:2048
2019-02-23 16:49:30,269 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,269 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:978944,chunkremain:2048,size:980992
2019-02-23 16:49:30,270 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:980992,b:479,factor:2048
2019-02-23 16:49:30,628 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,629 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:980992,chunkremain:2048,size:983040
2019-02-23 16:49:30,629 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:983040,b:480,factor:2048
2019-02-23 16:49:30,630 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,630 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:983040,chunkremain:2048,size:985088
2019-02-23 16:49:30,630 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:985088,b:481,factor:2048
2019-02-23 16:49:30,630 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:2048,b:1,factor:2048
2019-02-23 16:49:30,630 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 104] [HttpClient@2424a36f-68]-------TEST------infremain:985088,chunkremain:2048,size:987136
2019-02-23 16:49:30,630 WARN  [HttpClient@2424a36f-68][ROOT][org.eclipse.jetty.io.MappedByteBufferPool 65] [HttpClient@2424a36f-68]--------test-------size:987136,b:482,factor:2048
2019-02-23 16:53:06,929 WARN  [http-nio-auto-1-exec-1-72a9f9035ca1b9bd][ROOT][org.eclipse.jetty.util.ssl.SslContextFactory 259] No Client EndPointIdentificationAlgorithm configured for SslContextFactory@a338e64[provider=null,keyStore=null,trustStore=null]
...
sbordet

sbordet commented on Feb 25, 2019

@sbordet
sbordet
on Feb 25, 2019
Contributor

What is happening is that you are generating content that is always greater than the previous content.
From your logs, you can see that the b value is always increasing, and b is a key in a Map to a Bucket, which has a queue of ByteBuffers.
Because of this, the MappedByteBufferPool holds one Bucket with one ByteBuffer for each different size, and eventually your small heap is exhausted.

Is this a test case of some kind, or a real world scenario?

You can use ArrayByteBufferPool that allows you to pool ByteBuffer up to a certain size only, and that will constrain the overall memory used by the pool.

This issue is a duplicate of #1861.

sbordet
closed this as completedon Feb 25, 2019
ltp217

ltp217 commented on Feb 25, 2019

@ltp217
ltp217
on Feb 25, 2019 · edited by joakime
Author

@sbordet The log above is from a real world scenario.And I have sth new ,I did some log in GZIPContentDecoder.java ,and the generating content is not greater than the previous content, I found that the every time the response buffer size is always 2048 .

in GZIPContentDecoder.decodeChunks(),in case DATA:

case DATA:
 {
           while (true)
            {
                    if (buffer==null)
                    buffer = acquire(_bufferSize);
                    try
                    {
                            byte[] bytes = buffer.array();
                            int length = _inflater.inflate(bytes,buffer.arrayOffset(),buffer.capacity());
                             buffer.limit(length);                           
                            LOG.warn("["+Thread.currentThread().getName()+"]-------TEST0------bufferlength:"+new String(bytes).length());
                    }

and in GZIPContentDecoder.decodedChunk():

protected boolean decodedChunk(ByteBuffer chunk)
    {
        if (_inflated==null)
        {
            _inflated=chunk;
            LOG.warn("["+Thread.currentThread().getName()+"]-------TEST1------_inflated is null");
        }
        else
        {
            int size = _inflated.remaining() + chunk.remaining();
            LOG.warn("["+Thread.currentThread().getName()+"]-------TEST2------chunkremain:"+chunk.remaining()
            	+",infremain:"+_inflated.remaining()+",infcap:"+_inflated.capacity()+",size:"+size+",TOF:"+(size<=_inflated.capacity()));
            if (size<=_inflated.capacity())
            {
                BufferUtil.append(_inflated,chunk);
                release(chunk);
            }
            else
            {
                ByteBuffer bigger=acquire(size);
                int pos=BufferUtil.flipToFill(bigger);
                BufferUtil.put(_inflated,bigger);
                BufferUtil.put(chunk,bigger);
                BufferUtil.flipToFlush(bigger,pos);
                release(_inflated);
                release(chunk);
                _inflated = bigger;
            }
            LOG.warn("["+Thread.currentThread().getName()+"]-------TEST3------infremain:"+_inflated.remaining()
            +",infcap:"+_inflated.capacity()+",inflimit:"+_inflated.limit()+",infpos:"+_inflated.position());
        }
        return false;
    }

I get the log below:

2019-02-25 15:04:20,727 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 100] [HttpClient@4b733b9e-77]-------TEST1------_inflated is null
2019-02-25 15:04:20,727 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,727 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:2048,infcap:2048,size:4096,TOF:false
2019-02-25 15:04:20,727 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:4096,infcap:4096,inflimit:4096,infpos:0
2019-02-25 15:04:20,728 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,728 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:4096,infcap:4096,size:6144,TOF:false
2019-02-25 15:04:20,728 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:6144,infcap:6144,inflimit:6144,infpos:0
2019-02-25 15:04:20,729 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,729 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:6144,infcap:6144,size:8192,TOF:false
2019-02-25 15:04:20,729 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:8192,infcap:8192,inflimit:8192,infpos:0
2019-02-25 15:04:20,729 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,730 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:8192,infcap:8192,size:10240,TOF:false
2019-02-25 15:04:20,730 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:10240,infcap:10240,inflimit:10240,infpos:0
2019-02-25 15:04:20,730 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,731 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:10240,infcap:10240,size:12288,TOF:false
2019-02-25 15:04:20,731 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:12288,infcap:12288,inflimit:12288,infpos:0
2019-02-25 15:04:20,731 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,731 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:12288,infcap:12288,size:14336,TOF:false
2019-02-25 15:04:20,732 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:14336,infcap:14336,inflimit:14336,infpos:0
2019-02-25 15:04:20,732 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,732 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:14336,infcap:14336,size:16384,TOF:false
2019-02-25 15:04:20,733 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:16384,infcap:16384,inflimit:16384,infpos:0
2019-02-25 15:04:20,733 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,733 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:16384,infcap:16384,size:18432,TOF:false
2019-02-25 15:04:20,734 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:18432,infcap:18432,inflimit:18432,infpos:0
2019-02-25 15:04:20,734 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,734 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:18432,infcap:18432,size:20480,TOF:false
2019-02-25 15:04:20,735 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:20480,infcap:20480,inflimit:20480,infpos:0
2019-02-25 15:04:20,735 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,735 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:20480,infcap:20480,size:22528,TOF:false
2019-02-25 15:04:20,736 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:22528,infcap:22528,inflimit:22528,infpos:0
2019-02-25 15:04:20,736 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,736 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:22528,infcap:22528,size:24576,TOF:false
2019-02-25 15:04:20,737 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:24576,infcap:24576,inflimit:24576,infpos:0
2019-02-25 15:04:20,737 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,737 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:24576,infcap:24576,size:26624,TOF:false
2019-02-25 15:04:20,737 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:26624,infcap:26624,inflimit:26624,infpos:0
2019-02-25 15:04:20,738 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,738 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:26624,infcap:26624,size:28672,TOF:false
2019-02-25 15:04:20,738 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:28672,infcap:28672,inflimit:28672,infpos:0
2019-02-25 15:04:20,739 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 192] [HttpClient@4b733b9e-77]-------TEST0------bufferlength:2048
2019-02-25 15:04:20,739 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:28672,infcap:28672,size:30720,TOF:false
2019-02-25 15:04:20,739 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:30720,infcap:30720,inflimit:30720,infpos:0

...........

2019-02-25 15:04:21,851 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 105] [HttpClient@4b733b9e-77]-------TEST2------chunkremain:2048,infremain:978944,infcap:978944,size:980992,TOF:false
2019-02-25 15:04:21,852 WARN  [HttpClient@4b733b9e-77][ROOT][org.eclipse.jetty.http.GZIPContentDecoder 123] [HttpClient@4b733b9e-77]-------TEST3------infremain:980992,infcap:980992,inflimit:980992,infpos:0

In this log,in branch TEST0, the buffer size is 2048,it's not like that every time it increase, I found that In branch TEST3, it has run release(_inflated);, but we see the _inflated.remaining(),_inflated.capacity(),_inflated.limit() is increase 2048 every time, but only the position is 0, is this a nomal phenomenon? In my option, when the ByteBuffer is released, it's limit should be 0 too. And in BufferUtil.java, the position and limit is truly set to 0, Only maybe there is some problem in ByteBufferPool.Bucket.release():

public void release(ByteBuffer buffer)
        {
            BufferUtil.clear(buffer);
            if (_space == null)
                queueOffer(buffer);
            else if (_space.decrementAndGet() >= 0)
                queueOffer(buffer);
            else
                _space.incrementAndGet();
        }

In this method ,it not really release the _inflated buffer , If the _inflated.remaining() is not 0 but the size before, the next time ,
in GZIPContentDecoder.decodedChunk ():
int size = _inflated.remaining() + chunk.remaining();
the size will plus chunk.remaining(),in my environment ,it's 2048. And the log is matched to my thread dump.

for example: If the response is 10240 bytes,every time the chunk is 2048, it will need 2048+4096+6144+8192+10240 bytes to do this .And if the response is bigger, OOM may happend.

Look forward to your reply.Thank you very much.

sbordet
reopened this on Feb 25, 2019
sbordet

sbordet commented on Feb 25, 2019

@sbordet
sbordet
on Feb 25, 2019
Contributor

Okay so this is a combination of #1861 and GZipContentDecoder that by default is accumulating all decoded chunks into a single ByteBuffer.

The previous implementation (in jetty-client before it was moved to jetty-http) was using a byte[] so there was no interaction with the buffer pool.

@gregw I'm inclined to not use a pooled buffer for _inflated; your thoughts?

gregw

gregw commented on Feb 25, 2019

@gregw
gregw
on Feb 25, 2019
Contributor

@sbordet I tend to agree that a simple pool of buffers for _inflated is not going to work very well if the size of the inflated content varies greatly.
However, I'm not that keen to give up on pooling all together, nor do I like the multiple copies we need to do as we discover the inflated size of the content.

So my proposal is that we have a chunk size (that should probably be a lot bigger than 2048... let's say 32768). We should inflate into a chunk buffer.... if that is the complete content, then great we just use that.
If it is not complete, then we simply get another chunk buffer from the pool and we build a chain of chunk buffers..... if possible we can try to use this chain of buffers... but more likely once we need the content we will need to allocated a new non-pool buffer/array of exact size and copy all the chunks into it. This means that we will have a buffer pool that will be filled with buffers only of chunk size. When content is larger than a chunk we will allocated one and only one buffer of exactly the right size. We do need double the memory of the content size.... but I think any approach that results in a single buffer approaches that requirement.

sbordet

sbordet commented on Feb 26, 2019

@sbordet
sbordet
on Feb 26, 2019 · edited by sbordet
Contributor

I wrote a test case, and for 1 MiB of content, which is then gzipped by the server, the client ByteBufferPool retains 267 MiB 😮

sbordet

sbordet commented on Feb 26, 2019

@sbordet
sbordet
on Feb 26, 2019
Contributor

@gregw after the fixes, the client ByteBufferPool retains only 8 KiB and I made the client using the gzip decoder asynchronously, so there is no need for aggregation.

This means now that both the client and the server interceptor subclass GZIPContentDecoder to be fully async, so the aggregation code is never triggered.
I have implemented your suggestion of a chain of buffers for the aggregation - it's never triggered now, but if it will be at least we won't blow up the buffer pool.

The PR with these changes depends on #3391.

ltp217

ltp217 commented on Feb 26, 2019

@ltp217
ltp217
on Feb 26, 2019
Author

@sbordet yeah, it's similar to my service, in one of my service, the max heap size is 256M, 2MB of content has occured OOM. We now change the -Xmx=1024 to avoid this now. For so, there may need another way to decrease this.

sbordet
added a commit that references this issue on Feb 26, 2019

Issue #3373 - OutOfMemoryError: Java heap space in GZIPContentDecoder.

34938d6
sbordet

sbordet commented on Feb 26, 2019

@sbordet
sbordet
on Feb 26, 2019
Contributor

@ltp217 can you please try branch jetty-9.4.x-3373-gzip_oom and see if it fixes your problem?

sbordet
mentioned this on Feb 26, 2019
ltp217

ltp217 commented on Feb 27, 2019

@ltp217
ltp217
on Feb 27, 2019
Author

@sbordet I have tried the branch jetty-9.4.x-3373-gzip_oom, it seems that this can solve the problem, thank you very much! BTW, when will the official version be released, and the version number?

sbordet

sbordet commented on Feb 27, 2019

@sbordet
sbordet
on Feb 27, 2019
Contributor

It's going to be in 9.4.16, I don't know when yet.
Note that the issue is still under review and the fix may change.
Thanks for confirming that the issue is solved with the initial fix!

sbordet
added 2 commits that reference this issue on Feb 27, 2019

Issue #3373 - OutOfMemoryError: Java heap space in GZIPContentDecoder.

0c6ee94

Issue #3373 - OutOfMemoryError: Java heap space in GZIPContentDecoder.

057575f
sbordet
closed this as completedon Feb 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @olamy@gregw@sbordet@ltp217

        Issue actions
          OutOfMemoryError: Java heap space in GZIPContentDecoder · Issue #3373 · jetty/jetty.project