@zD98 Please create minimum reproducible test repo

project git: https://github.com/zD98/webpack-memory-test
npm start
Because the repo is very small, so multiple operations that change code quickly can increase the memory, and it would not be released.

Having same issue on webpack 4.5.0.
The webpack just crash (out of memory) after a few incremental rebuilds

Unchanged module string should be compiled and cached once. Unchanged modules are compiled every time in rebuilding Now. So does the CachePlugin have some errors ?

I just upgraded from 3.10.0 to 4.5.0 and I'm seeing this sporadically in development:

i ｢wdm｣: Compiling...
webpack building...

<--- Last few GCs --->

[9284:000000000028FA40]  1100280 ms: Mark-sweep 1385.1 (1411.9) -> 1385.1 (1411.9
) MB, 292.3 / 0.1 ms  allocation failure GC in old space requested
[9284:000000000028FA40]  1100624 ms: Mark-sweep 1385.1 (1411.9) -> 1385.1 (1407.9
) MB, 343.1 / 0.1 ms  last resort GC in old space requested
[9284:000000000028FA40]  1100867 ms: Mark-sweep 1385.1 (1407.9) -> 1385.1 (1407.9
) MB, 243.6 / 0.1 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000003025D1257C1 <JSObject>
    0: builtin exit frame: lastIndexOf(this=000003B48B1F6031 <Very long string[60
6700]>,00000315AAE8BF89 <String[1]\: \n>)

    1: /* anonymous */(aka /* anonymous */) [C:\www\node\poject\node_modu
les\webpack-sources\node_modules\source-list-map\lib\SourceListMap.js:~100] [pc=0
000029834AF4F94](this=0000006331D822D1 <undefined>,sln=00000157ABFAF271 <SourceNo
de map = 0000026155AF2B41>)
 ...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memor
y
 1: node_module_register
 2: v8::internal::FatalProcessOutOfMemory
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::Factory::NewRawTwoByteString
 5: v8::internal::Smi::SmiPrint
 6: v8::internal::StackGuard::HandleInterrupts
 7: v8::internal::SlicedString::SlicedStringGet
 8: v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString
 9: v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString
10: 0000029833B86B21

The only change I made to my config file was adding mode: 'development'.

@colinrcummings can you create minimum reproducible test repo?

@evilebottnawi, see above. I was able to reproduce it with this test repo by making "multiple operations that change code quickly" as @zD98 indicates.

This is the list of changes between the two modes https://medium.com/webpack/webpack-4-mode-and-optimization-5423a6bc597a so I'd disable each and see if it fixes anything. I'm slowly going through them now but for us even production eventually blows up. I imagine more people don't run production in development and so maybe it hasn't been spotted.

What might be the easiest way to fix this is to disable both modes but I can't find anything in the documentation for that.

mode = "none" still blows up...I'm not sure mode is the issue.

@briandipalma problem not in mode, problem inside plugin, need detect there and fix it

@evilebottnawi do you have some suggestions about detecting which plugin has problem? I can't get any ideas from the source code.

The use case where it's blowing up for us is during testing with webpack-karma. We launch ~40 sets of tests with a new compiler serially eventually it gets to ~30 in the list and node blows up. This worked OK with Webpack 3 and webpack-karma for Webpack 3.

I don't understand how this leak could work though, as we launch a new Karma Server per package (https://github.com/caplin/caplin-dev-tools/blob/master/karma-test-runner/src/utils.js#L90-L120) so that means a new compiler. Once the tests are finished wouldn't it garbage collect the compiler, is there some global cache that Webpack keeps adding to and needs cleaning down?

@zD98 just enable/disable loaders/plugins and run profile plugin

Same problem here. It seems that it happens after I upgraded webpack to 4.6.0

<--- Last few GCs --->

[2412:0x104002400] 25996906 ms: Mark-sweep 1379.2 (1478.5) -> 1379.1 (1479.5) MB, 1530.3 / 0.0 ms  allocation failure GC in old space requested
[2412:0x104002400] 25998428 ms: Mark-sweep 1379.1 (1479.5) -> 1379.1 (1443.0) MB, 1521.6 / 0.0 ms  last resort GC in old space requested
[2412:0x104002400] 25999977 ms: Mark-sweep 1379.1 (1443.0) -> 1379.1 (1443.0) MB, 1548.5 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1249908a55e9 <JSObject>
    1: update [internal/crypto/hash.js:57] [bytecode=0x1249429dddd9 offset=128](this=0x12499a34faf9 <Hash map = 0x12493c8ff401>,data=0x12499a34fa31 <Very long string[993669]>,encoding=0x1249334022d1 <undefined>)
    2: update [/Users/Carter/newlook/node_modules/webpack/lib/util/createHash.js:25] [bytecode=0x1249429d98b9 offset=85](this=0x12499a34fc61 <BulkUpdateDecorator map = 0x12493c8ff4b1>,da...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/local/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
 6: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/usr/local/bin/node]
 7: v8::String::WriteUtf8(char*, int, int*, int) const [/usr/local/bin/node]
 8: node::StringBytes::Write(v8::Isolate*, char*, unsigned long, v8::Local<v8::Value>, node::encoding, int*) [/usr/local/bin/node]
 9: node::StringBytes::InlineDecoder::Decode(node::Environment*, v8::Local<v8::String>, v8::Local<v8::Value>, node::encoding) [/usr/local/bin/node]
10: node::crypto::Hash::HashUpdate(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
11: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/usr/local/bin/node]
12: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/usr/local/bin/node]
13: v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
14: 0x1c55607842fd
15: 0x1c556083d196
16: 0x1c556083d196
17: 0x1c556078535f
18: 0x1c556083d196
make: *** [Makefile:2: dev] Abort trap: 6

@CarterLi Can you create minimum reproducible test repo?

I've figured out our major problem, it was Karma. PR here karma-runner/karma#2981 I think there are other leaks still present but they are smaller. It looks like my issue wasn't the same as this one anyway.

same when I use development mode.

I am having the same issue when building for production using webpack.

We need minimum reproducible repo, without this it makes no sense to write about the fact that you have a problem. Thanks!

Good point, sorry for not providing a repo, it's private (work-related). Configuration is hardly reproducible.
A few pointers:

• We use uglifyjs-webpack-plugin
• We use the latest versions of webpack and webpack-cli
• We use CompressionPlugin
• We have a lot of .scss files

Got the same issue when developing with hot reload.

Guys please read #6929 (comment) again, we can't solve this problem because in pure usage webpack no problems, i.e. problem with memory leak in some loader/plugin, please create minimum reproducible test repo if you want to solve problem fastly. Thanks!

Ok, here is a minimal test case for the leak that I am experiencing.

I extracted the example from a much larger project that I am working on. The problem here seems caused by a third-party plugin, DojoWebpackPlugin, but in some snapshot I recall seeing other plugins that were causing the issue.

It appears that webpack is keeping Compilation instances around, probably because of how tapable hooks are used. I don't know the webpack source code well enough to proceed further in the investigation. See my example README for details.

@lorenzodallavecchia Thanks!

I found the leak in Karma by using the Chrome dev tools and I'd recommend others to do the same. Starting node with --inspect-brk to debug the running process and run memory profiling. I doubt any of these issues are Webpack related, almost certainly they are in Loaders/Plugins or the usage of such.

@briandipalma if you found memory leak please create issue in loader/plugins repo. We very much appreciate this issue and fix their very fast (try to fix asap 😄 ).

I was talking about the leak in this comment #6929 (comment) I'd suggest people use the Chrome tooling to trace down the leaks themselves it's doable with some patience.

hi...maybe there are some conflicts between webpack's cache and other webpack plugin/loader's cache
In webpack4 config file, i set 'cache':true and comment cache-loader and cache option of the babel-loader，then the node don't crash.
So try it.

I did some more investigation on my leak involving dojo-webpack-plugin (see the updated test case) and I found the root cause of the leak.

Long story short: it is probably not webpack's fault.
The gist of what dojo-webpack-plugin is doing is the following.

compiler.hooks.compilation.tap("test", compilation => {
  const context = Object.create(this, {
    compilation: { value: compilation },
  });
  let fn = (function() {
    console.log("Making compilation"); 
  }).bind(context);
  compiler.hooks.make.tap("test", fn);
});

As you can see, it is registering a Compiler hook that internally holds a reference to each Compilation object. This leaks all compilations via the tapable hook.
I also tried the equivalent webpack 3 code and it leaks the same.

I will be reporting this to the plugin author since I guess this is not something a plugin should do, isn't it?

After i've commented out devtool: 'inline-source-map',
it worked without memory exception, obviously also without source maps and normal debugging.

After i've tried to add TerserPlugin to optimization (or plugins) section, and it fails again, so obvious that there is a problem with source maps library
node_modules\source-map\lib\util.js (Terser)
node_modules\source-map\lib\source-node.js ( devtool: 'inline-source-map' )

so i stick with inline-cheap-source-map option, at least it makes hot reload possible.

"dev": "node --max-old-space-size=8192 node_modules/webpack-dev-server/bin/webpack-dev-server.js --config webpack.config.dev.js"
help for me

Same issue without plugins, just loaders.
@ edit: Exactly what jokerosky mentioned! Thanks mate!

Something to note here:
If you are using a watch mode or dev-server, webpack opts to using an in-memory file system. This means, that anything generating a deterministic results [hashed file names] will persist in the memory file system forever until watch mode is killed and started again.

I cloned the ReactJs source code and ran the command
yarn build react/index,react-dom/index --type=UMD --watch.

I am still getting the error

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory.

Here's the full log

<--- Last few GCs --->

[73606:0x103800000]    27499 ms: Mark-sweep 1321.0 (1444.7) -> 1315.1 (1447.7) MB, 525.5 / 0.0 ms  (average mu = 0.206, current mu = 0.132) allocation failure scavenge might not succeed
[73606:0x103800000]    28115 ms: Mark-sweep 1326.1 (1447.7) -> 1316.2 (1450.7) MB, 533.2 / 0.0 ms  (average mu = 0.171, current mu = 0.134) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1094f435be3d]
    1: StubFrame [pc: 0x1094f435d3c6]
Security context: 0x02100039e6e1 <JSObject>
    2: /* anonymous */(aka /* anonymous */) [0x210f6ab5969] [/Users/300067056/playground/random/react/node_modules/magic-string/dist/magic-string.cjs.js:~375] [pc=0x1094f4e29ea7](this=0x0210990026f1 <undefined>,segment=0x0210c75e6e79 <JSArray[5]>)
    3: arguments adaptor frame: 3->1
    4: map [0x21000386e59](this=0x021...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10003b1db node::Abort() [/usr/local/bin/node]
 2: 0x10003b3e5 node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x1001a86b5 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x100573ad2 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 5: 0x1005765a5 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/bin/node]
 6: 0x10057244f v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 7: 0x100570624 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 8: 0x10057cebc v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 9: 0x10057cf3f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0x10054c884 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/local/bin/node]
11: 0x1007d4894 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
12: 0x1094f435be3d
13: 0x1094f435d3c6
error Command failed with signal "SIGABRT".

Can anyone please help me with this? TIA.

I'd mention that it's currently not possible to run webpack-dev-middleware without using a memory-based FS, which means that long-running webpack dev servers (or short-running ones in large projects) will have to be restarted regularly, or will need a custom development server. My team ended up doing the latter simply because of the size of our codebase.

webpack/webpack-dev-middleware#239

@salemhilal That's not ideal, of course. It is not unreasonable to expect webpack-dev-middleware to prune no longer needed files when their hash changes.

@evictor Potentially, but as I understand it, it can be non-trivial to tell when things like autogenerated bundles containing split chunks are no longer needed, since these modules may only be named by hash. Or, what if your app is loaded in the browser, a file is edited which causes an async bundle to get renamed, but then in the browser, your code requests the old async chunk? It would error out unexpectedly because the requested bundle was pruned, which could seem more like a misconfiguration than expected behavior.

It could be a whole lot more effective to allow users to just opt out of using memory-fs, since disk space is much cheaper to consume than memory. The change is only a couple of lines.

@salemhilal That's not ideal, of course. It is not unreasonable to expect webpack-dev-middleware to prune no longer needed files when their hash changes.

I think we can fix this behavior for next major release

Think that issue related only to chunk's splitting interface/plugin, so, we should free old chunks which are not important after recompilation. To reproduce we should just multiplie save diff. changes, for example 5-6 changes in one second, memory gonna be freed by gc, but not at all, for example normal vue app with a lot of components gonna take near ~300mb, if we gonna save changes it's gonna raise to 500(and cleared by gc, to 300mb again), if we gonna make to much changes it's gonna stuck at first on 500mb, then raise to 600mb-1gb, and return to 500mb(not 300mb), so, i think that there's only chunk splitting plugin issue.

After i've commented out devtool: 'inline-source-map',
it worked without memory exception, obviously also without source maps and normal debugging.

After i've tried to add TerserPlugin to optimization (or plugins) section, and it fails again, so obvious that there is a problem with source maps library
node_modules\source-map\lib\util.js (Terser)
node_modules\source-map\lib\source-node.js ( devtool: 'inline-source-map' )

so i stick with inline-cheap-source-map option, at least it makes hot reload possible.

👆 since doing this, we had no more memory leaks, developing a whole day on a huge project, with steadily running webpack dev server. just to mention again..

Ps. And all that also without the split chunks plugin

so, you suggest disable chunk splitting on dev server? But that's also not nice for recompile everything everytime.

No. This is not what I wrote. Should only be a hint to find out the root of the problem. And it seems to be with the source maps library.

i guess it is the Watching.close() (that is an async method) causes the memory leak...
in my electronjs project, each browser window has a Watching instance, so i need to call the close before the window closed to void the memory leaking, but the electron main process will crash randomly when i close windows, and if i remove the close code the crash will not happen(that's bad the watching can not be closed)

Same problem here. webpack version 4.39.2

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

I am also getting this issue:

webpack 3.8.1
webpack-dev-server 2.11.3

When adding --max-old-space-size=8192 the issue transforms into:

/Users/.../Documents/.../staging/node_modules/webpack/lib/Stats.js:221
text += `\n @ ${current.readableIdentifier(requestShortener)}`;
                                                                ^
RangeError: Invalid string length

This has progressively got worse over the last few days to the point now that I cannot work.

I close the issue because there are many different problems and it is not possible to track them, many problems have been fixed, if you encounter a problem please create a new issue with a reproducible test repo, thanks

Отличное решение. Так держать.

@afwn90cj93201nixr2e1re Нет ничего сложного открыть новую проблему с воспроизводимым примером ошибки/зависания/утечки и тд. Используйте только анлийский язык в будущем.

Мда, то есть тебе не достаточно этих репортов? 90% которых ссылаются на неверно реализованное кеширование дев сервера...
И да, это видимо главная проблема, которая до сих пор не исправлена, but issue closed, np,

@afwn90cj93201nixr2e1re Я настоятельнр рекомендую использовать анлийский для общения. Проблемы с переполнением памяти dev сервера известна при долгой работе и открыта в репозитории dev сервера, нет никакого смысла в дубликате и ее не возможно исправить на стороне webpack. Репортов недостаточно? Половина из них просто спам без какой-либо информации, я не могу починить что-то где-то не зная в чем проблема, извините я не джин или маг. Остальная половина или уже не актулально и устарело, или в сторонних плагинах, который мы тоже не можем исправить. Хочешь быстрого решения - создай хорошо описанную проблему с примером, хочешь еще быстрее - помоги в решении, это открое программное обеспечение.

ее не возможно исправить на стороне webpack

So, i think it's better to close this issue with that note, instead I close the issue because there are many different problems and it is not possible to track them, many problems have been fixed, if you encounter a problem please create a new issue with a reproducible test repo, thanks.

Here not only bug(s) with webpack-dev-server

btw most of them (90%) related to cache issue's...
5% - to old version, 4% - to old plugins, 1% - other stuff.