Getting error in Linux OS with .Net Core 2.2 - The configured user limit (128) on the number of inotify instances has been reached #7531
Comments
|
Hi @jaydeept could you show what your code looks like? Also, would it be possible to include a better-formatted log output? It's quite difficult to read that log with all the wrapping in odd places. |
|
BTW this error presumably happens in any instance where there are a lot of file watchers. Configuration file watchers are a common case, but there can be any number of other cases. You can also look at raising the limits on your system by doing something like what's described in the answers here: https://stackoverflow.com/questions/32281277/too-many-open-files-failed-to-initialize-inotify-the-user-limit-on-the-total |
|
Sure thing @Eilon , Here is the stacktrace and formatted log: And here is my code from Startup: public class Startup
{
public static void Main(string[] args)
{
var configurationSettings = new ConfigurationSettings(args);
var properties = GetConfigurationProperties(configurationSettings);
var host = new WebHostBuilder()
.UseContentRoot(Directory.GetCurrentDirectory())
.UseStartup<Startup>()
.UseKestrel()
.UseUrls(configurationSettings.GetBaseAddress())
.ConfigureServices(services => services.AddProperties(properties))
.Build();
host.Run();
}
// This method gets called by the runtime. Use this method to add services to the container.
// For more information on how to configure your application, visit https://go.microsoft.com/fwlink/?LinkID=398940
public IServiceProvider ConfigureServices(IServiceCollection services)
{
services.AddAutofac();
services.AddLogging();
services.AddHttpContextAccessor();
services.AddMvc().AddSerializerSettings();
services.AddSwaggerGen(SwaggerConfig.ConfigureHelpDocuments);
return services.GetAutofacServiceProvider();
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
app.UseOptions();
app.UseRewriter(SetAliasesForDocumentationPage("doc", "docs", "help"));
app.UseStaticFiles();
app.UseMvc();
app.UseSwagger();
app.UseSwaggerUI(SwaggerConfig.ConfigureHelpUi);
}
private static RewriteOptions SetAliasesForDocumentationPage(params string[] aliasesForDoc)
{
var options = new RewriteOptions();
foreach (var aliasForDoc in aliasesForDoc)
{
options.AddRedirect(aliasForDoc, "swagger");
}
return options;
}And starcktrace has reported line DITests.cs:line 85 which is below one where I am using TestServer for my testing purpose. : m_TestServer = new TestServer(webHostBuilder); |
|
Also regarding your post related to file watchers, I don't think I am using one. This is happening inside container as part of CI so I don't have privilege to modify the limits.conf file by going there. |
|
Hello @Eilon , Any input on this issue? |
|
@jaydeept sorry I'm not super sure. Maybe try disabling some of the features you're using and see if it makes a difference? There are some components in use there that might involve file watchers (though I'm not certain). Stuff such as Logging, Swagger, or other components. This would be helpful to at least see if we can figure out which specific component (if any) is causing this, and then figure out where to go from there. |
|
We are also seeing this in some helix queues (Fedora/some Ubuntu) now that identity functional tests are working, see https://mc.dot.net/#/user/aspnetcore/pr~2Faspnet~2Faspnetcore/ci/20190321.8/workItem/Microsoft.AspNetCore.Identity.FunctionalTests-netcoreapp3.0/wilogs |
|
Hi, @Eilon Try to set up environment variable DOTNET_USE_POLLING_FILE_WATCHER=1 |
|
follow this for ubuntu I found the following blog post that might help pinpoint the problem and solution. Hope it helps. https://www.danieldevelops.com/inotify-watcher-dotnet-core Alternative is to increase the inotify limit, which I have done until you (hopefully) fix the issue. Just for reference for others. You can get your current inotify file watch limit by executing: $ cat /proc/sys/fs/inotify/max_user_watches You can set a temporary new limit with: $ sudo sysctl -w fs.inotify.max_user_watches=16384 If you like to make your limit permanent use: $ echo fs.inotify.max_user_watches=16384 | sudo tee -a /etc/sysctl.conf Thx for the great work. Love the PT Magic! After fix restart your vs code |
|
I can confirm this issue in Ubuntu 18.04. Running a brand new webapi project is enough to trigger the exception. Steps to reproduce:
|
|
Same issue. The problem appeared in Jetbrain Rider while vs code is opened. When vscode it's closed, works well. vscode 1.31.1 I installed my system on a brand new drive a week ago and made the change to By the way the value in my file is |
|
Hi @jaydeept , Have you found the root cause of this issue by any chance?. I'm also facing the same issue |
|
use command |
|
I get this with increasing consistency when deploying new Linux hosts to my on-prem enterprise k8s cluster. I've assigned the configs/jsons to not refresh, supposedly disabling the file watch. I've set an env var in my dockerfile And I still usually, not always, get a fatal startup failing my pods. It goes away with an indeterminate number of pod refreshes. Locally the containers seem to startup consistently in my docker desktop k8s.
|
|
Can anyone provide a runnable sample project that reproduces this issue? That would help us diagnose this. It looks like something in config is still registering file system watchers. We do also have the |
|
I've created a boilerplate ASP.Net Core 2.2 WebAPI project, with no additional functionality and been able to "reproduce" this error in our environment. Unfortunately, it's an intermittent issue. We can't track down what resources aren't available during startup (Linux host has 100's of free GB and 2GB mem cap on pods). So, while it's consistent enough for it to become a critical issue, it's not consistent enough for us to nail down in a 100% repro project. |
My understanding is that there is a hard limit on the number of file watchers that you can have (the "inotify instances"), so even though you have a lot of free memory you have to actually adjust that limit to avoid this issue. You can do that with this command: There are some issues with inotify and docker. How many containers do you have running per machine? The inotify instance limit is per user not per process so if you have a lot of processes running it can cause issues. |
analogrelay
added
Needs: Author Feedback
|
It seems like our usage of the same user across all our pods in the cluster was creating this issue. I rewrote our dockerfiles to have their own unique users and our pods appear to be behaving better. |
|
Ah yeah, that could definitely cause a problem like this. In that situation (where you intentionally have a lot of processes wiht the same user) it's also reasonable to increase this limit (as long as you know why the current limit isn't sufficient, which in this case you do). Closing this for now on the supposition that this is due to sharing a single user for multiple applications. Please feel free to comment further, open a new issue, etc. if you're seeing this and don't fall into that category. |
|
your linux docker must have this extra setup: |
ghost
locked as resolved and limited conversation to collaborators
I am using .Net Core 2.2 for my web project. I got test cases which I am executing using
dotnet vstestas part of CI pipeline in Linux docker. However the execution of test cases fails sometimes with error "The configured user limit (128) on the number of inotify instances has been reached."However if I re-run, it works successfully. I am not sure what's going wrong. I had a look at this post but I am not using
JSONfile in myStartupclass, also I am callingbuild()only once in myMainmethod.Error Message:
The text was updated successfully, but these errors were encountered: