Too Many Open Files

[darkling235]

This may be more of a google issue than a GXT issue but I'm noticing this stacktrace intermittently but regularly. It seems to only occur on Linux and Macs. I did some research and some people claim that this is due to the ClientBundle having a serious bug in it but I haven't found any posted bugs from Google to confirm this. Has anyone found a good solution to this? I found a few posts which talk about increasingly the max number of files in linux but it doesn't seem to help how much we jack it up. Can anyone comment on this issue?
Thanks

javax.imageio.IIOException: Can't create output stream![INFO]  at javax.imageio.ImageIO.write(ImageIO.java:1560)
[INFO]  at com.google.gwt.resources.rg.ImageBundleBuilder.createImageBytes(ImageBundleBuilder.java:558)
[INFO]  at com.google.gwt.resources.rg.ImageBundleBuilder.toPng(ImageBundleBuilder.java:544)
[INFO]  at com.google.gwt.resources.rg.ImageResourceGenerator.reencodeToTempFile(ImageResourceGenerator.java:641)
[INFO]  at com.google.gwt.resources.rg.ImageResourceGenerator.prepare(ImageResourceGenerator.java:567)
[INFO]  at com.google.gwt.resources.rebind.context.AbstractClientBundleGenerator.initAndPrepare(AbstractClientBundleGenerator.java:1043)
[INFO]  at com.google.gwt.resources.rebind.context.AbstractClientBundleGenerator.initAndPrepare(AbstractClientBundleGenerator.java:1069)
[INFO]  at com.google.gwt.resources.rebind.context.AbstractClientBundleGenerator.generateIncrementally(AbstractClientBundleGenerator.java:412)
[INFO]  at com.google.gwt.dev.javac.StandardGeneratorContext.runGeneratorIncrementally(StandardGeneratorContext.java:647)
[INFO]  at com.google.gwt.dev.cfg.RuleGenerateWith.realize(RuleGenerateWith.java:41)
[INFO]  at com.google.gwt.dev.shell.StandardRebindOracle$Rebinder.rebind(StandardRebindOracle.java:78)
[INFO]  at com.google.gwt.dev.shell.StandardRebindOracle.rebind(StandardRebindOracle.java:268)
[INFO]  at com.google.gwt.dev.shell.StandardRebindOracle.rebind(StandardRebindOracle.java:257)
[INFO]  at com.google.gwt.dev.DistillerRebindPermutationOracle.getAllPossibleRebindAnswers(DistillerRebindPermutationOracle.java:91)
[INFO]  at com.google.gwt.dev.jdt.WebModeCompilerFrontEnd.doFindAdditionalTypesUsingRebinds(WebModeCompilerFrontEnd.java:96)
[INFO]  at com.google.gwt.dev.jdt.AbstractCompiler$Sandbox$CompilerImpl.process(AbstractCompiler.java:254)
[INFO]  at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:444)
[INFO]  at com.google.gwt.dev.jdt.AbstractCompiler$Sandbox$CompilerImpl.compile(AbstractCompiler.java:173)
[INFO]  at com.google.gwt.dev.jdt.AbstractCompiler$Sandbox$CompilerImpl.compile(AbstractCompiler.java:288)
[INFO]  at com.google.gwt.dev.jdt.AbstractCompiler$Sandbox$CompilerImpl.access$400(AbstractCompiler.java:139)
[INFO]  at com.google.gwt.dev.jdt.AbstractCompiler.compile(AbstractCompiler.java:588)
[INFO]  at com.google.gwt.dev.jdt.BasicWebModeCompiler.getCompilationUnitDeclarations(BasicWebModeCompiler.java:97)
[INFO]  at com.google.gwt.dev.jdt.WebModeCompilerFrontEnd.getCompilationUnitDeclarations(WebModeCompilerFrontEnd.java:52)
[INFO]  at com.google.gwt.dev.jjs.JavaToJavaScriptCompiler.precompile(JavaToJavaScriptCompiler.java:569)
[INFO]  at com.google.gwt.dev.jjs.JavaScriptCompiler.precompile(JavaScriptCompiler.java:33)
[INFO]  at com.google.gwt.dev.Precompile.precompile(Precompile.java:284)
[INFO]  at com.google.gwt.dev.Precompile.precompile(Precompile.java:233)
[INFO]  at com.google.gwt.dev.Precompile.precompile(Precompile.java:145)
[INFO]  at com.google.gwt.dev.Compiler.run(Compiler.java:232)
[INFO]  at com.google.gwt.dev.Compiler.run(Compiler.java:198)
[INFO]  at com.google.gwt.dev.Compiler$1.run(Compiler.java:170)
[INFO]  at com.google.gwt.dev.CompileTaskRunner.doRun(CompileTaskRunner.java:88)
[INFO]  at com.google.gwt.dev.CompileTaskRunner.runWithAppropriateLogger(CompileTaskRunner.java:82)
[INFO]  at com.google.gwt.dev.Compiler.main(Compiler.java:177)
[INFO] Caused by: javax.imageio.IIOException: Can't create cache file!
[INFO]  at javax.imageio.ImageIO.createImageOutputStream(ImageIO.java:397)
[INFO]  at javax.imageio.ImageIO.write(ImageIO.java:1558)
[INFO]  ... 33 more
[INFO] Caused by: java.io.IOException: Too many open files
[INFO]  at java.io.UnixFileSystem.createFileExclusively(Native Method)
[INFO]  at java.io.File.checkAndCreate(File.java:1704)
[INFO]  at java.io.File.createTempFile(File.java:1792)
[INFO]  at javax.imageio.stream.FileCacheImageOutputStream.<init>(FileCacheImageOutputStream.java:71)
[INFO]  at com.sun.imageio.spi.OutputStreamImageOutputStreamSpi.createOutputStreamInstance(OutputStreamImageOutputStreamSpi.java:50)
[INFO]  at javax.imageio.ImageIO.createImageOutputStream(ImageIO.java:393)
[INFO]  ... 34 more

[icfantv]

Yea, it's an issue in ClientBundle - there appears to be a leak. Per Colin from Sencha, a System.gc() fixes the issue but since this is not end-developer code, there's not a lot we can do on this.

We've seen this on our TeamCity build servers and had to increase the hard/soft nofile limits in /etc/security/limits.conf - not sure what the equivalents are for Windows machines.

The default is 1024 and is clearly too low as we were peaking out at over 2600 at one point and we have a pretty massive GXT application.

We thought we had an issue filed w/ GWT, but now are unable to locate it. I'll try to remember to update this in the morning with the results of any searching or a ticket URL for tracking purposes.

[sblommers]

We had this problem as well on Bamboo. To work around this problem we had to set limits pretty high for our Debian build machine. The problem did not occur always. The build may still run out of memory during compilation of the GWT frontend but this doesn't happen very often.

Note: when setting limits in /etc/security/limits.conf for a specific user they are ignored when changing to a user as root (with su -l).


We use an openvz container (virtual machine) as build server for bamboo (121 = id for our server).
vzctl set 121 --numfile 8192 --save


On the bamboo vm add/replaced the following lines in /etc/security/limits.conf
bamboo soft nofile 8192
bamboo hard nofile 8192


Unfortunately we needed to reboot the vm for the changes to be applied.


Good luck!


Sebastiaan

[Colin Alworth]

I've filed this as http://code.google.com/p/google-web-...&ts=1357248530 and have a code review at http://gwt-code-reviews.appspot.com/1880803/. If you are able, apply the patch and build GWT (should apply to trunk or 2.5.0), and test again. I'll see about making a mavenized build of this that you can test from for those that would be helped by this.

As mentioned in the issue, I'm not totally convinced that there isn't a nicer way to mop up, and am open to any suggestions.

[darkling235]

Thank you for all the help. We've managed to get around most of these issues so far with a combination of Linux tricks and reducing our permutations. I'll try some of these fixes to see if that makes it no longer necessary

[Colin Alworth]

Reducing permutations is unlikely to help with this (its possible though by affecting memory pressure, since a gc will clean up those open files), since that just avoids some builds - the images must be generated for any build to work, so that will only be done once. That 'once' is enough to cause this issue.