From tomas.kalibera at gmail.com  Mon Apr 24 12:24:04 2017
From: tomas.kalibera at gmail.com (Tomas Kalibera)
Date: Mon, 24 Apr 2017 12:24:04 +0200
Subject: [Rd] Crash after (wrongly) applying product operator on object
 from LIMMA package
In-Reply-To: <aa174a90-0d30-4442-8787-0d1f6814ca4a@mpiib-berlin.mpg.de>
References: <e3a03266-b7f3-d5df-6940-906f9bd41104@mpiib-berlin.mpg.de>
 <9c987df5-1852-56b6-3de5-89358aaffb0a@mpiib-berlin.mpg.de>
 <CA+A1kV6sNGsNW1-ULZ7-uDOq_M91zr7aJKX4aRaeogfGe_SSPw@mail.gmail.com>
 <aa174a90-0d30-4442-8787-0d1f6814ca4a@mpiib-berlin.mpg.de>
Message-ID: <067f5b53-8786-c9ef-64c5-769cf47e7a27@gmail.com>


Yes, the mentioned pull request for rJava is to workaround a JVM problem 
that is Linux-only.
I am not aware of any related problem on OSX.

In general, you can currently get two kinds of messages with infinite 
recursion:

Error: evaluation nested too deeply: infinite recursion / 
options(expressions=)?
Error: C stack usage XXX is too close to the limit

but in some situations you could get a segfault, an obvious one is 
infinite recursion in native code of a package.

If you're getting a segfault it is probably best to check with a 
debugger - infinite recursion would be easily identified in the stacktrace.

Best
Tomas


On 04/24/2017 11:58 AM, Hilmar Berger wrote:
> Hi January,
>
> I believe the root of the xlsx issue has been identified and a fix
> suggested by Tomas Kalibera (see https://github.com/s-u/rJava/pull/102).
> In a nutshell, Oracle Java on Linux modifies the stack in a way that
> makes it smaller and and the same time makes it impossible for R to
> detect this change, leading to segfaults. It is not clear to me that the
> same problem would occur on Mac, since the behavior of Oracle seems to
> be Linux specific. Possibly even Linux users on OpenJDK might not
> encounter any problems (not tested).
>
> So possibly the next release of rJava should also fix the xlsx problems
> with other packages.
>
> Best regards,
> Hilmar
>
>
> On 24/04/17 11:46, January W. wrote:
>> Hi Hilmar,
>>
>> weird. The memory problem seems be due to recursion (my R, version
>> 3.3.3, says: Error: evaluation nested too deeply: infinite recursion /
>> options(expressions=)?, just write traceback() to see how it happens),
>> but why does it segfault with xlsx? Nb xlsx is the culprit: neither
>> rJava nor xlsxjars cause the problem.
>>
>> On the other hand, quick googling for r+xlsx+segfault returns tons of
>> reports of how xlsx crashes in dozens of situations. See for example
>> http://r.789695.n4.nabble.com/segfault-in-gplots-heatmap-2-td4641808.html.
>> Also, the problem might be platform-specific. It would be interesting
>> to see whether anyone with a Mac can reproduce it.
>>
>> kind regards,
>>
>> j.
>>
>>
>>
>>
>>
>> On 19 April 2017 at 10:01, Hilmar Berger <berger at mpiib-berlin.mpg.de
>> <mailto:berger at mpiib-berlin.mpg.de>> wrote:
>>
>>      Hi,
>>
>>      following up on my own question, I found smaller example that does
>>      not require LIMMA:
>>
>>      setClass("FOOCLASS",
>>               representation("list")
>>      )
>>      ma = new("FOOCLASS", list(M=matrix(rnorm(300), 30,10)))
>>
>>      > ma * ma$M
>>      Error: C stack usage  7970512 is too close to the limit
>>
>>      > library(xlsx)
>>      Loading required package: rJava
>>      Loading required package: xlsxjars
>>      > ma * ma$M
>>      ---> Crash
>>
>>      xlsx seems to act like a catalyst here, with the product operator
>>      running in a deep nested iteration, exhausting the stack. Valgrind
>>      shows thousands of invalid stack accesses when loading xslx, which
>>      might contribute to the problem. Package xlsx has not been updated
>>      since 2014, so it might fail with more current versions of R or
>>      Java (I'm using Oracle Java 8).
>>
>>      Still, even if xlsx was the package to be blamed for the crash, I
>>      fail to understand what exactly the product operator is trying to
>>      do in the multiplication of the matrix with the object.
>>
>>      Best regards,
>>      Hilmar
>>
>>
>>      On 18/04/17 18:57, Hilmar Berger wrote:
>>
>>          Hi,
>>
>>          this is a problem that occurs in the presence of two libraries
>>          (limma, xlsx) and leads to a crash of R. The problematic code
>>          is the wrong application of sweep or the product ("*")
>>          function on an LIMMA MAList object. To my knowledge, limma
>>          does not define a "*" method for MAList objects.
>>
>>          If only LIMMA is loaded but not package xlsx, the code does
>>          not crash but rather produces an error ("Error: C stack usage
>>          7970512 is too close to the limit"). Loading only package
>>          rJava instead of xlsx does also not produce the crash but the
>>          error message instead. Note that xlsx functions are not
>>          explicitly used.
>>
>>          It could be reproduced on two different Linux machines running
>>          R-3.2.5, R-3.3.0 and R-3.3.2.
>>
>>          Code to reproduce the problem:
>>          ---------------------------------
>>          library(limma)
>>          library(xlsx)
>>
>>          # a MAList
>>          ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
>>          M=matrix(rnorm(300), 30,10)))
>>
>>          # This should actually be sweep(ma$M, ...) for functional
>>          code, but I omitted the $M...
>>          #sweep(ma, 2, c(1:10), "*")
>>          # sweep will crash when doing the final operation of applying
>>          the function over the input matrix, which in this case is
>>          function "*"
>>
>>          f = match.fun("*")
>>          # This is not exactly the same as in sweep but it also tries
>>          to multiply the MAList object with a matrix of same size and
>>          leads to the crash
>>          f(ma, ma$M)
>>          # ma * ma$M has the same effect
>>          ---------------------------------
>>
>>          My output:
>>
>>          R version 3.3.0 (2016-05-03) -- "Supposedly Educational"
>>          Copyright (C) 2016 The R Foundation for Statistical Computing
>>          Platform: x86_64-pc-linux-gnu (64-bit)
>>
>>          R is free software and comes with ABSOLUTELY NO WARRANTY.
>>          You are welcome to redistribute it under certain conditions.
>>          Type 'license()' or 'licence()' for distribution details.
>>
>>            Natural language support but running in an English locale
>>
>>          R is a collaborative project with many contributors.
>>          Type 'contributors()' for more information and
>>          'citation()' on how to cite R or R packages in publications.
>>
>>          Type 'demo()' for some demos, 'help()' for on-line help, or
>>          'help.start()' for an HTML browser interface to help.
>>          Type 'q()' to quit R.
>>
>>          > library(limma)
>>          > library(xlsx)
>>          Loading required package: rJava
>>          Loading required package: xlsxjars
>>          >
>>          > sessionInfo()
>>          R version 3.3.0 (2016-05-03)
>>          Platform: x86_64-pc-linux-gnu (64-bit)
>>          Running under: Ubuntu 14.04.5 LTS
>>
>>          locale:
>>           [1] LC_CTYPE=en_US.UTF-8          LC_NUMERIC=C
>>          LC_TIME=en_US.UTF-8
>>           [4] LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
>>          LC_MESSAGES=en_US.UTF-8
>>           [7] LC_PAPER=en_US.UTF-8          LC_NAME=en_US.UTF-8
>>          LC_ADDRESS=en_US.UTF-8
>>          [10] LC_TELEPHONE=en_US.UTF-8 LC_MEASUREMENT=en_US.UTF-8
>>          LC_IDENTIFICATION=en_US.UTF-8
>>
>>          attached base packages:
>>          [1] stats     graphics  grDevices utils     datasets methods base
>>
>>          other attached packages:
>>          [1] xlsx_0.5.7     xlsxjars_0.6.1 rJava_0.9-8 limma_3.30.7
>>
>>          loaded via a namespace (and not attached):
>>          [1] tools_3.3.0
>>          >
>>          > ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
>>          M=matrix(rnorm(300), 30,10)))
>>          > #sweep(ma, 2, c(1:10), "*")
>>          >
>>          > f = match.fun("*")
>>          > f
>>          function (e1, e2)  .Primitive("*")
>>
>>          > f(ma, ma$M)
>>
>>          ----> crash to command line with segfault.
>>
>>          Best regards,
>>          Hilmar
>>
>>
>>      --
>>      Dr. Hilmar Berger, MD
>>      Max Planck Institute for Infection Biology
>>      Charit?platz 1
>>      D-10117 Berlin
>>      GERMANY
>>
>>      Phone: + 49 30 28460 430 <tel:%2B%2049%2030%2028460%20430>
>>      Fax: + 49 30 28460 401 <tel:%2B%2049%2030%2028460%20401>
>>       E-Mail: berger at mpiib-berlin.mpg.de
>>      <mailto:berger at mpiib-berlin.mpg.de>
>>      Web   : www.mpiib-berlin.mpg.de <http://www.mpiib-berlin.mpg.de>
>>
>>      ______________________________________________
>>      R-devel at r-project.org <mailto:R-devel at r-project.org> mailing list
>>      https://stat.ethz.ch/mailman/listinfo/r-devel
>>      <https://stat.ethz.ch/mailman/listinfo/r-devel>
>>
>>
>>
>>
>> -- 
>> -------- January Weiner --------------------------------------



[Rd] Crash after (wrongly) applying product operator on object from LIMMA package
Tomas Kalibera