From bg2382 at columbia.edu  Wed Mar 28 17:20:33 2012
From: bg2382 at columbia.edu (Ben Goodrich)
Date: Wed, 28 Mar 2012 11:20:33 -0400
Subject: [Rd] serialization regression in 2.15.0 beta
In-Reply-To: <4F72A82A.4080008@stats.ox.ac.uk>
References: <20120323191346.lmq6atwzhcgcsg00@cubmail.cc.columbia.edu>
	<4F722A8D.6090105@columbia.edu> <4F72A82A.4080008@stats.ox.ac.uk>
Message-ID: <20120328112033.o329uzutgk88kswo@cubmail.cc.columbia.edu>

Quoting Prof Brian Ripley <ripley at stats.ox.ac.uk>:

> On 27/03/2012 22:01, Ben Goodrich wrote:
>> In case anyone is concerned that this regression will affect them, the
>> code was reverted to the 2.14.x behavior by
>>
>> ------------------------------------------------------------------------
>> r58842 | ripley | 2012-03-26 08:12:43 -0400 (Mon, 26 Mar 2012) | 1 line
>> Changed paths:
>>    M /branches/R-2-15-branch/doc/NEWS.Rd
>>    M /branches/R-2-15-branch/src/library/parallel/R/unix/forkCluster.R
>>    M /branches/R-2-15-branch/src/library/parallel/R/unix/mcfork.R
>>
>> revert to XDR serialization for 2.15.0
>> ------------------------------------------------------------------------
>
> But the underlying problem (in non-xdr binary unserialization) is AFAWK
> fixed: it was just that at this late stage there was too little time to
> test thoroughly before release.
>
> Please test R-devel on your own problem (we haven't: the issue was
> found using a different example from elsewhere).

Indeed, the issue seems to be fixed in r-devel for my example.

Thanks,
Ben


>>> I am experiencing a problem related to serialization behavior in
>>> 2.15.0 beta (binary installed from Debian unstable) and 2.16.0 (from
>>> svn) that is not present in 2.14.2 (binary from Debian testing).
>>>
>>> I don't fully understand the problem. Also, I tried but have not yet
>>> been able to create a small, self-contained example that reproduces
>>> the problem. However, I do have a large, not self-contained example,
>>> which requires an alpha version (not yet on CRAN) of the mi package
>>> (the mi package on CRAN would not exhibit this issue). Anyone
>>> interested in reproducing the problem can follow the readme.txt file
>>> in this directory:
>>>
>>> http://www.columbia.edu/~bg2382/mi/serialization/
>>>
>>> I track r-devel with git-svn and was able to git bisect to svn   
>>> commit r58219
>>>
>>> commit 799102bd9d0266fe89c3120981decf0b1f17ef11
>>> Author: ripley<ripley at 00db46b3-68df-0310-9c12-caf00c1e9a41>
>>> Date:   Sat Jan 28 15:02:34 2012 +0000
>>>
>>>      make use of non-xdr serialization;.
>>>
>>> although this commit could merely expose the problem rather than cause it.
>>>
>>> The problem occurs when the FUN called by mclapply() in the parallel
>>> package returns a S4 object that contains a slot (called X) that is a
>>> large matrix, specifically a "model matrix" similar to that produced
>>> by glm(). Some columns of this matrix get corrupted with wrong values
>>> (usually zero, but sometimes NaN or 10^300ish), which can be seen by
>>> examining X right before FUN returns (to mclapply()'s environment) and
>>> comparing to the "same" X after mclapply() returns to the calling
>>> environment.
>>>
>>> Part of svn commit r58219 is this hunk
>>>
>>> diff --git a/src/library/parallel/R/unix/mcfork.R
>>> b/src/library/parallel/R/unix/mcfork.R
>>> index 8e27534..4f92193 100644
>>> --- a/src/library/parallel/R/unix/mcfork.R
>>> +++ b/src/library/parallel/R/unix/mcfork.R
>>> @@ -82,7 +82,8 @@ mckill<- function(process, signal = 2L)
>>>   ## used by mcparallel, mclapply
>>>   sendMaster<- function(what)
>>>   {
>>> -    if (!is.raw(what)) what<- serialize(what, NULL, FALSE)
>>> +    # This is talking to the same machine, so no point in using xdr.
>>> +    if (!is.raw(what)) what<- serialize(what, NULL, xdr = FALSE)
>>>       .Call(C_mc_send_master, what, PACKAGE = "parallel")
>>>   }
>>>
>>> Contrary to the comment, I have found that if I specify xdr = TRUE, I
>>> get the expected (non-corrupted X slot) behavior in 2.16.0, even
>>> though it is forking locally on my 64bit Debian laptop with a little
>>> endian i7 processor, whose specs are
>>>
>>> goodrich at CYBERPOWERPC:/tmp/serialization$ cat /proc/cpuinfo
>>> processor       : 0
>>> vendor_id       : GenuineIntel
>>> cpu family      : 6
>>> model           : 42
>>> model name      : Intel(R) Core(TM) i7-2630QM CPU @ 2.00GHz
>>> stepping        : 7
>>> microcode       : 0x17
>>> cpu MHz         : 800.000
>>> cache size      : 6144 KB
>>> physical id     : 0
>>> siblings        : 8
>>> core id         : 0
>>> cpu cores       : 4
>>> apicid          : 0
>>> initial apicid  : 0
>>> fpu             : yes
>>> fpu_exception   : yes
>>> cpuid level     : 13
>>> wp              : yes
>>> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>>> syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl
>>> xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl
>>> vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt
>>> tsc_deadline_timer xsave avx lahf_lm ida arat epb xsaveopt pln pts dts
>>> tpr_shadow vnmi flexpriority ept vpid
>>> bogomips        : 3990.83
>>> clflush size    : 64
>>> cache_alignment : 64
>>> address sizes   : 36 bits physical, 48 bits virtual
>>> power management:
>>>
>>> ...
>>>
>>> processor       : 7
>>> [same as processor 0]
>>>
>>> So, to summarize I get the good behavior on R 2.14.2 when using
>>> mclapply(), on 2.15.0 beta when using lapply(), and on 2.16.0 using
>>> mclapply() iff I patch in xdr = TRUE in sendMaster(). I get the bad
>>> behavior on 2.15.0 beta and unpatched 2.16.0 when using mclapply().
>>>
>>> My session info:
>>>
>>>> sessionInfo()
>>> R version 2.15.0 beta (2012-03-16 r58769)
>>> Platform: x86_64-pc-linux-gnu (64-bit)
>>>
>>> locale:
>>>   [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
>>>   [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
>>>   [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
>>>   [7] LC_PAPER=C                 LC_NAME=C
>>>   [9] LC_ADDRESS=C               LC_TELEPHONE=C
>>> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>>>
>>> attached base packages:
>>> [1] stats4    stats     graphics  grDevices utils     datasets  methods
>>> [8] base
>>>
>>> other attached packages:
>>>   [1] mi_0.9-83        bigmemory_4.2.11 arm_1.5-03       foreign_0.8-49
>>>   [5] abind_1.4-0      R2WinBUGS_2.1-18 coda_0.14-5      lme4_0.999375-42
>>>   [9] Matrix_1.0-4     lattice_0.20-0   MASS_7.3-17
>>>
>>> loaded via a namespace (and not attached):
>>> [1] grid_2.15.0  nlme_3.1-103
>>>
>>> Thanks,
>>> Ben
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>
> -- 
> Brian D. Ripley,                  ripley at stats.ox.ac.uk
> Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
> University of Oxford,             Tel:  +44 1865 272861 (self)
> 1 South Parks Road,                     +44 1865 272866 (PA)
> Oxford OX1 3TG, UK                Fax:  +44 1865 272595



[Rd] serialization regression in 2.15.0 beta
Ben Goodrich