From tom@@@k@liber@ @ending from gm@il@com  Tue May 22 15:13:05 2018
From: tom@@@k@liber@ @ending from gm@il@com (Tomas Kalibera)
Date: Tue, 22 May 2018 15:13:05 +0200
Subject: [Rd] R Bug: write.table for matrix of more than 2, 147, 483,
 648 elements
In-Reply-To: <c9b2730a-0575-25a5-3bad-cd6690108af8@gmail.com>
References: <20180418212319.34878428@hypatia.math.ethz.ch>
 <e523f994-7f4c-b94f-dcda-1a0b96c70d06@gmail.com>
 <ccd7638e-3105-cad1-1eb4-87120a47d0da@gmail.com>
 <0b1ada3d-fdf0-b975-d5c4-70fe353de7c9@insa-toulouse.fr>
 <c9b2730a-0575-25a5-3bad-cd6690108af8@gmail.com>
Message-ID: <a1abded9-08a4-1673-952f-4ad28e262099@gmail.com>

Fixed in R-devel 74754.
Tomas

On 04/19/2018 12:15 PM, Tomas Kalibera wrote:
> On 04/19/2018 11:47 AM, Serguei Sokol wrote:
>> Le 19/04/2018 ? 09:30, Tomas Kalibera a ?crit?:
>>> On 04/19/2018 02:06 AM, Duncan Murdoch wrote:
>>>> On 18/04/2018 5:08 PM, Tousey, Colton wrote:
>>>>> Hello,
>>>>>
>>>>> I want to report a bug in R that is limiting my capabilities to 
>>>>> export a matrix with write.csv or write.table with over 
>>>>> 2,147,483,648 elements (C's int limit). I found this bug already 
>>>>> reported about before: 
>>>>> https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17182. 
>>>>> However, there appears to be no solution or fixes in upcoming R 
>>>>> version releases.
>>>>>
>>>>> The error message is coming from the writetable part of the utils 
>>>>> package in the io.c source 
>>>>> code(https://svn.r-project.org/R/trunk/src/library/utils/src/io.c):
>>>>> /* quick integrity check */
>>>>> ???????????????? if(XLENGTH(x) != (R_len_t)nr * nc)
>>>>> ???????????????????? error(_("corrupt matrix -- dims not not match 
>>>>> length"));
>>>>>
>>>>> The issue is that nr*nc is an integer and the size of my matrix, 
>>>>> 2.8 billion elements, exceeds C's limit, so the check forces the 
>>>>> code to fail.
>>>>
>>>> Yes, looks like a typo:? R_len_t is an int, and that's how nr was 
>>>> declared.? It should be R_xlen_t, which is bigger on machines that 
>>>> support big vectors.
>>>>
>>>> I haven't tested the change; there may be something else in that 
>>>> function that assumes short vectors.
>>> Indeed, I think the function won't work for long vectors because of 
>>> EncodeElement2 and EncodeElement0. EncodeElement2/0 would have to be 
>>> changed, including their signatures
>>
>> That would be a definite fix but before such deep rewriting is 
>> undertaken may the following small fix (in addition to "(R_xlen_t)nr 
>> * nc") will be sufficient for cases where nr and nc are in int range 
>> but their product can reach long vector limit:
>>
>> replace
>> ??? tmp = EncodeElement2(x, i + j*nr, quote_col[j], qmethod,
>> ??? ??? ??? ??? ??? &strBuf, sdec);
>> by
>> ??? tmp = EncodeElement2(VECTOR_ELT(x, (R_xlen_t)i + j*nr), 0, 
>> quote_col[j], qmethod,
>> ??? ??? ??? ??? ??? &strBuf, sdec);
>
> Unfortunately we can't do that, x is a matrix of an atomic vector 
> type. VECTOR_ELT is taking elements of a generic vector, so it cannot 
> be applied to "x". But even if we extracted a single element from "x" 
> (e.g. via a type-switch etc), we would not be able to pass it to 
> EncodeElement0 which expects a full atomic vector (that is, including 
> its header). Instead we would have to call functions like 
> EncodeInteger, EncodeReal0, etc on the individual elements. Which is 
> then the same as changing EncodeElement0 or implementing a new version 
> of it. This does not seem that hard to fix, just is not as trivial as 
> changing the cast..
>
> Tomas
>
>
>>
>> Serguei
>
>



[Rd] R Bug: write.table for matrix of more than 2, 147, 483, 648 elements
Tomas Kalibera