From wdunlap at tibco.com  Wed Mar 30 19:38:06 2011
From: wdunlap at tibco.com (William Dunlap)
Date: Wed, 30 Mar 2011 10:38:06 -0700
Subject: [Rd] Reading 64-bit integers
In-Reply-To: <A1631B10-DB4D-4914-8FCA-C1A97178F02F@r-project.org>
References: <AANLkTinndjFuoV17Th4PRUdpbrY-7RjTbyXwY6T6n+cd@mail.gmail.com>	<10962CE3-2F14-4BCE-8E78-AE18814C3F81@r-project.org>	<AANLkTimSWMqXM90Zg7WKYPbddk9zfQZhXXQif6eeLaGX@mail.gmail.com>	<5544174B-A567-4126-B99C-D39542260503@r-project.org><AANLkTinfH__3foy9oex9+Eq-84pr_i9M6PfKay8R4i9T@mail.gmail.com><4D927DAD.7070203@gmail.com>
	<A1631B10-DB4D-4914-8FCA-C1A97178F02F@r-project.org>
Message-ID: <77EB52C6DD32BA4D87471DCD70C8D70004124A5D@NA-PA-VBE03.na.tibco.com>


> -----Original Message-----
> From: r-devel-bounces at r-project.org 
> [mailto:r-devel-bounces at r-project.org] On Behalf Of Simon Urbanek
> Sent: Tuesday, March 29, 2011 6:49 PM
> To: Duncan Murdoch
> Cc: r-devel at r-project.org
> Subject: Re: [Rd] Reading 64-bit integers
> 
> 
> On Mar 29, 2011, at 8:47 PM, Duncan Murdoch wrote:
> 
> > On 29/03/2011 7:01 PM, Jon Clayden wrote:
> >> Dear Simon,
> >> 
> >> On 29 March 2011 22:40, Simon 
> Urbanek<simon.urbanek at r-project.org>  wrote:
> >>> Jon,
> >>> 
> >>> On Mar 29, 2011, at 1:33 PM, Jon Clayden wrote:
> >>> 
> >>>> Dear Simon,
> >>>> 
> >>>> Thank you for the response.
> >>>> 
> >>>> On 29 March 2011 15:06, Simon 
> Urbanek<simon.urbanek at r-project.org>  wrote:
> >>>>> 
> >>>>> On Mar 29, 2011, at 8:46 AM, Jon Clayden wrote:
> >>>>> 
> >>>>>> Dear all,
> >>>>>> 
> >>>>>> I see from some previous threads that support for 
> 64-bit integers in R
> >>>>>> may be an aim for future versions, but in the meantime 
> I'm wondering
> >>>>>> whether it is possible to read in integers of greater 
> than 32 bits at
> >>>>>> all. Judging from ?readBin, it should be possible to 
> read 8-byte
> >>>>>> integers to some degree, but it is clearly limited in 
> practice by R's
> >>>>>> internally 32-bit integer type:
> >>>>>> 
> >>>>>>> x<- as.raw(c(0,0,0,0,1,0,0,0))
> >>>>>>> (readBin(x,"integer",n=1,size=8,signed=F,endian="big"))
> >>>>>> [1] 16777216
> >>>>>>> x<- as.raw(c(0,0,0,1,0,0,0,0))
> >>>>>>> (readBin(x,"integer",n=1,size=8,signed=F,endian="big"))
> >>>>>> [1] 0
> >>>>>> 
> >>>>>> For values that fit into 32 bits it works fine, but 
> for larger values
> >>>>>> it fails. (I'm a bit surprised by the zero - should 
> the value not be
> >>>>>> NA if it is out of range?
> >>>>> 
> >>>>> No, it's not out of range - int is only 4 bytes so only 
> 4 first bytes (respecting endianness order, hence LSB) are used.
> >>>> 
> >>>> The fact remains that I ask for the value of an 8-byte 
> integer and
> >>>> don't get it.
> >>> 
> >>> I think you're misinterpreting the documentation:
> >>> 
> >>>     If 'size' is specified and not the natural size of the object,
> >>>     each element of the vector is coerced to an appropriate type
> >>>     before being written or as it is read.
> >>> 
> >>> The "integer" object type is defined as signed 32-bit in 
> R, so if you ask for "8 bytes into object type integer", you 
> get a coercion into that object type -- 32-bit signed integer 
> -- as documented. I think the issue may come from the 
> confusion of the object type "integer" with general "integer 
> number" in mathematical sense that has no representation 
> restrictions. (FWIW in C the "integer" type is "int" and it 
> is 32-bit on all modern OSes regardless of platform - that's 
> where the limitation comes from, it's not something R has made up).
> >> 
> >> OK, but it still seems like there is a case for raising a 
> warning. As
> >> it is there is no way to tell when reading an 8-byte integer from a
> >> file whether its value is really 0, or if it merely has 0 in its
> >> least-significant 4 bytes. If 99% of such stored numbers are below
> >> 2^31, one is going to need some extra logic to catch the other 1%
> >> where you (silently) get the wrong value. In essence, unless you're
> >> certain that you will never come across a number that actually uses
> >> the upper 4 bytes, you will always have to read it as two 4-byte
> >> numbers and check that the high-order one (which is endianness
> >> dependent, of course) is zero. A C-level sanity check seems more
> >> efficient and more helpful to me.
> > 
> > Seems to me that the S-PLUS solution (output="double") 
> would be a lot more useful.  I'd commit that if you write it; 
> I don't think I'd commit the warning.
> > 
> 
> I was going to write some thing similar (idea = good, patch 
> welcome ;)). My only worry is that the "output" argument is a 
> bit misleading in that one could expect to use any 
> combination of "input"/"output" which may be a maintenance 
> nightmare. If I understand it correctly it's only a special 
> case for integer input. I don't have S+ so can't say how they 
> deal with that.

In S+'s readBin the output argument can be
only double() or single() when what is double()
or single() (S+ still  has a real single
precision storage mode) and can be any
numeric type or logical when what is integer().

The output=double() seemed like the only useful case.

It does not warn when precision is lost in the 8-byte
integer to double conversion.  Perhaps it should.

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com  

> 
> Cheers,
> Simon
> 
> 
> > 
> >> 
> >>>> Pretending that it's really only four bytes because of
> >>>> the limits of R's integer type isn't all that helpful. Perhaps a
> >>>> warning should be put out if the cast will affect the 
> value of the
> >>>> result? It looks like the relevant lines in 
> src/main/connections.c are
> >>>> 3689-3697 in the current alpha:
> >>>> 
> >>>> #if SIZEOF_LONG == 8
> >>>>                   case sizeof(long):
> >>>>                       INTEGER(ans)[i] = (int)*((long *)buf);
> >>>>                       break;
> >>>> #elif SIZEOF_LONG_LONG == 8
> >>>>                   case sizeof(_lli_t):
> >>>>                       INTEGER(ans)[i] = (int)*((_lli_t *)buf);
> >>>>                       break;
> >>>> #endif
> >>>> 
> >>>>>> ) The value can be represented as a double,
> >>>>>> though:
> >>>>>> 
> >>>>>>> 4294967296
> >>>>>> [1] 4294967296
> >>>>>> 
> >>>>>> I wouldn't expect readBin() to return a double if an 
> integer was
> >>>>>> requested, but is there any way to get the correct 
> value out of it?
> >>>>> 
> >>>>> Trivially (for your unsigned big-endian case):
> >>>>> 
> >>>>> y<- readBin(x, "integer", n=length(x)/4L, endian="big")
> >>>>> y<- ifelse(y<  0, 2^32 + y, y)
> >>>>> i<- seq(1,length(y),2)
> >>>>> y<- y[i] * 2^32 + y[i + 1L]
> >>>> 
> >>>> Thanks for the code, but I'm not sure I would call that trivial,
> >>>> especially if one needs to cater for little endian and 
> signed cases as
> >>>> well!
> >>> 
> >>> I was saying for your case and it's trivial as in read as 
> integers, convert to double precision and add.
> >>> 
> >>> 
> >>>> This is what I meant by reconstructing the number manually...
> >>>> 
> >>> 
> >>> You didn't say so - you were talking about reconstructing 
> it from a raw vector which seems a lot more painful since you 
> can't compute with enough precision on raw vectors.
> >> 
> >> True - I should have been more specific. Sorry.
> >> 
> >> Jon
> >> 
> >> ______________________________________________
> >> R-devel at r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-devel
> > 
> > 
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
> 



[Rd] Reading 64-bit integers
William Dunlap