From simon.urbanek at r-project.org  Sat Apr 19 19:06:15 2014
From: simon.urbanek at r-project.org (Simon Urbanek)
Date: Sat, 19 Apr 2014 13:06:15 -0400
Subject: [Rd] type.convert and doubles
In-Reply-To: <21330.29530.72292.435730@stat.math.ethz.ch>
References: <3736EC2D-7360-4AF6-BF96-5A2DF3D3EF14@warnes.net>
	<326FE12B-D6FE-471B-9E0C-10BEC0584DB7@r-project.org>
	<534860CC.2090206@gmail.com>
	<17B09E7789D3104E8F5EEB0582A8D66F0129CED9FD8C@MSGRTPCCRF2WIN.DMN1.FMR.COM>
	<CAECWziJq92keAD2sHHE7ky6CHhqatrBOONq_Vf18AHJ8GActZw@mail.gmail.com>
	<CAP01uRmdnTUgDH=sNryx9=w5Rtmw3b+8KiZOWPg48r6rc86ssQ@mail.gmail.com>
	<CAECWziKLBRjRNpneVJ2hNbTG3jF2ec6GeF9m+1S092WoufCdiQ@mail.gmail.com>
	<17B09E7789D3104E8F5EEB0582A8D66F0129CEDA0169@MSGRTPCCRF2WIN.DMN1.FMR.COM>
	<21330.29530.72292.435730@stat.math.ethz.ch>
Message-ID: <A9466CB2-2B20-413D-986C-7EEC93CEC190@r-project.org>

On Apr 19, 2014, at 9:00 AM, Martin Maechler <maechler at stat.math.ethz.ch> wrote:

>>>>>> McGehee, Robert <Robert.McGehee at geodecapital.com>
>>>>>>    on Thu, 17 Apr 2014 19:15:47 -0400 writes:
> 
>>> This is all application specific and
>>> sort of beyond the scope of type.convert(), which now behaves as it
>>> has been documented to behave.
> 
>> That's only a true statement because the documentation was changed to reflect the new behavior! The new feature in type.convert certainly does not behave according to the documentation as of R 3.0.3. Here's a snippit:
> 
>> The first type that can accept all the
>> non-missing values is chosen (numeric and complex return values
>> will represented approximately, of course).
> 
>> The key phrase is in parentheses, which reminds the user to expect a possible loss of precision. That important parenthetical was removed from the documentation in R 3.1.0 (among other changes).
> 
>> Putting aside the fact that this introduces a large amount of unnecessary work rewriting SQL / data import code, SQL packages, my biggest conceptual problem is that I can no longer rely on a particular function call returning a particular class. In my example querying stock prices, about 5% of prices came back as factors and the remaining 95% as numeric, so we had random errors popping in throughout the morning.
> 
>> Here's a short example showing us how the new behavior can be unreliable. I pass a character representation of a uniformly distributed random variable to type.convert. 90% of the time it is converted to "numeric" and 10% it is a "factor" (in R 3.1.0). In the 10% of cases in which type.convert converts to a factor the leading non-zero digit is always a 9. So if you were expecting a numeric value, then 1 in 10 times you may have a bug in your code that didn't exist before.
> 
>>> options(digits=16)
>>> cl <- NULL; for (i in 1:10000) cl[i] <- class(type.convert(format(runif(1))))
>>> table(cl)
>> cl
>> factor numeric
>> 990    9010
> 
> Yes.  
> 
> Murray's point is valid, too.
> 
> But in my view, with the reasoning we have seen here,
> *and* with the well known software design principle of
> "least surprise" in mind,
> I also do think that the default for type.convert() should be what
> it has been for > 10 years now.
> 

I think there should be two separate discussions:

a) have an option (argument to type.convert and possibly read.table) to enable/disable this behavior. I'm strongly in favor of this.

b) decide what the default for a) will be. I have no strong opinion, I can see arguments in both directions

But most importantly I think a) is better than the status quo - even if the discussion about b) drags out.

Cheers,
Simon



[Rd] type.convert and doubles
Simon Urbanek