From p.dalgaard at biostat.ku.dk  Fri Apr  9 20:37:23 2004
From: p.dalgaard at biostat.ku.dk (Peter Dalgaard)
Date: Fri Apr  9 20:40:02 2004
Subject: [Rd] Incorrect handling of NA's in cor() (PR#6750)
In-Reply-To: <20040409180817.D6C275E197@biostat.mgh.harvard.edu>
References: <20040409172243.B026310476@slim.kubism.ku.dk>
	<Pine.A41.4.58.0404091032330.61772@homer32.u.washington.edu>
	<20040409180817.D6C275E197@biostat.mgh.harvard.edu>
Message-ID: <x2vfk9q9p8.fsf@biostat.ku.dk>

Marek Ancukiewicz <msa@biostat.mgh.harvard.edu> writes:

> Dear Thomas,
> 
> The question becomes: how do we rank missing values?  In
> version 1.8.1 at least, cor () uses default handling of
> missing values by rank() [by na.last parameter], that is
> missing values are assigned the highest rank. However, if
> nothing is known about the meaning of NA what would be the
> basis of such an assumption?  Assigning the NAs highest,
> lowest values, or any other values requires some additional
> information.
> 
> It seems that the default handling on missing values should be
> to assign them missing ranks: within cor(), rank() should be
> called with na.last="keep". 

Yes, and that is what 1.9.0beta is doing (it's not like this issue
hasn't been brought up before, just that the fix didn't quite fix it).
I think what we have now is still buggy, but at least it isn't biasing
rho towards +1 whenever x and y tend to be both missing at the same
time.

It's fairly easy to do something more sensible in the complete.cases
case, but getting pairwise.complete.cases right is tricky. 1.9.0
is in deep code freeze, so I don't think we should change things at
this point, except perhaps add a note to the help page.

-- 
   O__  ---- Peter Dalgaard             Blegdamsvej 3  
  c/ /'_ --- Dept. of Biostatistics     2200 Cph. N   
 (*) \(*) -- University of Copenhagen   Denmark      Ph: (+45) 35327918
~~~~~~~~~~ - (p.dalgaard@biostat.ku.dk)             FAX: (+45) 35327907


[Rd] Incorrect handling of NA's in cor() (PR#6750)
Peter Dalgaard