From pdalgd at gmail.com  Sun Nov 23 10:39:09 2014
From: pdalgd at gmail.com (peter dalgaard)
Date: Sun, 23 Nov 2014 10:39:09 +0100
Subject: [Rd] R string comparisons may vary with platform (plain text)
In-Reply-To: <CAFDcVCQeXKWL87wOzpa=7pX1nMe91b73VxOC5=3Fby9M8U+2Hg@mail.gmail.com>
References: <D0962B34.6441%sambler@linkedin.com> <5470F51F.1000802@gmail.com>
	<CAFDcVCQeXKWL87wOzpa=7pX1nMe91b73VxOC5=3Fby9M8U+2Hg@mail.gmail.com>
Message-ID: <26A0DFF7-0E3C-4062-AE73-F7C0052E3380@gmail.com>


> On 23 Nov 2014, at 01:05 , Henrik Bengtsson <hb at biostat.ucsf.edu> wrote:
> 
> On Sat, Nov 22, 2014 at 12:42 PM, Duncan Murdoch
> <murdoch.duncan at gmail.com> wrote:
>> On 22/11/2014, 2:59 PM, Stuart Ambler wrote:
>>> A colleague?s R program behaved differently when I ran it, and we thought
>>> we traced it probably to different results from string comparisons as
>>> below, with different R versions.  However the platforms also differed.  A
>>> friend ran it on a few machines and found that the comparison behavior
>>> didn?t correlate with R version, but rather with platform.
>>> 
>>> I wonder if you?ve seen this.  If it?s not some setting I?m unaware of,
>>> maybe someone should look into it.  Sorry I haven?t taken the time to read
>>> the source code myself.
>> 
>> Looks like a collation order issue.  See ?Comparison.
> 
> With the oddity that both platforms use what look like similar locales:
> 
> LC_COLLATE=en_US.UTF-8
> LC_COLLATE=en_US.utf8

It's the sort of thing thay I've tried to wrap my mind around multiple times and failed, but have a look at

http://stackoverflow.com/questions/19967555/postgres-collation-differences-osx-v-ubuntu

which seems to be essentially the same issue, just for Postgres. If you have the stamina, also look into the python question that it links to.

As I understand it, there are two potential reasons: Either the two platforms are not using the same collation table for en_US, or at least one of them is not fully implementing the Unicode Collation Algorithm.

In general, collation is a minefield: Some languages have the same letters in different order (e.g. Estonian with Z between S and T); accented characters sort with the unaccented counterpart in some languages but as separate characters in others; some locales sort ABab, others AaBb, yet others aAbB; sometimes punctuation is ignored, sometimes not; sometimes multiple characters count as one, etc.

-- 
Peter Dalgaard, Professor,
Center for Statistics, Copenhagen Business School
Solbjerg Plads 3, 2000 Frederiksberg, Denmark
Phone: (+45)38153501
Email: pd.mes at cbs.dk  Priv: PDalgd at gmail.com



[Rd] R string comparisons may vary with platform (plain text)
peter dalgaard