From h.wickham at gmail.com  Thu May  2 15:46:26 2013
From: h.wickham at gmail.com (Hadley Wickham)
Date: Thu, 2 May 2013 09:46:26 -0400
Subject: [Rd] Windows, format.POSIXct and character encodings
In-Reply-To: <77A76E74-6304-4236-8609-A5E08BC5B1A7@r-project.org>
References: <CABdHhvGT1hXm694y+yRSgD2JC_+KS3eUFGCycVVcjsneUGpAGA@mail.gmail.com>
	<77A76E74-6304-4236-8609-A5E08BC5B1A7@r-project.org>
Message-ID: <CABdHhvHY_QWE3x-QBUvRXDqDLEyLM3fSR4YBoET2aSWFVxj4Bg@mail.gmail.com>

>> identical(x, y)
>> # [1] TRUE
>>
>> # But, confusingly, ...
>>
>> charToRaw(x)
>> # [1] e5 8d 88 e5 89 8d 2b 2a e5 8d 88 e5 be 8c
>>
>> charToRaw(y)
>> # [1] 8c df 91 4f 2b 2a 8c df 8c e3
>>
>
> That's not confusing at all:
>
>> Encoding(x)
> [1] "UTF-8"
>> Encoding(y)
> [1] "unknown"
>
> The first string is in UTF-8 the second is in the local locale (here 932).

It's confusing because two "identical" objects have different
behaviour. Thanks for pointing out that it's documented, but it
doesn't make it any less confusing.

>> # And this causes a problem when you attempt to do
>> # stuff with the string
>>
>> gsub("+", "*", x, fixed = T)
>> # Error in gsub("+", "*", x, fixed = T) :
>> #  invalid multibyte string at '<8c>'
>> gsub("+", "*", y, fixed = T)
>> # [1] "??**??"
>>
>
> This is where the problem lies - and it has nothing to do with format:
>
>> z=enc2utf8("??+*??")
>> gsub("+", "*", z, fixed = T)
> Error in gsub("+", "*", z, fixed = T) :
>   invalid multibyte string at '<8c>'

So is there a way I can convert x into a utf-8 string in general? i.e.
how can I this regular expression not fail given that the text is
encoded in locale 932?


Hadley

--
Chief Scientist, RStudio
http://had.co.nz/



[Rd] Windows, format.POSIXct and character encodings
Hadley Wickham