From murdoch.duncan at gmail.com  Wed Jun 28 12:32:08 2017
From: murdoch.duncan at gmail.com (Duncan Murdoch)
Date: Wed, 28 Jun 2017 06:32:08 -0400
Subject: [Rd] Windows iconv() "failure" in certain locales
In-Reply-To: <22866.31638.949797.725782@stat.math.ethz.ch>
References: <1d01ccce-cb18-30d9-9d93-644ed4dbd995@uw.edu>
 <70548b39-a0ee-80d5-302f-2433a700192d@gmail.com>
 <6b7f9592-1a53-37dd-f8ae-fe805dcbc337@uw.edu>
 <7e843534-d9f2-9cac-1c53-7c2e8b687c39@gmail.com>
 <CA+E84Q4SKEHOA7Gy_wCZs_9twTmzE-M=EeUDABshmREyAutyNg@mail.gmail.com>
 <08f33a36-329c-b40a-202a-80f2e61f53a3@gmail.com>
 <SN1PR21MB0061EB36C510723541CC25B7B7D80@SN1PR21MB0061.namprd21.prod.outlook.com>
 <CY4PR21MB04713D49F2DB7C9BFC2E2A56DED90@CY4PR21MB0471.namprd21.prod.outlook.com>
 <SN1PR21MB0061957F2745E5277E239C69B7DF0@SN1PR21MB0061.namprd21.prod.outlook.com>
 <22866.13492.183329.958202@stat.math.ethz.ch>
 <22866.31638.949797.725782@stat.math.ethz.ch>
Message-ID: <7859662c-4ed0-1dd1-a11f-5c3e59055d50@gmail.com>

On 27/06/2017 11:36 AM, Martin Maechler wrote:
> This is a continuation of the R-devel thread with subject
>  "suggestion to fix packageDescription() for Windows users" :
>
> As I said there, a patch should rather address the underlying
> problem in packageDescription rather than a kludgy workaround
> patch for  citation().
> (For that same reason, Ben Marwick proposed to fix
>  packageDescription() rather than the symptom seen in citation().)
>
> It's not hard to see that the problem is that  iconv() in
> Windows does not always succeed to translate from "UTF-8" to the
> "current locale", in the case mentioned there.
>
> I'm giving some easier reproducible examples:  no need to install
> half of tidyverse just to get citation("readr") :
>
>> x <- c("Ekstr\xf8m", "J\xf6reskog", "bi\xdfchen Z\xfcrcher")
>> Encoding(x1) <- "latin1"
>> xU <- iconv(x1, "latin1", "UTF-8")
>
>> Sys.setlocale("LC_CTYPE", "Chinese")
> [1] "Chinese (Simplified)_People's Republic of China.936"
>>
>> iconv(x1, "latin1", "") # NA NA NA
> [1] NA NA NA
>> iconv(xU, "UTF-8", "") # NA NA NA
> [1] NA NA NA
>> iconv(xU, "UTF-8", "//TRANSLIT")
> [1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
>> iconv(xU, "UTF-8", "", sub = "byte")
> [1] "Ekstr<c3><b8>m"         "J<c3><b6>reskog"        "bi<c3><9f>chen Z??rcher"
>
>
>> Sys.setlocale("LC_CTYPE", "Arabic")
> [1] "Arabic_Saudi Arabia.1256"
>> iconv(x1, "latin1", "")  # NA NA NA
> [1] NA NA NA
>> iconv(xU, "UTF-8", "")  # NA NA NA
> [1] NA NA NA
>> iconv(xU, "UTF-8", "//TRANSLIT")
> [1] "Ekstr\370m"         "J\366reskog"        "bi?chen Z?rcher"
>> iconv(xU, "UTF-8", "", sub="byte")
> [1] "Ekstr<c3><b8>m"         "J<c3><b6>reskog"        "bi<c3><9f>chen Z?rcher"
>> iconv(xU, "UTF-8", "", sub="?")
> [1] "Ekstr??m"         "J??reskog"        "bi??chen Z?rcher"
>
> Etc... .  As the above is typically garbled between e-mail
> transfer agents, I append both the iconv-Windows.R R script and
> the corresponding iconv-Windows.Rout  R transcript to this
> e-mail (using MIME type text/plain (easy using emacs for mail..)),
> and they contain a bit more than the above.
>
> Note that the above shows that using 'sub = *' and using
> "//TRANSLIT" in case of a previous NA  result helps quite a bit,
> in the sense that it gives much more information to see
>   "J?reskog"  instead   NA.
>
> I'm considering updating  packageDescription() to try these in
> case it first returns NA.   This would make the citation() hack
> unnecessary.

I agree with the general sentiment (fix the underlying problem).  I 
haven't traced through this one, but the usual cause of problems like 
this is that we too frequently convert to the local encoding even when 
that loses information.

Kirill M?ller and I are gradually working through internal code and 
fixing these issues.  I don't know if this one will be fixed sooner or 
later, but I would hope it would be fixed by 3.5.0.

So in order that we don't hide it, I'd ask you not to apply the patch in 
R-devel.

Duncan Murdoch



[Rd] Windows iconv() "failure" in certain locales
Duncan Murdoch