From maechler at stat.math.ethz.ch  Tue Jun 27 17:36:54 2017
From: maechler at stat.math.ethz.ch (Martin Maechler)
Date: Tue, 27 Jun 2017 17:36:54 +0200
Subject: [Rd] Windows iconv() "failure" in certain locales
In-Reply-To: <22866.13492.183329.958202@stat.math.ethz.ch>
References: <1d01ccce-cb18-30d9-9d93-644ed4dbd995@uw.edu>
 <70548b39-a0ee-80d5-302f-2433a700192d@gmail.com>
 <6b7f9592-1a53-37dd-f8ae-fe805dcbc337@uw.edu>
 <7e843534-d9f2-9cac-1c53-7c2e8b687c39@gmail.com>
 <CA+E84Q4SKEHOA7Gy_wCZs_9twTmzE-M=EeUDABshmREyAutyNg@mail.gmail.com>
 <08f33a36-329c-b40a-202a-80f2e61f53a3@gmail.com>
 <SN1PR21MB0061EB36C510723541CC25B7B7D80@SN1PR21MB0061.namprd21.prod.outlook.com>
 <CY4PR21MB04713D49F2DB7C9BFC2E2A56DED90@CY4PR21MB0471.namprd21.prod.outlook.com>
 <SN1PR21MB0061957F2745E5277E239C69B7DF0@SN1PR21MB0061.namprd21.prod.outlook.com>
 <22866.13492.183329.958202@stat.math.ethz.ch>
Message-ID: <22866.31638.949797.725782@stat.math.ethz.ch>

This is a continuation of the R-devel thread with subject
 "suggestion to fix packageDescription() for Windows users" :

As I said there, a patch should rather address the underlying
problem in packageDescription rather than a kludgy workaround
patch for  citation().
(For that same reason, Ben Marwick proposed to fix
 packageDescription() rather than the symptom seen in citation().)

It's not hard to see that the problem is that  iconv() in
Windows does not always succeed to translate from "UTF-8" to the
"current locale", in the case mentioned there.

I'm giving some easier reproducible examples:  no need to install
half of tidyverse just to get citation("readr") :

> x <- c("Ekstr\xf8m", "J\xf6reskog", "bi\xdfchen Z\xfcrcher")
> Encoding(x1) <- "latin1"
> xU <- iconv(x1, "latin1", "UTF-8")

> Sys.setlocale("LC_CTYPE", "Chinese")
[1] "Chinese (Simplified)_People's Republic of China.936"
> 
> iconv(x1, "latin1", "") # NA NA NA
[1] NA NA NA
> iconv(xU, "UTF-8", "") # NA NA NA
[1] NA NA NA
> iconv(xU, "UTF-8", "//TRANSLIT")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(xU, "UTF-8", "", sub = "byte")
[1] "Ekstr<c3><b8>m"         "J<c3><b6>reskog"        "bi<c3><9f>chen Z??rcher"


> Sys.setlocale("LC_CTYPE", "Arabic")
[1] "Arabic_Saudi Arabia.1256"
> iconv(x1, "latin1", "")  # NA NA NA
[1] NA NA NA
> iconv(xU, "UTF-8", "")  # NA NA NA
[1] NA NA NA
> iconv(xU, "UTF-8", "//TRANSLIT")
[1] "Ekstr\370m"         "J\366reskog"        "bi?chen Z?rcher"
> iconv(xU, "UTF-8", "", sub="byte")
[1] "Ekstr<c3><b8>m"         "J<c3><b6>reskog"        "bi<c3><9f>chen Z?rcher"
> iconv(xU, "UTF-8", "", sub="?")
[1] "Ekstr??m"         "J??reskog"        "bi??chen Z?rcher"

Etc... .  As the above is typically garbled between e-mail
transfer agents, I append both the iconv-Windows.R R script and
the corresponding iconv-Windows.Rout  R transcript to this
e-mail (using MIME type text/plain (easy using emacs for mail..)),
and they contain a bit more than the above.

Note that the above shows that using 'sub = *' and using
"//TRANSLIT" in case of a previous NA  result helps quite a bit,
in the sense that it gives much more information to see
  "J?reskog"  instead   NA.

I'm considering updating  packageDescription() to try these in
case it first returns NA.   This would make the citation() hack
unnecessary.

Martin

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: iconv-Windows.R
URL: <https://stat.ethz.ch/pipermail/r-devel/attachments/20170627/1e5f6924/attachment.ksh>
-------------- next part --------------

R Under development (unstable) (2017-06-25 r72854) -- "Unsuffered Consequences"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-w64-mingw32/x64 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> #### iconv() behavior depending on Locales  LC_CTYPE  in Windows
> #### =======                       ==============================
> ###
> ### In a *shell* in Windows (emacs), after doing R.home() in R, use that to do something like
> ###   c:/PROGRA~1/R/R-devel/bin/R CMD BATCH iconv-Windows.R
> ###   ^^^^^^^^^^^^^^^^^^^^^^^^^^= === ===== ===============  ==> producing  iconv-Windows.Rout
> ###
> sessionInfo() ## does not matter so much
R Under development (unstable) (2017-06-25 r72854)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows Server 2008 R2 x64 (build 7601) Service Pack 1

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 
[2] LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.5.0
> ## -- should be Windows to exhibit the problems
> 
> ## From  help(iconv) 's  example : Using "latin1" European language letters:
> x1 <- c("Ekstr\xf8m", "J\xf6reskog", "bi\xdfchen Z\xfcrcher")
> Encoding(x1) <- "latin1"
> xU <- iconv(x1, "latin1", "UTF-8")
> 
> 
> ## 2 locales that do not work well : ---------------------------------
> Sys.setlocale("LC_CTYPE", "Chinese")
[1] "Chinese (Simplified)_People's Republic of China.936"
> 
> iconv(x1, "latin1", "") # NA NA NA
[1] NA NA NA
> iconv(x1, "latin1", "//TRANSLIT") # perfect for Chinese
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(x1, "latin1", "", sub = "byte")
[1] "Ekstr<f8>m"         "J<f6>reskog"        "bi<df>chen Z??rcher"
> iconv(xU, "UTF-8", "") # NA NA NA
[1] NA NA NA
> iconv(xU, "UTF-8", "//TRANSLIT")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(xU, "UTF-8", "", sub = "byte")
[1] "Ekstr<c3><b8>m"         "J<c3><b6>reskog"        "bi<c3><9f>chen Z??rcher"
> ##--
> Sys.setlocale("LC_CTYPE", "Arabic")
[1] "Arabic_Saudi Arabia.1256"
> iconv(x1, "latin1", "")  # NA NA NA
[1] NA NA NA
> iconv(x1, "latin1", "//TRANSLIT") # not bad, but not perfect
[1] "Ekstr\370m"         "J\366reskog"        "bi?chen Z?rcher"
> iconv(x1, "latin1", "", sub="byte")
[1] "Ekstr<f8>m"         "J<f6>reskog"        "bi<df>chen Z?rcher"
> iconv(x1, "latin1", "", sub="?")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(xU, "UTF-8", "")  # NA NA NA
[1] NA NA NA
> iconv(xU, "UTF-8", "//TRANSLIT")
[1] "Ekstr\370m"         "J\366reskog"        "bi?chen Z?rcher"
> iconv(xU, "UTF-8", "", sub="byte")
[1] "Ekstr<c3><b8>m"         "J<c3><b6>reskog"        "bi<c3><9f>chen Z?rcher"
> iconv(xU, "UTF-8", "", sub="?")
[1] "Ekstr??m"         "J??reskog"        "bi??chen Z?rcher"
> 
> ## 2 locales that work well for these examples (no wonder) -----------
> 
> Sys.setlocale("LC_CTYPE", "German_Switzerland")
[1] "German_Switzerland.1252"
> iconv(x1, "latin1", "")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(x1, "latin1", "//TRANSLIT")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(x1, "latin1", "", sub="?")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(xU, "UTF-8", "")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(xU, "UTF-8", "//TRANSLIT")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(xU, "UTF-8", "", sub="?")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> ##--
> Sys.setlocale("LC_CTYPE", "English")
[1] "English_United States.1252"
> iconv(x1, "latin1", "")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(x1, "latin1", "//TRANSLIT")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(x1, "latin1", "", sub="?")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(xU, "UTF-8", "")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(xU, "UTF-8", "//TRANSLIT")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> iconv(xU, "UTF-8", "", sub="?")
[1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"
> 
> proc.time()
   user  system elapsed 
   0.18    0.14    0.98 


[Rd] Windows iconv() "failure" in certain locales
Martin Maechler