From maechler at stat.math.ethz.ch  Thu Jun 29 12:27:51 2017
From: maechler at stat.math.ethz.ch (Martin Maechler)
Date: Thu, 29 Jun 2017 12:27:51 +0200
Subject: [Rd] Windows iconv() "failure" in certain locales
In-Reply-To: <79597dc9-db1b-cd7b-2fb2-0c236db03f48@statistik.tu-dortmund.de>
References: <1d01ccce-cb18-30d9-9d93-644ed4dbd995@uw.edu>
 <70548b39-a0ee-80d5-302f-2433a700192d@gmail.com>
 <6b7f9592-1a53-37dd-f8ae-fe805dcbc337@uw.edu>
 <7e843534-d9f2-9cac-1c53-7c2e8b687c39@gmail.com>
 <CA+E84Q4SKEHOA7Gy_wCZs_9twTmzE-M=EeUDABshmREyAutyNg@mail.gmail.com>
 <08f33a36-329c-b40a-202a-80f2e61f53a3@gmail.com>
 <SN1PR21MB0061EB36C510723541CC25B7B7D80@SN1PR21MB0061.namprd21.prod.outlook.com>
 <CY4PR21MB04713D49F2DB7C9BFC2E2A56DED90@CY4PR21MB0471.namprd21.prod.outlook.com>
 <SN1PR21MB0061957F2745E5277E239C69B7DF0@SN1PR21MB0061.namprd21.prod.outlook.com>
 <22866.13492.183329.958202@stat.math.ethz.ch>
 <22866.31638.949797.725782@stat.math.ethz.ch>
 <79597dc9-db1b-cd7b-2fb2-0c236db03f48@statistik.tu-dortmund.de>
Message-ID: <22868.54823.81543.37466@stat.math.ethz.ch>

>>>>> Uwe Ligges <ligges at statistik.tu-dortmund.de>
>>>>>     on Wed, 28 Jun 2017 18:45:59 +0200 writes:

    > On 27.06.2017 17:36, Martin Maechler wrote:
    >> This is a continuation of the R-devel thread with subject
    >> "suggestion to fix packageDescription() for Windows users" :
    >> 
    >> As I said there, a patch should rather address the underlying
    >> problem in packageDescription rather than a kludgy workaround
    >> patch for  citation().
    >> (For that same reason, Ben Marwick proposed to fix
    >> packageDescription() rather than the symptom seen in citation().)
    >> 
    >> It's not hard to see that the problem is that  iconv() in
    >> Windows does not always succeed to translate from "UTF-8" to the
    >> "current locale", in the case mentioned there.
    >> 
    >> I'm giving some easier reproducible examples:  no need to install
    >> half of tidyverse just to get citation("readr") :
    >> 
    >>> x <- c("Ekstr\xf8m", "J\xf6reskog", "bi\xdfchen Z\xfcrcher")
    >>> Encoding(x1) <- "latin1"
    >>> xU <- iconv(x1, "latin1", "UTF-8")
    >> 
    >>> Sys.setlocale("LC_CTYPE", "Chinese")
    >> [1] "Chinese (Simplified)_People's Republic of China.936"
    >>> 
    >>> iconv(x1, "latin1", "") # NA NA NA
    >> [1] NA NA NA
    >>> iconv(xU, "UTF-8", "") # NA NA NA
    >> [1] NA NA NA
    >>> iconv(xU, "UTF-8", "//TRANSLIT")
    >> [1] "Ekstr?m"         "J?reskog"        "bi?chen Z?rcher"

    > Interesting, I get chinese characters here.

For which one of the above cases; can you show them
 (it may survive E-mail servers; we had other
  Chinese R strings on R-help / R-devel recently, right?)

In any case, I think  that is even worse, isn't it?  
As also in a Chinese locale you'd want explicit-latin1 text to
see in something that looks like latin-1 (I know from a master's
 student that Windows+Chinese can well show latin-1-like
 letters also interspersed in the Chinese text),
no ? 


    > Beside the comments from Duncan Murdoch:

    > iconv(x1, "latin1", "", sub="?")
    > etc. would be an alternative in case some characters really cannot be 
    > converted into the target encoding and should perhaps be considered for 
    > the time after Duncan commits the fix for the underlying porblem.

Yes. I'd had the same idea that's why I used it in the code I
sent along.

So,

1)  we definitely won't commit the workaround patch for citation().

2) I have a "workaround patch" for packageDescription() which is
   more useful in the sense that only if iconv() produces NA's, it
   tries alternatives, notably   "//TRANSLIT",  "ASCII//TRANSLIT"
   (the latter Ben also mentioned, but my patch would only use it
    in the NA case) and also the same  'sub="?"' that you mention
    above, Uwe.

   That patch is not Windows-specific and will automatically
   also help in other cases / platforms where the iconv()
   re-encoding leads to partial NAs.
   
  @Duncan M: would you _not_ want me to commit that either?

Martin



[Rd] Windows iconv() "failure" in certain locales
Martin Maechler