From jeroenoom@ @end|ng |rom gm@||@com  Wed Apr 10 13:14:25 2019
From: jeroenoom@ @end|ng |rom gm@||@com (Jeroen Ooms)
Date: Wed, 10 Apr 2019 13:14:25 +0200
Subject: [Rd] 
 R 3.5.3 and 3.6.0 alpha Windows bug: UTF-8 characters in code
 are simplified to wrong ones
In-Reply-To: <CAAg2TgJboQWc3QAeR_snGEMxkmaTbkf5B331qtkvgn_BO2w1nA@mail.gmail.com>
References: <CAAg2TgJboQWc3QAeR_snGEMxkmaTbkf5B331qtkvgn_BO2w1nA@mail.gmail.com>
Message-ID: <CABFfbXuciaNVbyufSjXxAXs4V2jbZZxsbaChEBmzaPWOzCc6yg@mail.gmail.com>

On Wed, Apr 10, 2019 at 12:19 PM Tom?? Bo?il <borilt at gmail.com> wrote:
>
> Minimalistic example:
> Let's type "?" (LATIN SMALL LETTER R WITH CARON) in RGui console:
> > "?"
> [1] "r"
>
> Although the script is in UTF-8, the characters are replaced by
> "simplified" substitutes uncontrollably (depending on OS locale). The
> same goes with simply entering the code statements in R Console.
>
> The problem does not occur on OS with UTF-8 locale (Mac OS, Linux...)

I think this is a "feature" of win_iconv that is bundled with base R
on Windows (./src/extra/win_iconv). The character from your example is
not part of the latin1 (iso-8859-1) set, however, win-iconv seems to
do so anyway:

> x <- "\U00159"
> print(x)
[1] "?"
> iconv(x, 'UTF-8', 'iso-8859-1')
[1] "r"

On MacOS, iconv tells us this character cannot be represented as latin1:

> x <- "\U00159"
> print(x)
[1] "?"
> iconv(x, 'UTF-8', 'iso-8859-1')
[1] NA

I'm actually not sure why base-R needs win_iconv (but I'm not an
encoding expert at all). Perhaps we could try to unbundle it and use
the standard libiconv provided by the Rtools toolchain bundle to get
more consistent results.



[Rd] R 3.5.3 and 3.6.0 alpha Windows bug: UTF-8 characters in code are simplified to wrong ones
Jeroen Ooms