From simon.urbanek at r-project.org  Fri Jan 20 02:58:54 2012
From: simon.urbanek at r-project.org (Simon Urbanek)
Date: Thu, 19 Jan 2012 20:58:54 -0500
Subject: [Rd] use of UTF-8 \uxxxx escape sequences in function arguments
In-Reply-To: <201201200039.18352.thomas@zumbrunn.name>
References: <201201182354.43607.thomas@zumbrunn.name>
	<2855A65F-5760-4076-BF1D-E80FEDCA0DB0@gmail.com>
	<201201200039.18352.thomas@zumbrunn.name>
Message-ID: <0D9CB9D9-E945-4761-BDA4-AC347E9F70C3@r-project.org>


On Jan 19, 2012, at 6:39 PM, Thomas Zumbrunn wrote:

> On Thursday 19 January 2012, peter dalgaard wrote:
>> On Jan 18, 2012, at 23:54 , Thomas Zumbrunn wrote:
>>>  plain("Z?rich")  ## works
>>>  plain("Z\u00BCrich")  ## fails
>>>  escaped("Z?rich")  ## fails
>>>  escaped("Z\u00BCrich")  ## works
>> 
>> Using the correct UTF-8 code helps quite a bit:
>> 
>> U+00BC	?	c2 bc	VULGAR FRACTION ONE QUARTER
>> U+00FC	?	c3 bc	LATIN SMALL LETTER U WITH DIAERESIS
> 
> Thank you for pointing that out. How embarrassing - I systematically used the 
> wrong representations. Even worse, I didn't carefully read "Writing R 
> Extensions" which speaks of "Unicode as \uxxxx escapes" rather than "UTF-8 as 
> \uxxxx escapes", so e.g. looking up the UTF-16 byte representations would have 
> done the trick.
> 
> I didn't find a recommended method of replacing non-ASCII characters with 
> Unicode \uxxxx escape sequences and ended up using the Unix command line tool 
> "iconv". However, the iconv version installed on my GNU/Linux machine 
> (openSUSE 11.4) seems to be outdated and doesn't support the very useful "--
> unicode-subst" option yet. I installed "libiconv" from 
> http://www.gnu.org/software/libiconv/, and now I can easily replace all non-
> ASCII characters in my UTF-8 encoded R files with:
> 
>  iconv -f UTF-8 -t ASCII --unicode-subst="\u%04X" my-utf-8-encoded-file.R
> 


You can actually do that with R alone:

## you'll have to make sure that you're in C locale so R does the conversion for you
Sys.setlocale(,"C")

utf8conv <- function(conn) gsub("<U\\+([0-9A-F]{4})>","\\\\u\\1",capture.output(writeLines(readLines(conn,encoding="UTF-8"))))

> writeLines(utf8conv("test.txt"))
M\u00F6gliche L\u00F6sung
ne nebezpe\u010Dn\u00E9

Cheers,
Simon



[Rd] use of UTF-8 \uxxxx escape sequences in function arguments
Simon Urbanek