From h@wickh@m @ending from gm@il@com  Tue May  8 22:47:57 2018
From: h@wickh@m @ending from gm@il@com (Hadley Wickham)
Date: Tue, 8 May 2018 13:47:57 -0700
Subject: [Rd] 
 download.file does not process gz files correctly (truncates them?)
In-Reply-To: <CABdHhvE-H68updR0gs49h4bLYVW_WT6e+An41WmHKCP=-jVWtg@mail.gmail.com>
References: <CAO1zAVbXOY-8PSCeHj5DRGj=MOoL5OmMX-ZcdY-P-yU6yVXVXw@mail.gmail.com>
 <CAFDcVCTcwYdpBayHAUuvncZ00RWTHtLGujf6-8_EUOLanQtOuQ@mail.gmail.com>
 <CABFfbXvBWLiFjCbpGtDN_HwRmYmh5Hpy6SFCfaS1meQmwC5hyw@mail.gmail.com>
 <CAO1zAVaGZYKFt12MhvUJFG_UjzkFnTToJjD2S6Mkpq=nLwdjAg@mail.gmail.com>
 <CAFDcVCSqj1woeDOdJmgP2rRwEpgTk0ZagFrHAUNkwRvk1cXnKQ@mail.gmail.com>
 <0a82df6b-556f-41ae-101f-8a95eaa64f75@gmail.com>
 <CABdHhvE-H68updR0gs49h4bLYVW_WT6e+An41WmHKCP=-jVWtg@mail.gmail.com>
Message-ID: <CABdHhvHpdywicYS17PaPho46V12F4SGFKVB0aRZuH4WbKTe57A@mail.gmail.com>

On Tue, May 8, 2018 at 8:15 AM, Hadley Wickham <h.wickham at gmail.com> wrote:
> On Thu, May 3, 2018 at 11:34 PM, Tomas Kalibera
> <tomas.kalibera at gmail.com> wrote:
>> On 05/03/2018 11:14 PM, Henrik Bengtsson wrote:
>>>
>>> Also, as mentioned in my
>>> https://stat.ethz.ch/pipermail/r-devel/2012-August/064739.html, when
>>> not specifying the mode argument, the default on Windows is mode = "w"
>>> *except* for certain, case-sensitive, filename extensions:
>>>
>>>      if(missing(mode) && length(grep("\\.(gz|bz2|xz|tgz|zip|rda|RData)$",
>>> url)))
>>>          mode <- "wb"
>>>
>>> Just like the need for mode = "wb" on Windows, the above
>>> special-file-extension-hack is only happening on Windows, and is only
>>> documented in ?download.file if you're on Windows; so someone who's on
>>> Linux/macOS trying to help someone on Windows may not be aware of
>>> this. This adds to even more confusions, e.g. "works for me".
>>
>> If we were designing the API today, it would probably make more sense not to
>> convert any line endings by default. Today's editors _usually_ can cope with
>> different line endings and it is probably easier to detect that a text file
>> has incorrect line endings rather than detecting that a binary file has been
>> corrupted by an attempt to convert line endings. But whether to change
>> existing, documented behavior is a different question. In order to help
>> users and programmers who do not read the documentation carefully we would
>> create problems for users and programmers who do. The current heuristic/hack
>> is in line with the compatibility approach: it detects files that are
>> obviously binary, so it changes the default behavior only for cases when it
>> would obviously cause damage.
>
> From a purely utilitarian standpoint, there are far more users who do
> not carefully read the documentation than users who do ;)
>
> (I'd also argue that basing the decision on the file extension is
> suboptimal, and it would be better to use the mime type if provided by
> the server)

Also note that MS just announced support for unix line endings in notepad

https://blogs.msdn.microsoft.com/commandline/2018/05/08/extended-eol-in-notepad/

Hadley

-- 
http://hadley.nz



[Rd] download.file does not process gz files correctly (truncates them?)
Hadley Wickham