From tom@@@k@liber@ @ending from gm@il@com  Thu Oct  4 18:12:26 2018
From: tom@@@k@liber@ @ending from gm@il@com (Tomas Kalibera)
Date: Thu, 4 Oct 2018 18:12:26 +0200
Subject: [Rd] 
 segfault issue with parallel::mclapply and download.file() on
 Mac OS X
In-Reply-To: <CAB75gRUv1LitTgWSbMFunqp7P9wfWG_BJ_xYBPN4K5KJkDu+_g@mail.gmail.com>
References: <CAB75gRUv1LitTgWSbMFunqp7P9wfWG_BJ_xYBPN4K5KJkDu+_g@mail.gmail.com>
Message-ID: <f4b472d8-95d1-dcbe-5889-f88f31077f9d@gmail.com>


Thanks for the report, but unfortunately I cannot reproduce on my system 
(either macOS nor Linux, from the command line) to debug. Did you run 
this in the command line version of R?

I would not be surprised to see such a crash if executed from a 
multi-threaded application, say from some GUI or frontend that runs 
multiple threads, or from some other R session where a third party 
library (curl?) already started some threads. In such situations 
mcfork/mclapply is unsafe (?mcfork warns against GUI and frontends and 
I've now expanded slightly) and and it could not be fixed without being 
turned into something like parLapply(). parLapply() on a non-FORK 
cluster should work fine even with such applications.

Best
Tomas

On 09/19/2018 11:19 PM, Seth Russell wrote:
> I have an lapply function call that I want to parallelize. Below is a very
> simplified version of the code:
>
> url_base <- "https://cloud.r-project.org/src/contrib/"
> files <- c("A3_1.0.0.tar.gz", "ABC.RAP_0.9.0.tar.gz")
> res <- parallel::mclapply(files, function(s) download.file(paste0(url_base,
> s), s))
>
> Instead of download a couple of files in parallel, I get a segfault per
> process with a 'memory not mapped' message. I've been working with Henrik
> Bengtsson on resolving this issue and he recommended I send a message to
> the R-Devel mailing list.
>
> Here's the output:
>
> trying URL 'https://cloud.r-project.org/src/contrib/A3_1.0.0.tar.gz'
> trying URL 'https://cloud.r-project.org/src/contrib/ABC.RAP_0.9.0.tar.gz'
>
>   *** caught segfault ***
> address 0x11575ba3a, cause 'memory not mapped'
>
>   *** caught segfault ***
> address 0x11575ba3a, cause 'memory not mapped'
>
> Traceback:
>   1: download.file(paste0(url_base, s), s)
>   2: FUN(X[[i]], ...)
>   3: lapply(X = S, FUN = FUN, ...)
>   4: doTryCatch(return(expr), name, parentenv, handler)
>   5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
>   6: tryCatchList(expr, classes, parentenv, handlers)
>   7: tryCatch(expr, error = function(e) {    call <- conditionCall(e)    if
> (!is.null(call)) {        if (identical(call[[1L]], quote(doTryCatch)))
>          call <- sys.call(-4L)        dcall <- deparse(call)[1L]
>   prefix <- paste("Error in", dcall, ": ")
>          LONG <- 75LTraceback:
>          sm <- strsplit(conditionMessage(e), "\n")[[1L]] 1:         w <- 14L
> + nchar(dcall, type = "w") + nchar(sm[1L], type = "w")        if (is.na(w))
> download.file(paste0(url_base, s), s)            w <- 14L + nchar(dcall,
> type = "b") + nchar(sm[1L],
>                  type = "b")        if (w > LONG)  2: FUN(X[[i]], ...)
>   3: lapply(X = S, FUN = FUN, ...)
>   4: doTryCatch(return(expr), name, parentenv, handler)
>   5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
>   6:             prefix <- paste0(prefix, "\n  ")tryCatchList(expr, classes,
> parentenv, handlers)
>      }    else prefix <- "Error : " 7:     msg <- paste0(prefix,
> conditionMessage(e), "\n")tryCatch(expr, error = function(e) {
>   .Internal(seterrmessage(msg[1L]))    call <- conditionCall(e)    if
> (!silent && isTRUE(getOption("show.error.messages"))) {    if
> (!is.null(call)) {        cat(msg, file = outFile)        if
> (identical(call[[1L]], quote(doTryCatch)))
> .Internal(printDeferredWarnings())            call <- sys.call(-4L)    }
>       dcall <- deparse(call)[1L]    invisible(structure(msg, class =
> "try-error", condition = e))        prefix <- paste("Error in", dcall, ":
> ")})        LONG <- 75L        sm <- strsplit(conditionMessage(e),
> "\n")[[1L]]
>          w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w")
>     if (is.na(w))  8:             w <- 14L + nchar(dcall, type = "b") +
> nchar(sm[1L], try(lapply(X = S, FUN = FUN, ...), silent = TRUE)
>     type = "b")
>          if (w > LONG)             prefix <- paste0(prefix, "\n  ") 9:
> }sendMaster(try(lapply(X = S, FUN = FUN, ...), silent = TRUE))    else
> prefix <- "Error : "
>      msg <- paste0(prefix, conditionMessage(e), "\n")
>   .Internal(seterrmessage(msg[1L]))10:     if (!silent &&
> isTRUE(getOption("show.error.messages"))) {FUN(X[[i]], ...)        cat(msg,
> file = outFile)
>          .Internal(printDeferredWarnings())    }11:
> invisible(structure(msg, class = "try-error", condition =
> e))lapply(seq_len(cores), inner.do)})
>
> 12:  8: parallel::mclapply(files, function(s)
> download.file(paste0(url_base, try(lapply(X = S, FUN = FUN, ...), silent =
> TRUE)    s), s))
>
>   9:
> sendMaster(try(lapply(X = S, FUN = FUN, ...), silent = TRUE))Possible
> actions:
>
> 1: abort (with core dump, if enabled)
> 2: normal R exit
> 10: 3: exit R without saving workspace
> FUN(X[[i]], ...)4: exit R saving workspace
>
> 11: lapply(seq_len(cores), inner.do)
> 12: parallel::mclapply(files, function(s) download.file(paste0(url_base,
>    s), s))
>
> Here's my sessionInfo()
>
>> sessionInfo()
> R version 3.5.1 (2018-07-02)
> Platform: x86_64-apple-darwin16.7.0 (64-bit)
> Running under: macOS Sierra 10.12.6
>
> Matrix products: default
> BLAS/LAPACK: /usr/local/Cellar/openblas/0.3.3/lib/libopenblasp-r0.3.3.dylib
>
> locale:
> [1] en_US/en_US/en_US/C/en_US/en_US
>
> attached base packages:
> [1] parallel  stats     graphics  grDevices utils     datasets  methods
> [8] base
>
> loaded via a namespace (and not attached):
> [1] compiler_3.5.1
>
> My version of R I'm running was installed via homebrew with "brew install r
> --with-java --with-openblas"
>
> Also, the provided example code works as expected on Linux. Also, if I
> provide a non-default download method to the download.file() call such as:
>
> res <- parallel::mclapply(files, function(s) download.file(paste0(url_base,
> s), s, method="wget"))
> res <- parallel::mclapply(files, function(s) download.file(paste0(url_base,
> s), s, method="curl"))
>
> It works correctly - no segfault. If I use method="libcurl" it does
> segfault.
>
> I'm not sure what steps to take to further narrow down the source of the
> error.
>
> Is this a known bug? if not, is this a new bug or an unexpected feature?
>
> Thanks,
> Seth
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel



[Rd] segfault issue with parallel::mclapply and download.file() on Mac OS X
Tomas Kalibera