From murdoch@dunc@n @end|ng |rom gm@||@com  Mon Dec  7 11:31:32 2020
From: murdoch@dunc@n @end|ng |rom gm@||@com (Duncan Murdoch)
Date: Mon, 7 Dec 2020 05:31:32 -0500
Subject: [Rd] New pipe operator
In-Reply-To: <MEAPR01MB2407865F5622461515F409EDB4CE0@MEAPR01MB2407.ausprd01.prod.outlook.com>
References: <e55c2093-0bb5-5f9b-5521-0ca8bdf35953@gmail.com>
 <CALyqOb8c1hHOSaMV+fqXTwpeOsP--KJ_t-oeqMYYafVXeNpqZg@mail.gmail.com>
 <7586e034-e287-0970-6480-f82d747b5445@gmail.com>
 <CADfFDC7bT58_+52uvmpHuMMQRNbbmqV3_7t_fc4sgJ_Esw=xag@mail.gmail.com>
 <CAJmOi+PO_8XDC23=rT=pQ76ENL=gx2V-41ax2fHSVs87xURcEA@mail.gmail.com>
 <CAP01uRkkd3jOjvaS9JgKYCD1KA83HfzCd+MQSv_2aYJPgcOmSQ@mail.gmail.com>
 <CAP01uRkPcbCBRGwAzKM=W19Ohh-zDGwZH8gM7=A-sK2rRaQxaQ@mail.gmail.com>
 <CAD4oTHFBY4rBhiRsJUkA6jwD2YWaxPMz2OZDOqSvHfv6A7DZPg@mail.gmail.com>
 <CAP01uRmTYna1nM+ugELHrCqLjfWuWwHSfE4cvqoaq2537sHpAw@mail.gmail.com>
 <MEAPR01MB2407865F5622461515F409EDB4CE0@MEAPR01MB2407.ausprd01.prod.outlook.com>
Message-ID: <5179b8d4-628e-ff73-fa53-92fe942e12ea@gmail.com>

On 06/12/2020 8:22 p.m., Bravington, Mark (Data61, Hobart) wrote:
> Seems like this *could* be a good thing, and thanks to R core for considering it. But, FWIW:
> 
>   - I agree with Gabor G that consistency of "syntax" should be paramount here. Enough problems have been caused by earlier superficially-convenient non-standard features in R.  In particular:
> 
>   -- there should not be any discrepancy between an in-place function-definition, and a predefined function attached to a symbol (as per Gabor's point).
>   
>   -- Hence, the ability to say x |> foo  ie without parentheses, seems bound to lead to inconsistency, because x |> foo is allowed, x |> base::foo isn't allowed without tricks, but x |> function( y) foo( y) isn't... So, x |> foo is not worth keeping. Parentheses are a price well worth paying.
>   
>   -- it is still inconsistent and confusing to (apparently) invoke a function in some places--- normally--- via 'foo(x)', yet in others--- pipily--- via 'foo()'. Especially if 'foo' already has a default value for its first argument.
> 
>   - I don't see the problem with a placeholder--- doesn't it remove all ambiguity? Sure there needs to be a standard unclashable name and people can argue about what that should be, but the following seems clear and flexible... to me, anyway:
>   
>   thing |>
>     foo( _PIPE_) |>           # standard
>     bah( arg1, _PIPE_) |>   # multi-arg function
>     _ANON_({ x <- sum( _PIPE_); _PIPE_/x + x/_PIPE_ })   # anon function
>    
> where '_PIPE_' is the ordained name of the placeholder, and '_ANON_' constructs-and-calls a function with single argument '_PIPE_'. There is just one rule (I think...): each pipe-stage must be a *call* involving the argument '_PIPE_'.

I believe there's no ambiguity if the placeholder is *only* allowed in  
the RHS of a pipe expression.  I think the ambiguity arises if you allow  
the same syntax to be used to generate anonymous functions.  We can't  
use _PIPE_ as the placeholder, because it's a legal name.  But we could  
use _.  Then

   x |> (_ + 1) + mean(_)

could expand unambiguously to

   (function(_) (_  + 1) + mean(_))(x)

but

   (_ + 1) + mean(_)

shouldn't be taken to be an anonymous function declaration, otherwise  
things like

   mean(_ |> _)

do become ambiguous:  is the second placeholder the argument to the anon  
function, or is it the placeholder for the embedded pipe?

However, implementing this makes the parser pretty ugly:  its handling  
of _ depends on the outer context.  I now agree that leaving out  
placeholder syntax was the right decision.


> 
> 
>   - The proposed anonymous-function syntax looks quite ugly to me, diminishing readability and inviting errors. The new pipe symbol |> already looks scarily like quantum mechanics; adding \( just puts fishbones into the symbolic soup.
> 
>   - IMO it's not worth going too far to try to lure magritter-etc fans to swap to the new; my experience is that many people keep using older inferior R syntax for years after better replacements become available (even if they are aware of replacements), for various reasons. Just provide a good framework, and let nature take its course.
>   
>   - Disclaimer: personally I'm not much of a pipehead anyway, so maybe I'm not the audience. But if I was to consider piping, I wouldn't be very tempted by the current proposal. OTOH, I might even be tempted to write--- and use!--- my own version of '%|>%' as above (maybe someone already has). And if R did it for me, that'd be great :)

Yours would suffer one of the same problems as magrittr's:  it has the  
wrong operator precedence.  The current precedence ordering (from  
?Syntax) is, from highest to lowest:


:: :::	access variables in a namespace
$ @	component / slot extraction
[ [[	indexing
^	exponentiation (right to left)
- +	unary minus and plus
:	sequence operator
%any%	special operators (including %% and %/%)
* /	multiply, divide
+ -	(binary) add, subtract
< > <= >= == !=	ordering and comparison
!	negation
& &&	and
| ||	or
~	as in formulae
-> ->>	rightwards assignment
<- <<-	assignment (right to left)
=	assignment (right to left)
?	help (unary and binary)


The %>% operator has higher precedence than the arithmetic operators, so

x*y %>% f()

is equivalent to x*f(y), not

f(x*y)

as it should "obviously" be.  I believe the new |> operator falls  
between "| ||" and "~", so

x || y |> f()

is the same as f(x || y), and

x ~ y |> f()

is x ~ f(y).   There could be arguments about where the new one appears  
(and there probably have been), but *clearly* magrittr's precedence is  
wrong, and yours would be too, because they are both fixed at the quite  
high precedence given to %any%.

Duncan Murdoch

>   
> [*] Definition of _ANON_ could be something like this--- almost certainly won't work as-is, this is just to point out that it could be done in standard R.
> 
> `_ANON_` <- function( expr) {
>    #1. Construct a function with arg '_PIPE_' and body 'expr'
>    #2. Construct a call() to that function
>    #3. Do the call
> 
>    f <- function( `_PIPE_`) NULL
>    body( f) <- expr
>    environment( f) <- parent.frame() # or something... yes these details are almost certainly wrong
>    expr2 <- substitute( f( `_PIPE_`)) # or something...
>    eval.parent( expr2) # or something...
> }
> 
> cheers
> Mark
> 
> Mark Bravington
> CSIRO Marine Lab
> Hobart
> Australia
> 
> 
> ________________________________________
> From: R-devel <r-devel-bounces at r-project.org> on behalf of Gabor Grothendieck <ggrothendieck at gmail.com>
> Sent: Monday, 7 December 2020 10:21
> To: Gabriel Becker
> Cc: r-devel at r-project.org
> Subject: Re: [Rd] New pipe operator
> 
> I understand very well that it is implemented at the syntax level;
> however, in any case the implementation is irrelevant to the principles.
> 
> Here a similar example to the one I gave before but this time written out:
> 
> This works:
> 
>    3 |> function(x) x + 1
> 
> but this does not:
> 
>    foo <- function(x) x + 1
>    3 |> foo
> 
> so it breaks the principle of functions being first class objects.  foo and its
> definition are not interchangeable.  You have
> to write 3 |> foo() but don't have to write 3 |> (function(x) x + 1)().
> 
> This isn't just a matter of notation, i.e. foo vs foo(), but is a
> matter of breaking
> the way R works as a functional language with first class functions.
> 
> On Sun, Dec 6, 2020 at 4:06 PM Gabriel Becker <gabembecker at gmail.com> wrote:
>>
>> Hi Gabor,
>>
>> On Sun, Dec 6, 2020 at 12:52 PM Gabor Grothendieck <ggrothendieck at gmail.com> wrote:
>>>
>>> I think the real issue here is that functions are supposed to be
>>> first class objects in R
>>> or are supposed to be and |> would break that if if is possible
>>> to write function(x) x + 1 on the RHS but not foo (assuming foo
>>> was defined as that function).
>>>
>>> I don't think getting experience with using it can change that
>>> inconsistency which seems serious to me and needs to
>>> be addressed even if it complicates the implementation
>>> since it drives to the heart of what R is.
>>>
>>
>> With respect I think this is a misunderstanding of what is happening here.
>>
>> Functions are first class citizens. |> is, for all intents and purposes, a macro.
>>
>> LHS |> RHS(arg2=5)
>>
>> parses to
>>
>> RHS(LHS, arg2 = 5)
>>
>> There are no functions at the point in time when the pipe transformation happens, because no code has been evaluated. To know if a symbol is going to evaluate to a function requires evaluation which is a step entirely after the one where the |> pipe is implemented.
>>
>> Another way to think about it is that
>>
>> LHS |> RHS(arg2 = 5)
>>
>> is another way of writing RHS(LHS, arg2 = 5), NOT R code that is (or even can be) evaluated.
>>
>>
>> Now this is a subtle point that only really has implications in as much as it is not the case for magrittr pipes, but its relevant for discussions like this, I think.
>>
>> ~G
>>
>>> On Sat, Dec 5, 2020 at 1:08 PM Gabor Grothendieck
>>> <ggrothendieck at gmail.com> wrote:
>>>>
>>>> The construct utils::head  is not that common but bare functions are
>>>> very common and to make it harder to use the common case so that
>>>> the uncommon case is slightly easier is not desirable.
>>>>
>>>> Also it is trivial to write this which does work:
>>>>
>>>> mtcars %>% (utils::head)
>>>>
>>>> On Sat, Dec 5, 2020 at 11:59 AM Hugh Parsonage <hugh.parsonage at gmail.com> wrote:
>>>>>
>>>>> I'm surprised by the aversion to
>>>>>
>>>>> mtcars |> nrow
>>>>>
>>>>> over
>>>>>
>>>>> mtcars |> nrow()
>>>>>
>>>>> and I think the decision to disallow the former should be
>>>>> reconsidered.  The pipe operator is only going to be used when the rhs
>>>>> is a function, so there is no ambiguity with omitting the parentheses.
>>>>> If it's disallowed, it becomes inconsistent with other treatments like
>>>>> sapply(mtcars, typeof) where sapply(mtcars, typeof()) would just be
>>>>> noise.  I'm not sure why this decision was taken
>>>>>
>>>>> If the only issue is with the double (and triple) colon operator, then
>>>>> ideally `mtcars |> base::head` should resolve to `base::head(mtcars)`
>>>>> -- in other words, demote the precedence of |>
>>>>>
>>>>> Obviously (looking at the R-Syntax branch) this decision was
>>>>> considered, put into place, then dropped, but I can't see why
>>>>> precisely.
>>>>>
>>>>> Best,
>>>>>
>>>>>
>>>>> Hugh.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sat, 5 Dec 2020 at 04:07, Deepayan Sarkar <deepayan.sarkar at gmail.com> wrote:
>>>>>>
>>>>>> On Fri, Dec 4, 2020 at 7:35 PM Duncan Murdoch <murdoch.duncan at gmail.com> wrote:
>>>>>>>
>>>>>>> On 04/12/2020 8:13 a.m., Hiroaki Yutani wrote:
>>>>>>>>>    Error: function '::' not supported in RHS call of a pipe
>>>>>>>>
>>>>>>>> To me, this error looks much more friendly than magrittr's error.
>>>>>>>> Some of them got too used to specify functions without (). This
>>>>>>>> is OK until they use `::`, but when they need to use it, it takes
>>>>>>>> hours to figure out why
>>>>>>>>
>>>>>>>> mtcars %>% base::head
>>>>>>>> #> Error in .::base : unused argument (head)
>>>>>>>>
>>>>>>>> won't work but
>>>>>>>>
>>>>>>>> mtcars %>% head
>>>>>>>>
>>>>>>>> works. I think this is a too harsh lesson for ordinary R users to
>>>>>>>> learn `::` is a function. I've been wanting for magrittr to drop the
>>>>>>>> support for a function name without () to avoid this confusion,
>>>>>>>> so I would very much welcome the new pipe operator's behavior.
>>>>>>>> Thank you all the developers who implemented this!
>>>>>>>
>>>>>>> I agree, it's an improvement on the corresponding magrittr error.
>>>>>>>
>>>>>>> I think the semantics of not evaluating the RHS, but treating the pipe
>>>>>>> as purely syntactical is a good decision.
>>>>>>>
>>>>>>> I'm not sure I like the recommended way to pipe into a particular argument:
>>>>>>>
>>>>>>>     mtcars |> subset(cyl == 4) |> \(d) lm(mpg ~ disp, data = d)
>>>>>>>
>>>>>>> or
>>>>>>>
>>>>>>>     mtcars |> subset(cyl == 4) |> function(d) lm(mpg ~ disp, data = d)
>>>>>>>
>>>>>>> both of which are equivalent to
>>>>>>>
>>>>>>>     mtcars |> subset(cyl == 4) |> (function(d) lm(mpg ~ disp, data = d))()
>>>>>>>
>>>>>>> It's tempting to suggest it should allow something like
>>>>>>>
>>>>>>>     mtcars |> subset(cyl == 4) |> lm(mpg ~ disp, data = .)
>>>>>>
>>>>>> Which is really not that far off from
>>>>>>
>>>>>> mtcars |> subset(cyl == 4) |> \(.) lm(mpg ~ disp, data = .)
>>>>>>
>>>>>> once you get used to it.
>>>>>>
>>>>>> One consequence of the implementation is that it's not clear how
>>>>>> multiple occurrences of the placeholder would be interpreted. With
>>>>>> magrittr,
>>>>>>
>>>>>> sort(runif(10)) %>% ecdf(.)(.)
>>>>>> ## [1] 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
>>>>>>
>>>>>> This is probably what you would expect, if you expect it to work at all, and not
>>>>>>
>>>>>> ecdf(sort(runif(10)))(sort(runif(10)))
>>>>>>
>>>>>> There would be no such ambiguity with anonymous functions
>>>>>>
>>>>>> sort(runif(10)) |> \(.) ecdf(.)(.)
>>>>>>
>>>>>> -Deepayan
>>>>>>
>>>>>>> which would be expanded to something equivalent to the other versions:
>>>>>>> but that makes it quite a bit more complicated.  (Maybe _ or \. should
>>>>>>> be used instead of ., since those are not legal variable names.)
>>>>>>>
>>>>>>> I don't think there should be an attempt to copy magrittr's special
>>>>>>> casing of how . is used in determining whether to also include the
>>>>>>> previous value as first argument.
>>>>>>>
>>>>>>> Duncan Murdoch
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Hiroaki Yutani
>>>>>>>>
>>>>>>>> 2020?12?4?(?) 20:51 Duncan Murdoch <murdoch.duncan at gmail.com>:
>>>>>>>>>
>>>>>>>>> Just saw this on the R-devel news:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> R now provides a simple native pipe syntax ?|>? as well as a shorthand
>>>>>>>>> notation for creating functions, e.g. ?\(x) x + 1? is parsed as
>>>>>>>>> ?function(x) x + 1?. The pipe implementation as a syntax transformation
>>>>>>>>> was motivated by suggestions from Jim Hester and Lionel Henry. These
>>>>>>>>> features are experimental and may change prior to release.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is a good addition; by using "|>" instead of "%>%" there should be
>>>>>>>>> a chance to get operator precedence right.  That said, the ?Syntax help
>>>>>>>>> topic hasn't been updated, so I'm not sure where it fits in.
>>>>>>>>>
>>>>>>>>> There are some choices that take a little getting used to:
>>>>>>>>>
>>>>>>>>>    > mtcars |> head
>>>>>>>>> Error: The pipe operator requires a function call or an anonymous
>>>>>>>>> function expression as RHS
>>>>>>>>>
>>>>>>>>> (I need to say mtcars |> head() instead.)  This sometimes leads to error
>>>>>>>>> messages that are somewhat confusing:
>>>>>>>>>
>>>>>>>>>    > mtcars |> magrittr::debug_pipe |> head
>>>>>>>>> Error: function '::' not supported in RHS call of a pipe
>>>>>>>>>
>>>>>>>>> but
>>>>>>>>>
>>>>>>>>> mtcars |> magrittr::debug_pipe() |> head()
>>>>>>>>>
>>>>>>>>> works.
>>>>>>>>>
>>>>>>>>> Overall, I think this is a great addition, though it's going to be
>>>>>>>>> disruptive for a while.
>>>>>>>>>
>>>>>>>>> Duncan Murdoch
>>>>>>>>>
>>>>>>>>> ______________________________________________
>>>>>>>>> R-devel at r-project.org mailing list
>>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>>>
>>>>>>>> ______________________________________________
>>>>>>>> R-devel at r-project.org mailing list
>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>>>
>>>>>>>
>>>>>>> ______________________________________________
>>>>>>> R-devel at r-project.org mailing list
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>
>>>>>> ______________________________________________
>>>>>> R-devel at r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>
>>>>> ______________________________________________
>>>>> R-devel at r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>
>>>>
>>>>
>>>> --
>>>> Statistics & Software Consulting
>>>> GKX Group, GKX Associates Inc.
>>>> tel: 1-877-GKX-GROUP
>>>> email: ggrothendieck at gmail.com
>>>
>>>
>>>
>>> --
>>> Statistics & Software Consulting
>>> GKX Group, GKX Associates Inc.
>>> tel: 1-877-GKX-GROUP
>>> email: ggrothendieck at gmail.com
>>>
>>> ______________________________________________
>>> R-devel at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
> 
> 
> 
> --
> Statistics & Software Consulting
> GKX Group, GKX Associates Inc.
> tel: 1-877-GKX-GROUP
> email: ggrothendieck at gmail.com
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>



[Rd] New pipe operator
Duncan Murdoch