From aixtools at gmail.com  Mon Jan  4 07:34:45 2016
From: aixtools at gmail.com (Michael Felt)
Date: Mon, 4 Jan 2016 07:34:45 +0100
Subject: [Rd] R,
 AIX 64-bit builds - trying to understand root cause for message:
 "Error: Line starting 'Package: tools ...' is malformed!"
In-Reply-To: <5138BF7E-44C2-4ABE-9D3F-0730BCF90BC9@r-project.org>
References: <56825A4B.7010508@gmail.com> <56839C9E.2010401@gmail.com>
	<5685466E.7000804@gmail.com>
	<4BED4A65-8EE9-49B8-8AFB-4E6CC3BB5BBF@r-project.org>
	<F92F30A1-4E5C-4312-B151-692F27013F8B@r-project.org>
	<F68557F2-67ED-4EF3-8BDF-F7981D8F38C0@gmail.com>
	<56894556.3020902@gmail.com>
	<5138BF7E-44C2-4ABE-9D3F-0730BCF90BC9@r-project.org>
Message-ID: <568A1285.5000709@gmail.com>

I would be "pleased" if you would try packages - i.e., none of "Toolbox" 
or Perzl rpm's.

The iconv I supply might not be working as is (I will need to install a 
new system to check). However, this is why I have been testing with 
R-3.2.3 - to side-step the system library dependencies.

re: xz - I have "installp" version - and any of my packages should work 
side-by-side with the Toolbox/.rpm files - my PATH prefix is /opt (so, 
/opt/bin and /opt/sbin rather than /usr/bin, /usr/local/bin (etc.).

I am testing on a Power6, usually with 2 to 4G RAM active. When I get 
the compile to finish I will up to 8G to test some data processing that 
fails with the 32-bit limitations.

Another question: are you also getting lots of duplicate symbol 
warnings? Curious me. If you are I would like to send my modifications 
to configure(.ac) for you to test (I put .ac in () because I am actually 
editting configure atm but shall put them into configure.ac when I finish)

FYI: I shall be downloading the "try and buy" xlc and xlfortran - and I 
think you will certainly prefer my packaging then as they work without 
the libgc dependencies that many of the rpm packages need.

And, at your option - we can take this discussion over tools - "out of 
forums". Or at least start a new thread.

re: unsigned short - I have adopted the convention to use the *NN_t 
types after running into a problem with unsigned longlong (from a very 
old program). It goes back many years - and maybe they have finalized 
short to mean 16-bits - but I remember when short was meant to help when 
moving from 16-bit to 32-bit and the "word" size changed - i.e., int 
became 32-bit same as long. nd for a long (no pun intended) long was 
32-bit and long long was 64-bit. Those definitions are extinct.

imho - the standard for wint_t is wrong as well - based on an assumption 
about how "short" is defined. And, I consider it poor practice that 
there are som many cases of type cast switches between ushort and int.

Michael

On 04-Jan-16 01:26, Simon Urbanek wrote:
> Michael,
>
> I'm using xlc + xlf - the exact flags are
>
> configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread' --prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include
>
> with export OBJECT_MODE=64 in the env (and libiconv from perzl.org RPMs - iconv is a serious pain).
>
> I have changed the TRE typedef from "wint_t" to "unsigned int" since that is what the other platforms use anyway.
>
> On my AIX7 VM I'm running out of memory in xz when lazy-loading is enabled which his odd - the VM has 4Gb which one would think should be enough. The symptom is that installing MASS fails when creating lazy-load rds or the same happens when running make check as memCompress() test for xz fails. I wish there was a way to disable xz altogether..
>
> I'd like to get IBM compilers to work first since those are the official ones. I may try gcc later, but I'm really interested in the IBM ones because that gives us a good non-GNU test (e.g., TRE fails in JIT due some alleged GNUisms so you can't use the strict compiler mode).
>
> Cheers,
> Simon
>
>
> On Jan 3, 2016, at 10:59 AM, Michael Felt <aixtools at gmail.com> wrote:
>
>> On 2016-01-01 23:48, peter dalgaard wrote:
>>> Nice catch you two!!!
>>>
>>> Happy New Year
>>> -pd
>> I am much happier with this great start!
>>
>> Simon - which compiler)s) did you use: xlc and xlfortran, or gcc/gfortran?
>>
>> I have made some changes to configure(.ac) so maybe my problems are self-inflicted. But would be good to know what environment you are using.
>>
>> Thanks for looking - and finding!!!
>>
>> Michael
>>>> On 01 Jan 2016, at 22:06 , Simon Urbanek<simon.urbanek at r-project.org>  wrote:
>>>>
>>>> Ok, found the problem - on platforms that support it TRE uses wint_t (from wchar.h) as its type for characters (tre_cint_t) which on AIX is *signed* int. TRE uses liberally conversions between int and tre_cint_t apparently assuming that the latter is unsigned so conversions back to int are suitable for comparisons etc. On other platforms wint_t is unsigned so it works. Manually defining tre_cint_t to unsigned int fixes the issue.
>>>>
>>>> Cheers,
>>>> Simon
>>>>
>>>>
>>>> On Jan 1, 2016, at 12:20 PM, Simon Urbanek<simon.urbanek at r-project.org>  wrote:
>>>>
>>>>> Michael,
>>>>>
>>>>> thanks, I'll have a look once my PDP VMs are up again (later today). This may be a signedness issue although it's unclear why other platforms wouldn't be affected.
>>>>>
>>>>> Cheers,
>>>>> Simon
>>>>>
>>>>>
>>>>> On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools at gmail.com>  wrote:
>>>>>
>>>>>> On 2015-12-30 09:58, Michael Felt wrote:
>>>>>>> On 2015-12-29 11:02, Michael Felt wrote:
>>>>>>>> This seems to be a problem that goes back a long time - and I hope someone who understands what tre is suppossed to be doing will look at this.
>>>>>>>>
>>>>>>>> A short history of other people who have reported on this on different versions of AIX. I shall only add that I get the same results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3.
>>>>>>>>
>>>>>>>> Basically, with settings that work for AIX and 32-bit - the only changes being
>>>>>>>> -maix32 becomes -maix64
>>>>>>>> and
>>>>>>>> export OBJECT_MODE=32 becomes export OBJECT_MODE=64
>>>>>>>>
>>>>>>>> Then to shorten the 'make' bla bla, first run just make, then
>>>>>>>>
>>>>>>>> cd src/library/tools
>>>>>>>> make -s sysdata
>>>>>>>>
>>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed
>>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed
>>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 hours and 30 minutes ago)
>>>>>>>>
>>>>>>>> To that, to get debug data, I have
>>>>>>>>
>>>>>>>> * added -DTRE_DUGUG to src/extra/tre/Makefile # ALL_CFLAGS = $(ALL_CFLAGS_LO) -DTRE_DEBUG
>>>>>>>> * rm src/extra/tre/tre-match-parallel.o
>>>>>>>> * find . -name \*.so -exec rm {} \;
>>>>>>>> * make
>>>>>>>> * cd src/library/tools
>>>>>>>> * make -s sysdata
>>>>>>>>
>>>>>>>> Attached are the two script files of the screen output. The 32-bit one is more verbose - and contains magically lines such as:
>>>>>>>> found match 3037fd14 (while "found" does not occur in the 64-bit output)
>>>>>>>>
>>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc /tmp/sysdata.??.*
>>>>>>>>    4730   14123  139916 /tmp/sysdata.32.text
>>>>>>>>    1312    3688   40528 /tmp/sysdata.64.text
>>>>>>>>    6042   17811  180444 total
>>>>>>>>
>>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep -c found /tmp/sysdata.??.*
>>>>>>>> /tmp/sysdata.32.text:19
>>>>>>>> /tmp/sysdata.64.text:0
>>>>>>>>
>>>>>>>>
>>>>>>>> Hope this brings us (or me), closer to a resolution to an old concern.
>>>>>>>>
>>>>>>>> And, best wishes for the new year!
>>>>>>>>
>>>>>>>> Michael
>>>>>>>>
>>>>>>>>
>>>>>>> Still hoping for someones curiosity/willingness.
>>>>>>>
>>>>>>> The differences show up in the first comparision that is made (of the string "3.2.3" it seems) - 32-bit is on the left, 64-bit on the right.
>>>>>>>
>>>>>>> Script command is started on Tue Dec 29 08:39:16 UTC 2015.                     |  Script command is started on Tue Dec 29 08:39:56 UTC 2015.
>>>>>>> root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make -s sysdata     |  root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make -s sysdata
>>>>>>> installing 'sysdata.rda'                                                       |  installing 'sysdata.rda'
>>>>>>> tre_tnfa_run_parallel, input type 1                                            |  tre_tnfa_run_parallel, input type 1
>>>>>>> length: -1                                                                     |  length: -1
>>>>>>> pos:chr/code | states and tags                                                 |  pos:chr/code | states and tags
>>>>>>> -------------+------------------------------------------------                 |  -------------+------------------------------------------------
>>>>>>> init>  30380200 3038014c 30380098                                             |   init>  110cc3040 110cc2f28 110cc2e10
>>>>>>> match end offset = -1                                                          |  match end offset = -1
>>>>>>> tre_tnfa_run_parallel, input type 1                                            |  tre_tnfa_run_parallel, input type 1
>>>>>>> length: -1                                                                     |  length: -1
>>>>>>> pos:chr/code | states and tags                                                 |  pos:chr/code | states and tags
>>>>>>> -------------+------------------------------------------------                 |  -------------+------------------------------------------------
>>>>>>> init>  3037fb88                                                               |   init>  110cc3310
>>>>>>> 0: 3/00051 | 3037fb88/0:0                                                    |    0: 3/00051 | 110cc3310/0:0
>>>>>>> 1: ./00046 | 3037fb88/0:0                                                    |    1: ./00046 | 110cc3310/0:0
>>>>>>> init>  3037fb88                                                               |   init>  110cc3310
>>>>>>> 1: ./00046 | 3037fb88/0:1                                                    |    1: ./00046 | 110cc3310/0:1
>>>>>>> 2: 2/00050 | 3037fb88/0:1                                                    |    2: 2/00050 | 110cc3310/0:1
>>>>>>> assertion failed                                                               |  assertion failed
>>>>>>> init>  3037fb88                                                               |   init>  110cc3310
>>>>>>> 2: 2/00050 | 3037fc18/0:1 3037fb88/0:2                                       |    2: 2/00050 | 110cc33f0/0:1 110cc3310/0:2
>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:2                                       |    3: ./00046 | 110cc33f0/0:1 110cc3310/0:2
>>>>>>> assertion failed                           *** DIFFERENCE ***                  |   init>  110cc3310
>>>>>>> init>  3037fb88                                                               |    3: ./00046 | 110cc3310/0:3
>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:3                                       |    4: 3/00051 | 110cc3310/0:3
>>>>>>> 4: 3/00051 | 3037fc18/0:1 3037fb88/0:3                                       |  assertion failed
>>>>>>> assertion failed                                                               |   init>  110cc3310
>>>>>>> init>  3037fb88                                                               |    4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4
>>>>>>> 4: 3/00051 | 3037fc18/0:3 3037fb88/0:4                                       |    5: /00000 | 110cc33f0/0:3 110cc3310/0:4
>>>>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 |   init>  110cc3310
>>>>>>> found match 3037fd14 *** DIFFERENCE ***                   |  match end offset = -1
>>>>>>> match end offset = 5 *** DIFFERENCE ***                   |  tre_tnfa_run_parallel, input type 1
>>>>>>> tre_tnfa_run_parallel, input type 1                                            |  length: -1
>>>>>>> length: -1                                                                     |  pos:chr/code | states and tags
>>>>>>> pos:chr/code | states and tags                                                 |  -------------+------------------------------------------------
>>>>>>> -------------+------------------------------------------------                 |   init>  110cc4780 110cc4668 110cc4550
>>>>>>> init>  303811c0 3038110c 30381058                                             |  match end offset = -1
>>>>>>> match end offset = -1                                                          |  tre_tnfa_run_parallel, input type 1
>>>>>>> tre_tnfa_run_parallel, input type 1                                            |  length: -1
>>>>>>> length: -1                                                                     |  pos:chr/code | states and tags
>>>>>>> pos:chr/code | states and tags                                                 |  -------------+------------------------------------------------
>>>>>>> -------------+------------------------------------------------                 |   init>  110cc5700 110cc55e8 110cc54d0
>>>>>>>
>>>>>> One day further - looks like tre_compile (or just before, after all).
>>>>>>
>>>>>> With TRE_DEBUG switched on in tre-compile.c and tre-ast.c I see (snip)
>>>>>>
>>>>>> --- /tmp/x.32   2015-12-31 15:09:44.000000000 +0000
>>>>>> +++ /tmp/x.64   2015-12-31 15:09:30.000000000 +0000
>>>>>> @@ -1,5 +1,5 @@
>>>>>> - Script command is started on Thu Dec 31 15:04:39 2015.
>>>>>> - root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make sysdata
>>>>>> + Script command is started on Thu Dec 31 15:08:43 2015.
>>>>>> + root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make sysdata
>>>>>> installing 'sysdata.rda'
>>>>>> echo "tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")" | \
>>>>>>   R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R --vanilla --slave
>>>>>> @@ -167,7 +167,7 @@
>>>>>> initial: 1/1,0, assert 0
>>>>>> initial: 0/0, assert 0
>>>>>> initial: 0/0, assert 0
>>>>>> - final state 30370718
>>>>>> + final state 110cba530
>>>>>> tre_compile: parsing '(^|[^%])(%%)*%V'
>>>>>> AST:
>>>>>> catenation, sub 0, 0 tags
>>>>>> @@ -177,7 +177,7 @@
>>>>>>         assertions: bol
>>>>>>         union, sub -1, 0 tags
>>>>>>           literal (, $) (0, 36), pos 0, sub -1, 0 tags
>>>>>> -           literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags
>>>>>> +           literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags
>>>>>>       iteration {0, -1}, sub -1, 0 tags, greedy
>>>>>>         catenation, sub 2, 0 tags
>>>>>>           literal (%, %) (37, 37), pos 1, sub -1, 0 tags
>>>>>> @@ -197,7 +197,7 @@
>>>>>> Union
>>>>>> Literal 0-36
>>>>>> After union left
>>>>>> - Literal 38-65535
>>>>>> + Literal 38--1
>>>>>> After union right
>>>>>> After union right
>>>>>>   num_tags += 2
>>>>>> @@ -231,7 +231,7 @@
>>>>>>         assertions: bol
>>>>>>         union, sub -1, 0 tags
>>>>>>           literal (, $) (0, 36), pos 0, sub -1, 0 tags
>>>>>> -           literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags
>>>>>> +           literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags
>>>>>>       iteration {0, -1}, sub -1, 2 tags, greedy
>>>>>>         catenation, sub 2, 1 tags
>>>>>>           literal (%, %) (37, 37), pos 1, sub -1, 1 tags
>>>>>> @@ -255,7 +255,7 @@
>>>>>> Union
>>>>>> Literal 0-36
>>>>>> After union left
>>>>>> - Literal 38-65535
>>>>>> + Literal 38--1
>>>>>> After union right
>>>>>> After union right
>>>>>> tre_add_tag_right: tag 3
>>>>>> @@ -342,7 +342,7 @@
>>>>>>           catenation, sub -1, 0 tags
>>>>>>             union, sub -1, 0 tags
>>>>>>               literal (, $) (0, 36), pos 0, sub -1, 0 tags
>>>>>> -               literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags
>>>>>> +               literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags
>>>>>>             tag 4
>>>>>>
>>>>>> It seems in 32-bit mode -1 is unsigned (65535) but -1 == -1 in 64-bit mode.
>>>>>>
>>>>>> I suspect I will "find it" - but a proposed change is appreciated.
>>>>>>
>>>>>> Happy New Year,
>>>>>> Michael
>>>>>>
>>>>>> ______________________________________________
>>>>>> R-devel at r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>
>>>>> ______________________________________________
>>>>> R-devel at r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>
>>>> ______________________________________________
>>>> R-devel at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-devel



[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
Michael Felt