From SheltonS at stemcell.ucsf.edu  Tue Aug 20 20:04:30 2013
From: SheltonS at stemcell.ucsf.edu (Shelton, Samuel)
Date: Tue, 20 Aug 2013 18:04:30 +0000
Subject: [Rd] Problem with R >3.0.0
In-Reply-To: <CA+hbrhVzUye1C3ZLpTWnPB21gaZYhiWeffHsf+sDCe0oeV0=RA@mail.gmail.com>
Message-ID: <CE38FC69.1882B%sheltons@stemcell.ucsf.edu>

Hi Peter,


But this is still an issue bellow the old R limit. I just tried the same
with a matrix of 30000*30000 and I see the same problem. This never used
to happen with R v2.15.2 and we could regularly build similarity matrices
of 45000*45000. This behavior of filling up the bottom of the matrix with
0's after calling diag is only happening with v3.0.0 and v3.0.1.

As I said I don't think that this is an issue with WGCNA but it has
implications for WGCNA because it limits the number of genes to be
included in network construction.

Thanks

Sam






On 8/20/13 10:53 AM, "Peter Langfelder" <peter.langfelder at gmail.com> wrote:

>Hi Samuel,
>
>WGCNA currently does not support calculations with matrices larger
>than the old R limit, and it will take some time to update it to
>support the large matrices. Furthermore, WGCNA relies on other
>functions (most notably hclust) that would have to be updated as well
>to support long vectors.
>
>In the meantime I suggest using the "blockwise" functions to handle
>large data sets, or, if possible, reducing the number of genes to less
>than the old limit of 46340 or so.
>
>Sorry I can't be of more help.
>
>Best,
>
>Peter
>
>On Tue, Aug 20, 2013 at 10:42 AM, Shelton, Samuel
><SheltonS at stemcell.ucsf.edu> wrote:
>> Hi all,
>>
>> Thanks for getting back to me. We would like to move over to v3.0.0 on
>>our
>> cluster so that we can build matrices larger than 46300*46300 (limit in
>>R
>> <3.0.0)
>> but so far we can't get things to work with R v3.0.0 and higher. I am
>> trying to trouble shoot at the moment and I am now thinking that the
>> problem is actually with the diag function that has been rewritten in
>> version 3.0.0.
>>
>>
>> The problem is definitely with the diag function and it does not occur
>>on
>> smaller matrices (20000*20000) and I think it maybe a bug.
>> This illustrates the problem:
>>
>> This was done on an iMac i5 with OSX 10.8.5 16GB Ram and with R 3.0.1
>>(but
>> I do see the same for 3.0.0). This does not occur when I run it with R
>> 2.15.2.
>>
>>
>> mat1=matrix(rexp(20000^2), 20000)
>>
>> mat1[1:10,1:10]
>>            [,1]       [,2]       [,3]      [,4]       [,5]      [,6]
>>  [,7]       [,8]       [,9]      [,10]
>>  [1,] 0.1829090 0.39867734 0.80499126 4.1746377 0.20717066 1.1477365
>> 0.469843567 2.57767543 0.17449595 0.01949358
>>  [2,] 0.5731522 0.15835939 0.29165029 0.6781249 0.64553728 2.4438404
>> 2.140374938 0.40091195 0.51201369 0.98904490
>>  [3,] 0.3250310 0.09934147 0.79962549 0.4933385 0.30473422 0.4556765
>> 0.002640034 0.90606861 2.58772944 0.89884208
>>  [4,] 1.4195017 0.16082660 0.01377838 0.2115803 1.43992672 0.3883675
>> 0.040903805 0.51011305 0.41998024 0.44209926
>>  [5,] 0.8328441 1.10335604 0.11875332 0.1600287 0.17333324 0.3388678
>> 3.206179119 0.52170966 1.03084845 0.05843232
>>  [6,] 1.3179906 0.76376188 1.24231798 0.9424030 0.04440514 1.0237664
>> 2.547528816 1.35629450 0.87983354 0.25236343
>>  [7,] 0.6990544 1.17003075 0.66063936 0.8632534 0.28965611 0.6718020
>> 1.137348735 0.08371053 0.23144290 0.18915132
>>  [8,] 0.9908026 1.20471979 0.08816010 0.2652131 0.03537790 0.3295816
>> 0.144371435 3.03299285 0.09728111 0.39890260
>>  [9,] 0.9557305 0.29196500 0.43955758 0.7332643 2.03457020 0.5858431
>> 2.437192399 0.34689557 0.02039205 0.54898488
>> [10,] 3.7220703 0.13572389 0.18888673 0.5683698 1.79209016 1.3495723
>> 0.571159401 0.63375850 0.63221987 1.32840290
>>> mat1[19990:20000,19990:20000]
>>            [,1]      [,2]       [,3]         [,4]      [,5]        [,6]
>>   [,7]      [,8]       [,9]     [,10]     [,11]
>>  [1,] 1.7910910 0.5719982 0.38689588 1.2157545685 0.8530179 1.464105574
>> 0.5986705 1.1623393 0.55244563 0.1770146 0.4326310
>>  [2,] 0.2862914 1.5267870 0.98214645 0.0004617244 0.6395319 0.075217874
>> 0.6725620 0.2403549 0.08436217 0.1435451 0.7487862
>>  [3,] 2.0492301 0.7216115 0.16951284 0.2726676762 2.1893806 1.202518385
>> 0.9897710 1.4813026 2.42517705 0.3398811 0.7285074
>>  [4,] 0.6538994 0.2437594 2.08848881 0.3917212249 0.4441824 0.433749415
>> 1.3022991 1.3695241 0.07057642 0.4296937 2.9307556
>>  [5,] 2.3688094 2.3970048 0.03545232 0.5986997508 0.8914097 0.497023176
>> 0.4210650 1.5337767 0.01141066 1.1562830 1.0572076
>>  [6,] 2.0626934 0.6186995 0.99197835 1.4794321654 0.1549314 1.296227000
>> 0.2790942 0.9327613 0.84131377 0.8782590 0.3279970
>>  [7,] 1.2423823 0.2385994 0.11390071 2.0745023842 1.9152523 0.754186281
>> 1.5474078 2.5899490 5.19298969 1.4680934 1.0537164
>>  [8,] 1.3657070 1.9502828 1.07681438 0.9339731540 1.7532474 0.186193421
>> 1.8699504 1.9187339 5.13248671 0.4621520 0.4753582
>>  [9,] 0.6512000 0.5104660 0.17820166 0.3965162944 0.0919119 0.187808660
>> 0.7391137 0.1574844 0.65985494 0.4066742 0.8072494
>> [10,] 0.7435028 1.1395666 2.46096009 0.7060164691 1.7965986 0.008278685
>> 0.4642319 0.1582297 1.71676326 0.3662139 0.7864957
>> [11,] 0.3537041 0.6622001 2.01642141 1.8225423060 0.3295436 1.260737179
>> 0.8430396 0.5132811 0.30547431 1.6088725 0.4001791
>>
>> diag(mat1)=0
>>
>> mat1[1:10,1:10]
>>            [,1]       [,2]       [,3]      [,4]       [,5]      [,6]
>>  [,7]       [,8]       [,9]      [,10]
>>  [1,] 0.0000000 0.39867734 0.80499126 4.1746377 0.20717066 1.1477365
>> 0.469843567 2.57767543 0.17449595 0.01949358
>>  [2,] 0.5731522 0.00000000 0.29165029 0.6781249 0.64553728 2.4438404
>> 2.140374938 0.40091195 0.51201369 0.98904490
>>  [3,] 0.3250310 0.09934147 0.00000000 0.4933385 0.30473422 0.4556765
>> 0.002640034 0.90606861 2.58772944 0.89884208
>>  [4,] 1.4195017 0.16082660 0.01377838 0.0000000 1.43992672 0.3883675
>> 0.040903805 0.51011305 0.41998024 0.44209926
>>  [5,] 0.8328441 1.10335604 0.11875332 0.1600287 0.00000000 0.3388678
>> 3.206179119 0.52170966 1.03084845 0.05843232
>>  [6,] 1.3179906 0.76376188 1.24231798 0.9424030 0.04440514 0.0000000
>> 2.547528816 1.35629450 0.87983354 0.25236343
>>  [7,] 0.6990544 1.17003075 0.66063936 0.8632534 0.28965611 0.6718020
>> 0.000000000 0.08371053 0.23144290 0.18915132
>>  [8,] 0.9908026 1.20471979 0.08816010 0.2652131 0.03537790 0.3295816
>> 0.144371435 0.00000000 0.09728111 0.39890260
>>  [9,] 0.9557305 0.29196500 0.43955758 0.7332643 2.03457020 0.5858431
>> 2.437192399 0.34689557 0.00000000 0.54898488
>> [10,] 3.7220703 0.13572389 0.18888673 0.5683698 1.79209016 1.3495723
>> 0.571159401 0.63375850 0.63221987 0.00000000
>>
>> mat1[19990:20000,19990:20000]
>>            [,1]      [,2]       [,3]         [,4]      [,5]        [,6]
>>   [,7]      [,8]       [,9]     [,10]     [,11]
>>  [1,] 0.0000000 0.5719982 0.38689588 1.2157545685 0.8530179 1.464105574
>> 0.5986705 1.1623393 0.55244563 0.1770146 0.4326310
>>  [2,] 0.2862914 0.0000000 0.98214645 0.0004617244 0.6395319 0.075217874
>> 0.6725620 0.2403549 0.08436217 0.1435451 0.7487862
>>  [3,] 2.0492301 0.7216115 0.00000000 0.2726676762 2.1893806 1.202518385
>> 0.9897710 1.4813026 2.42517705 0.3398811 0.7285074
>>  [4,] 0.6538994 0.2437594 2.08848881 0.0000000000 0.4441824 0.433749415
>> 1.3022991 1.3695241 0.07057642 0.4296937 2.9307556
>>  [5,] 2.3688094 2.3970048 0.03545232 0.5986997508 0.0000000 0.497023176
>> 0.4210650 1.5337767 0.01141066 1.1562830 1.0572076
>>  [6,] 2.0626934 0.6186995 0.99197835 1.4794321654 0.1549314 0.000000000
>> 0.2790942 0.9327613 0.84131377 0.8782590 0.3279970
>>  [7,] 1.2423823 0.2385994 0.11390071 2.0745023842 1.9152523 0.754186281
>> 0.0000000 2.5899490 5.19298969 1.4680934 1.0537164
>>  [8,] 1.3657070 1.9502828 1.07681438 0.9339731540 1.7532474 0.186193421
>> 1.8699504 0.0000000 5.13248671 0.4621520 0.4753582
>>  [9,] 0.6512000 0.5104660 0.17820166 0.3965162944 0.0919119 0.187808660
>> 0.7391137 0.1574844 0.00000000 0.4066742 0.8072494
>> [10,] 0.7435028 1.1395666 2.46096009 0.7060164691 1.7965986 0.008278685
>> 0.4642319 0.1582297 1.71676326 0.0000000 0.7864957
>> [11,] 0.3537041 0.6622001 2.01642141 1.8225423060 0.3295436 1.260737179
>> 0.8430396 0.5132811 0.30547431 1.6088725 0.0000000
>>
>> mat1=matrix(rexp(38000^2), 38000)
>> dim(mat1)
>> [1] 38000 38000
>> mat1[1:10,1:10]
>>            [,1]      [,2]       [,3]      [,4]        [,5]      [,6]
>> [,7]      [,8]       [,9]      [,10]
>>  [1,] 3.8622815 0.1357886 1.64090976 0.4494637 0.812315613 0.5328906
>> 0.45672475 0.2891504 1.57087882 1.27375802
>>  [2,] 3.8229940 0.1540735 0.30189392 1.2100152 0.003323061 0.7195875
>> 0.60251052 1.9820380 0.18637086 0.05154236
>>  [3,] 2.0411498 0.8371707 0.02714550 1.3032572 0.330472063 0.3502267
>> 0.11908140 0.4155857 3.46471729 0.31890778
>>  [4,] 1.5503390 0.7377494 2.01433675 0.6109255 1.844484309 1.2492693
>> 0.09365743 0.4006219 2.37769616 0.38643521
>>  [5,] 0.4815804 0.5824312 0.61003728 0.4782871 0.526454982 0.1207842
>> 0.93567987 1.7369767 4.47922786 0.20033928
>>  [6,] 0.3791645 0.1015489 1.96832962 1.5417178 1.030250434 0.1362716
>> 1.72807083 0.2570055 0.02127689 0.80225716
>>  [7,] 1.5212795 2.8133952 0.15990367 0.4337506 0.526532536 2.9926685
>> 0.01432572 0.6064162 0.69264596 0.50871566
>>  [8,] 1.2600365 0.1901277 2.34806048 1.1472887 0.141571521 2.0355007
>> 1.12583466 0.3391067 0.18707165 3.71877247
>>  [9,] 3.0197258 2.3693633 0.94571337 0.2756933 0.938999190 1.5892456
>> 0.18612994 1.0498866 1.89162156 1.56643880
>> [10,] 0.3573243 0.3047595 0.01894034 2.4666841 0.660994174 0.2248711
>> 0.25436398 1.1275389 0.20960212 0.63957112
>>
>>
>>
>>
>> mat1[37990:38000,37990:38000]
>>            [,1]       [,2]       [,3]       [,4]      [,5]       [,6]
>> [,7]       [,8]       [,9]     [,10]     [,11]
>>  [1,] 0.8349326 1.32098263 2.68879316 0.52361090 1.4066094 0.00754308
>> 2.4489865 1.76284621 0.09097126 2.1598758 0.8805701
>>  [2,] 2.5754994 1.15753606 1.76895066 3.06645700 0.8767225 0.33247639
>> 1.5726808 0.12698495 0.75271682 6.4336476 0.1330457
>>  [3,] 2.5737957 0.58234929 0.40403205 0.34882433 0.4074048 1.54867135
>> 2.5971068 0.27276140 0.56926494 0.2129180 1.3027215
>>  [4,] 0.9352113 0.46839288 0.41284388 1.40216119 0.8936151 2.98304058
>> 0.4350446 1.14864094 0.26756970 2.6662998 0.1802141
>>  [5,] 1.6430435 1.22137017 0.06943644 0.03251737 0.2142083 0.16964865
>> 1.6099314 0.95709429 0.22884734 0.5087986 0.9048555
>>  [6,] 2.5340159 1.28618183 3.86698388 1.05946189 3.5006776 0.39320471
>> 0.4927357 0.66159842 0.18235678 0.5172896 2.2221550
>>  [7,] 0.1961937 0.17305031 0.62327325 3.14622730 3.4905449 3.08823676
>> 0.5282165 0.48879156 0.11807913 0.7372706 0.9975103
>>  [8,] 0.8392414 0.17107463 1.08732839 0.28981611 0.4722655 0.17587788
>> 1.7426814 0.13985161 1.86885446 1.9031580 1.1088431
>>  [9,] 3.4648342 0.33834943 0.07891645 0.07206860 1.1792628 0.54092203
>> 0.8141844 2.82687085 0.78395229 0.5313417 3.2164664
>> [10,] 0.1524299 0.02045616 0.67881610 1.51491647 2.4390115 0.89033581
>> 0.9026651 0.46858464 1.47711638 2.1821178 2.2149071
>> [11,] 0.9397162 0.42376104 0.20034405 1.47672836 1.0461904 0.97296202
>> 2.1658717 0.04487329 0.30611082 1.1680312 0.9952517
>>
>> diag(mat1)=0
>>
>> mat1[1:10,1:10]
>>            [,1]      [,2]       [,3]      [,4]        [,5]      [,6]
>> [,7]      [,8]       [,9]      [,10]
>>  [1,] 0.0000000 0.1357886 1.64090976 0.4494637 0.812315613 0.5328906
>> 0.45672475 0.2891504 1.57087882 1.27375802
>>  [2,] 3.8229940 0.0000000 0.30189392 1.2100152 0.003323061 0.7195875
>> 0.60251052 1.9820380 0.18637086 0.05154236
>>  [3,] 2.0411498 0.8371707 0.00000000 1.3032572 0.330472063 0.3502267
>> 0.11908140 0.4155857 3.46471729 0.31890778
>>  [4,] 1.5503390 0.7377494 2.01433675 0.0000000 1.844484309 1.2492693
>> 0.09365743 0.4006219 2.37769616 0.38643521
>>  [5,] 0.4815804 0.5824312 0.61003728 0.4782871 0.000000000 0.1207842
>> 0.93567987 1.7369767 4.47922786 0.20033928
>>  [6,] 0.3791645 0.1015489 1.96832962 1.5417178 1.030250434 0.0000000
>> 1.72807083 0.2570055 0.02127689 0.80225716
>>  [7,] 1.5212795 2.8133952 0.15990367 0.4337506 0.526532536 2.9926685
>> 0.00000000 0.6064162 0.69264596 0.50871566
>>  [8,] 1.2600365 0.1901277 2.34806048 1.1472887 0.141571521 2.0355007
>> 1.12583466 0.0000000 0.18707165 3.71877247
>>  [9,] 3.0197258 2.3693633 0.94571337 0.2756933 0.938999190 1.5892456
>> 0.18612994 1.0498866 0.00000000 1.56643880
>> [10,] 0.3573243 0.3047595 0.01894034 2.4666841 0.660994174 0.2248711
>> 0.25436398 1.1275389 0.20960212 0.00000000
>>
>>
>> ## After calling the diag function the bottom of the matrix is all set
>>to
>> 0.
>>
>>
>> mat1[37990:38000,37990:38000]
>>       [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11]
>>  [1,]    0    0    0    0    0    0    0    0    0     0     0
>>  [2,]    0    0    0    0    0    0    0    0    0     0     0
>>  [3,]    0    0    0    0    0    0    0    0    0     0     0
>>  [4,]    0    0    0    0    0    0    0    0    0     0     0
>>  [5,]    0    0    0    0    0    0    0    0    0     0     0
>>  [6,]    0    0    0    0    0    0    0    0    0     0     0
>>  [7,]    0    0    0    0    0    0    0    0    0     0     0
>>  [8,]    0    0    0    0    0    0    0    0    0     0     0
>>  [9,]    0    0    0    0    0    0    0    0    0     0     0
>> [10,]    0    0    0    0    0    0    0    0    0     0     0
>> [11,]    0    0    0    0    0    0    0    0    0     0     0
>>
>> It looks like there is an issue with larger matrices when calling diag
>> function and it has nothing to do with WGCNA.
>>
>>
>>
>>
>>
>>
>> On 8/20/13 9:43 AM, "Peter Langfelder" <peter.langfelder at gmail.com>
>>wrote:
>>
>>>Hi Sam,
>>>
>>>I assume you mean that correlation for _genes_ (not samples)
>>>11262:30000 is 0? I am the maintainer of the WGCNA package but
>>>unfortunately I don't have access to a Mac big enough to try
>>>30000x30000 correlation matrix, but I would be thankful if you could
>>>try reproducing the problem with smaller matrices (e.g. 20000x20000)
>>>and try to produce a small reproducible example by
>>>generating the data using say rnorm, say like this:
>>>
>>>nGenes = 20000 # as small as possible that still produces the error
>>>nSamples = 100
>>>datExpr1 = matrix(rnorm(nSamples * nGenes), nSamples, nGenes)
>>>
>>>simMat = bicor(datExpr1, use = 'p')
>>>
>>>Best,
>>>
>>>Peter
>>>
>>>On Mon, Aug 19, 2013 at 11:53 AM, Shelton, Samuel
>>><SheltonS at stemcell.ucsf.edu> wrote:
>>>> Dear R developer,
>>>>
>>>> I am an R user and am currently having a problem with versions of R
>>>>>3.0.0. We build larger correlation matrices of >30000 with pairwise
>>>>>correlations made. We have been using a line of code
>>>>
>>>> simMat=bicor(datExpr1,use="p")
>>>>
>>>> To build the similarity matrix with datExpr1 being a matrix with genes
>>>>as columns (>30000) and rows being samples. This code works fine on an
>>>>iMac running OSX 10.8.5 and using R 2.15.2 and the Rblas library. When
>>>>I
>>>>try to run the same code with R version 3.0.0 and 3.0.1 it only
>>>>partially builds the matrix. It will return correlations for samples
>>>>1:11262 but from 11262:30000 the matrix is full of 0's.
>>>>
>>>> We would like to be able to use R 3.0.0 to allow us to build larger
>>>>correlation matrices on our cluster but we can't use it at the moment
>>>>due to this problem. My colleague has previously had a similar issue
>>>>when using snow leopard and this was to do with a compatibility issue
>>>>between snow leopard and the veclib. Has something changed in version
>>>>3.0.0 to give an incompatibility with the veclib?
>>>>
>>>> Many thanks
>>>>
>>>> Sam
>>>>
>>>> Samuel Shelton, Ph.D.
>>>> Postdoctoral Researcher,
>>>> The Oldham Lab,
>>>> The institute of Regeneration Medicine,
>>>> University of California San Francisco,
>>>> San Francisco, CA 94143
>>>>
>>>> email: sheltons at stemcell.ucsf.edu
>>>>
>>>>
>>>>         [[alternative HTML version deleted]]
>>>>
>>>> ______________________________________________
>>>> R-devel at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>>



[Rd] Problem with R >3.0.0
Shelton, Samuel