text:
# The Local Hole in the Galaxy Distribution: New Optical Evidence

## 1 Introduction

It has been apparent for many years that the B galaxy number counts may be steeper at bright magnitudes than expected in a homogeneous, unevolving, cosmological model (Shanks et al. 1990). The counts appeared to have a Euclidean slope whereas the models predict a much flatter slope due to the expansion and the K-correction. Shanks et al. suggested that these steep counts maybe caused by large-scale galaxy clustering whereas other authors such as Maddox et al. (1990a) tended to favour the suggestion that the steepness was due to evolution. Both ideas had difficulties - the unevolved n(z) at B=17 and B=21 argued against galaxy luminosity evolution, while the local void would have to persist to almost 300$`h^1`$Mpc if the low number counts were caused by galaxy clustering.

Most of these authors used counts made from measurements of photographic plates. This meant that there remained the possibility that these observations were in error, due to non-linearities in the photographic magnitude scales. More recently it has become possible to survey large areas of the sky with linear CCD detectors. At first these were used to calibrate the photographic magnitudes but this effort has been hindered by the quite large rms errors in the photographic photometry found even at relatively bright magnitudes. The CCD detectors are now large enough to provide the survey data by themselves and in this paper we describe a new survey primarily in the 2dF galaxy redshift survey areas, using the CTIO Curtis Schmidt CCD camera.

The advent of deeper galaxy redshift surveys such as 2dFGRS opens up a new possibility of investigating the scale at which the local hole ceases to dominate and the Universe appears to become homogeneous. In previous surveys such as the Durham-UKST Galaxy Redshift survey, the n(z) extended to z$``$0.1 and it was found that count models which quite successfully fitted galaxy counts at B$`>`$18 did not fit the B$`<`$17 n(z) even in the range 0.05$`<`$z$`<`$0.1 which might have been thought to be safely outside any local inhomogeneities. The fact that the model n(z) over-predicted the observed n(z) all the way to z=0.1 meant that if the local void hypothesis was correct then the ‘hole’ in the galaxy distribution had to extend to z$``$0.1 or r$``$ 300$`h^1`$Mpc.

In this paper we first present details of our observations and data reduction in Section 2. In section 3 we use our CTIO CCD data to check the calibration of several previous galaxy number counts so that the corrected results from the various authors can be compared to the CTIO counts themselves. In section 4 the CTIO counts are presented in each band. The number redshift distributions from the 2dF Galaxy Redshift Survey (2dFGRS; Colless et al. 2001) and the Durham-UKST Redshift Survey (Ratcliffe et al. 1998) are examined in Section 5 to investigate the possible existence of a ‘local hole’ in the galaxy distribution in the Southern Galactic Cap (SGC). A discussion follows in Section 6.

## 2 Data Reduction

### 2.1 Observations

The observations were taken using the 0.6m Curtis Schmidt Telescope at CTIO, La Serena, Chile. We had two filters, Harris B and R, with magnitude limits of 20.5 and 19.5 respectively, and the imaging data was taken over 2 observing runs of 7 nights, each in excellent weather conditions. The North Galactic Cap (NGC) observations were taken from 8-14 April 1999 inclusive (observers GSB and TS), and the data in the South Galactic Cap (SGC) from 17-23 October 2000 inclusive (observers GSB and PJO). The CCD is a 2048x2048 24 micron chip with 2.3 arcsecond pixels, so that when the bias was subtracted this resulted in a 1.69 degree2 field of view for each exposure.

In the NGC, we observed 3 main strips of sky in B and R at declinations of 0, -5 and -10 degrees, where the equatorial and -5 degree strips overlap with the 2dF Galaxy Redshift Survey (Colless et al. 2001). The strips were 1.3 degrees wide, which was dictated by the field of view of the telescope, and had an RA range from 9hrs. 45mins. up to 15hrs. The strips were joined at the ends by two smaller strips which were constant at declinations of 10hrs and 15 hrs. In total, this gave a potential 300 hundred square degrees of data in our B and R filters, assuming we had photometric conditions throughout all 7 nights.

In the SGC, we again observed in 3 main strips at constant declinations, this time at -28, -30 and -32, with the knowledge that this would completely overlap with the 2dFGRS fields. The ends of the strips were at RA’s of 21hrs. 40mins. and 03hrs. 15mins. Because the strips were much closer together than in the NGC, it was possible to connect the strips by simply doing single exposures every half hour of RA at declinations of -29 and -31 degrees. In addition to these three long strips of $``$ 100 square degrees each, we also observed a shorter strip at a declination of -45 degrees from an RA of 02hrs. to 02hrs. 45mins. giving an area of $`15`$ square degrees. In total this gave us an imaging area in the SGC of 337 square degrees.

The format of our observations would be to take two standard star frames of suitable Landolt equatorial fields in each band at the beginning, the middle and the end of each night. We would then begin the observations in our chosen field by taking an exposure in the R band for 120 seconds and stop the tracking of the telescope for 1 min 18 seconds (when observing in the North, 1 min. 30 secs. in the South), so that the sky moved over by one quarter of the CCD chip. This 1 min 18 seconds was sufficient to let the CCD read out which took $`45`$ seconds. An exposure of the same length of time would then be taken in the B filter, and so on. The result would be a series of frames at a constant declination in the sky where each frame overlapped its adjacent frames in the same filter by about half a chip. We would typically be able to cover about 90 mins. of RA (30 deg2) using this method before the sky was too far over and we would then move the telescope and begin again.

The bias was subtracted, images trimmed and bad pixels corrected using the IRAF quadproc package. Typically, five or six dawn and twilight flat field images were taken in each bandpass. A master flat field image was produced in each filter for each night by first using the imcombine routine to median together each of the dawn flats and evening flats separately. The resulting frames were then averaged to produce our B and R master flats for that night. By dividing the median-ed dawn and twilight flats we typically found a 1-2 per cent gradient from top to bottom of the resulting frame, implying an error of about 0.005-0.01 to our galaxy photometry due to this effect. The frames for a particular night in each filter were then flat-fielded using the IRAF ccdproc package and the appropriate flat-field master frame.

### 2.2 Photometric Analysis

The fact that adjacent frames of the same filter overlapped by half-a-chip meant that a detailed analysis of the photometric conditions throughout each night could be performed. The results of this photometric analysis are shown in Table 1. Given that the weather was so good during both observing runs we decided to enforce a relatively strict criterion to determine the nature of the observing conditions. A night was deemed partially photometric if two or more adjacent frames show significant offsets ($`>0.1`$ mag) from each other when performing best fits to the magnitude residuals of the stars. If a particular night was deemed to be “partially photometric” then none of the frames were used on the strip where the $`>0.1`$ magnitude offset was found.

In order to ensure accurate zero-point calibration our strategy was to scale each frame based on its airmass relative to the first frame in the sequence. We found that, assuming photometric conditions, the cumulative magnitude offset across a whole sequence was consistent with the airmass variation from the first to the last frame. Since we enforced strict criteria on whether frames had been observed in photometric conditions, this assumption was justified. This also meant that the calibration of each frame was completely independent of all the others in the sequence, eliminating the possibility of the propagation of magnitude offset errors. To take these airmass variations into account the frames were scaled according to the equations:

$$f_B=10^{0.209(X_{1_B}X_{n_B})/2.5}$$
(1)
$$f_R=10^{0.108(X_{1_R}X_{n_R})/2.5}$$
(2)
where $`X_{1_B}`$, $`X_{n_B}`$ are the air-masses in the first and nth frames in a B filter sequence and $`X_{1_R}`$, $`X_{n_R}`$ the corresponding air-masses for the R filter. $`f_B`$ and $`f_R`$ are the scale factors for the B and R filters respectively and the values 0.209 and 0.108 are the quoted airmass coefficients in B and R for the CTIO observatory. By using high-mass standard star frames in the B and R filters we obtained values for the airmass coefficients of 0.19 and 0.10 respectively, in good agreement with the quoted values.

### 2.3 Standard Stars and the Colour Equation

Once the internal zero-points had been calibrated a global zero-point for each sequence had to be calculated. We did this by using the B and R magnitudes of the Landolt (1992) standards stars from our standard star frames which were taken at the beginning, middle and the end of each night using the fields SA101, SA107, and SA110. Our standard star exposure times were 10 secs. in R and 20 secs. in B. The IRAF fitparams routine was then used to determine the best fit zero-point offsets (b1 and r1) and colour-term coefficients (b3 and r3) for each band in the equations:

$$m_b=B+b_1+b_2X_b+b_3(BR)$$
(3)
$$m_r=R+r_1+r_2X_r+r_3(BR)$$
(4)
where B and R are the Landolt standard star magnitudes, Xb and Xr are the air-masses of the standard star frame for the B and R bands respectively and mb, mr are the calculated magnitudes of the stars. The parameters b2 and r2 are equal to 0.209 and 0.108 respectively and, as has been mentioned, are the quoted airmass coefficients for the CTIO observatory. We found typical values of 0.05 for b3, the colour-term coefficient, but the error on this value was 0.04. Since this was comparable to b3, which was itself small, we decided only to fit b1 in the colour equation. Because of the same reasons we only fitted the R band zero-point offset, r2 in equation 4. Since we did not use any colour terms in equations 3 and 4 all reduced magnitudes of our final sources will be in B and Rkc, as used by Landolt (1992). The mean of the b1’s and r1’s was calculated over all seven nights and our resulting colour equations were then:

$$m_b=B+4.533(_{}^+0.0053)+0.209X_b$$
(5)
$$m_r=R+4.203(_{}^+0.0067)+0.108X_r$$
(6)
We used the SEXtractor software package (Bertin et al. 1996) to measure the magnitudes of our sources and perform star/galaxy separation. The sources were extracted, sequence by sequence, and frame by frame in each sequence. The seeing stellar FWHM was first estimated for a particular frame with an initial call of SExtractor and then the objects were extracted with a second pass, making use of the calculated stellar FWHM. We used the MAG\_BEST parameter in SExtractor for the source magnitudes and since we had at least two observations of each source because of our frame overlaps, the mean of the two magnitudes was used in order to minimize errors in the photometry. A cosmic ray was defined as being a source which appeared on one frame but on neither of the adjacent frames and these would not be included in our source catalogue, but written to a cosmic ray file.

### 2.4 Star/Galaxy Separation

Star/Galaxy classification was carried out using the SExtractor software. The parameter CLASS\_STAR is assigned a particular value for a source which varies between 0 and 1 for galaxy and star-like sources respectively. For B$`<`$18 and R$`<`$17 the SExtraxtor CLASS\_STAR parameter is a good separator of stars and galaxies with 91 per cent in B and 90 per cent in R of sources with either CLASS\_STAR$`>`$0.9 or $`<`$0.1. The fact that we have two filters is useful since we get four attempts to classify a particular source (as we have overlaps in each filter) instead of two, and can consequently achieve lower errors on the CLASS\_STAR parameter. A full account of this technique is given in Busswell (2001).

To verify our star/galaxy separation we have checked our results with other reliable external sources. We find a galaxy completeness of $`>`$90 per cent for B$`<`$18 when comparing our northern equatorial field with $`90`$deg2 of the currently available Sloan Digital Sky Survey (SDSS; Yasuda et al. 2001). Stellar contamination is consequently less than 10 per cent in this magnitude range; the star/galaxy separation begins to break down for B$`>`$18 as the stellar contamination increases. A similar trend is observed in the R-band data except the completeness begins to decrease at R=16.5, but is still equal to 84 per cent for R=17. A similar check of the completeness with the Millennium Galaxy Catalogue (MGC; Driver - personal communication) in the B-band indicates a 100 per cent agreement in the source classification for B$`<`$16, with completeness and stellar contamination levels of 93 per cent and 5 per cent respectively at B=18. It should be noted that the MGC also used SEXtractor for the photometry and star/galaxy separation calculations.

## 3 CTIO Curtis Schmidt CCD Survey

### 3.1 Photometry Comparisons

In order to check the accuracy of our photometry we have compared our galaxy sample with other very accurate data-sets. In Fig. 1 we show a comparison with the B-band MGC data for 5,778 matched galaxies over a 32 deg2 area. The agreement is good, with negligible zero-point or scale errors. However, our magnitude errors can be seen to increase significantly for B$`>`$18.

We can make a similar comparison with the SDSS data (Yasuda et al. 2001) over the 90 square degree overlap region, using the g and r filters which need to be converted to our B Landolt band. The colour equation used in Yasuda et al. (2001) is:

$$B=g^{}+0.482(g^{}r^{})+0.169$$
(7)
where $`g^{}`$ and $`r^{}`$ are in the AB magnitude system and the asterisks represent the fact that the SDSS photometry is preliminary. By plotting B-g against $`g^{}r^{}`$, shown in Fig 2 we can test this colour transformation for our galaxy sample. We in fact find the best fit line corresponds to the colour equation $`B=g+0.549(\pm 0.07)(g^{}r^{})+0.114`$. The discrepancy between the colour terms, which is consistent with the $`\pm 0.07`$ error, accounts for the 0.169-0.114=0.055 difference in the zero-point. In fact, if we insist on using the 0.482 colour term from Yasuda et al. (2001) then the zero-point difference (i.e. the average of the relation B-g-0.482(g-r)-0.169 for all common galaxies between the SDSS and our CTIO survey) is less then 0.01 mag. The photometry system used for the SDSS data is new and the appropriate transformations to other band-passes are relatively poorly understood. This, combined with the fact that their photometry is still preliminary, could explain the discrepancy between the colour terms.

A comparison with the SDSS photometry, this time in the R Kron-Cousins pass-band, is shown in Fig. 3, where we have plotted the SDSS Rkc magnitude estimate against the magnitude difference SDSS(Rkc)-CTIO(Rkc). The SDSS Rkc magnitude is calculated using the colour equation quoted in Blanton et al. (2001) of RGKC=r \- 0.05 - 0.089(g-r). A further correction is performed using the relation RGKC-Rkc=0.08 (Shectman et al. 1996) in order to obtain an estimate of the SDSS Rkc magnitude. We find excellent agreement over the magnitude range shown in Fig. 3, with a mean zero-point difference of SDSS(R$`kc`$)-CTIO(Rkc)=0.02($`\pm 10.01)`$.

Further checks of our R-band photometry in the NGC and comparisons to our B and R data in the SGC has been carried out in Busswell (2001) using data from Metcalfe et al. (1998) in the DARS GSA and GNB fields. This data shows an intrinsic scatter of 0.06 mags. in both B and R. Mean magnitude differences were found of -0.02$`\pm 0.018`$, +0.02$`\pm 0.02`$, and, -0.03$`\pm 0.02`$ for comparisons to R-band data in the NGC and B-band and R-band data in the SGC respectively.

### 3.2 The APM Bright Galaxy Photometry Correction

In order to investigate the APM photometry at bright magnitudes we have used our CTIO SGC data (our un-dust-corrected magnitudes), covering 297 deg2, to check the accuracy of the publicly available APM Bright Galaxy Catalogue (APMBGC; Loveday et al. 1996)). This data reaches a magnitude limit of bJ=16.44 and so to reach slightly fainter magnitudes we have also used the publicly available photometry from the APM-Stromlo Redshift Survey (APMSRS), where the magnitude limit is bJ=17.15. We have converted our CTIO data to the bJ magnitude system using both our B Landolt and R Kron-Cousins data, in conjunction with the colour equation from Pimbblet et al. (2001):

$$b_J=B0.17(BR)$$
(8)
Fig. 4 shows this comparison with the bJ mag. plotted vs the mag. difference APM-CTIO. In our 297 deg2 survey area we find 629 matching galaxies with the APMBGC, which are of course brighter than bJ=16.44 and 96 matching galaxies with the APMSRS brighter than bJ=17.15. Note that the APMSRS has a sampling rate of 1/20 over the full 4300 deg2 APM area and so we therefore expect to find 20 times more matched galaxies with the APMBGC than for the APMSRS in a given mag. interval. This low sampling rate is clearly illustrated in Fig. 4, where the number of matched galaxies drops sharply as one moves to fainter magnitudes than bJ=16.44, the magnitude limit of the APMBGC. We find a mean zero-point difference of APM-CTIO=0.313$`\pm 0.01`$ mag. The difference is quite large and we will see in the discussion section what this 0.313 mag. correction could mean in terms of the implications for a hole in the SGC distribution of galaxies.

### 3.3 The Durham/UKST Photometry Correction

In this section we use our CTIO CCD photometry to check the accuracy of the Durham/UKST photometry. The DUKST redshift survey covers 1500 deg2 in 4 strips and our CTIO SGC data overlaps with one of these 4 strips. The angular size of the survey is 20x75 in the SGC to a limiting apparent magnitude of bJ=16.86. The catalogue consists of $``$2500 galaxy redshifts and therefore by comparing the model of Metcalfe et al. with the DUKST n(z) distribution, and taking into account any zero-point discrepancies with our CCD data, we can probe a larger area (1500 deg2) in the SGC than the 2dFGRS can ($``$300 deg2 at present). Although the DUKST will probe lower redshifts, the larger area will mean tighter constraints on the size and angular extent of the SGC deficiency in the galaxy distribution.

The DUKST galaxies were selected from the Edinburgh/Durham Southern Galaxy Catalogue (EDSGC: Collins, Heydon-Dumbleton & MacGillivray 1988). Ratcliffe et al. (1998) applied small corrections to the EDSGC bJ magnitudes in each of the 60 fields in an attempt to put them on the same zero-point scale of the APM catalogue. These corrections were calculated from results of Dalton et al. (1995) with the mean correction over all 60 fields, $`<b_J^{APM}b_J^{EDSGC}>`$=- 0.05$`\pm 0.11`$ mag. It should also be pointed out that the zero-point corrections were performed at b$`{}_{J}{}^{}`$19.5 as the APM is expected to give reliable photometry here. If this mean offset is to be believed in the magnitude range of the DUKST (b$`{}_{J}{}^{}<17`$), one is assuming there is no scale error in either the APM or EDSGC photometry.

Our CTIO SGC fields overlap with 28 of the 60 EDSGC fields and therefore we can make extensive tests of a large fraction of the DUKST photometry using our accurate CCD photometry. To convert to the DUKST bJ magnitude system we use our B Landolt and R Kron-Cousins photometry in conjunction with equation 8 (Pimbblet et al. 2001). In Fig. 5 we plot all the galaxies from each of the 28 fields in order to determine a mean zero-point offset. The dotted line shows the mean zero-point offset binned in half magnitude intervals and the solid line shows the mean zero-point offset over all magnitudes which we calculate to be 0.310. In fact the mean offset between the EDSGC photometry and the APM data at bJ=19.5 is 0.07 for the 28 fields which overlap with our CTIO data as opposed to 0.05 for all 60 fields. This means that we find a 0.31-0.07=0.24 zero-point offset between our CCD photometry and the corrected EDSGC photometry used for the DUKST survey. Although the DUKST photometry was corrected in order to be consistent with the zero-point of the APM, it is clear that there are discrepancies compared to our accurate CCD data. These zero-point differences appear field dependent although in some fields there are very few galaxies.

This mean zero-point offset between our CTIO CCD data and that of the DUKST may seem surprising given that the DUKST photometry has already been corrected/brightened by an average value of 0.05 mag. in order that it is consistent with the APM data at bJ=19.5 However, as we have already mentioned, any scale errors inherent in the APM or the DUKST could mean that this 0.05 correction may not apply at b$`{}_{J}{}^{}<`$17. Our checks of the 2dFGRS photometry (based on the original APM data) showed up no significant scale error in the magnitude range $`16<`$b$`{}_{J}{}^{}<19.5`$ and therefore we claim that the problem lies with the EDSGC photometry used for the DUKST survey. A scale error of order 0.1 mag/mag in the range 17$`<b_J<`$19.5 would explain the 0.24 mag. discrepancy with our CCD data which we have found to be accurate to a few hundredths of a magnitude at B$`<`$17 from checks performed in Section 3.1.

### 3.4 Photometry Checks of the 2dFGRS

As described in Section 2, our 297 deg2 SGC field is a complete subset of the southern 2dFGRS region. Therefore, providing our photometry is consistent with that of the 2dF, their SGC n(z) distribution can help us understand the exact nature of the apparent under-density observed relative to the galaxy count model of Metcalfe et al. (2001). Furthermore, for b$`{}_{J}{}^{}>`$17, the 2dFGRS photometry has a zero-point that has changed slightly from the original APM data due to re-calibration with external CCD magnitudes. In the SGC this zero-point difference is very small, APM-2dFGRS=0.02 mag., and so any comparisons of our CTIO data with the 2dFGRS magnitudes will also be good checks of the original APM photometry for b$`{}_{J}{}^{}>17`$. In the NGC this zero-point difference is APM-2dFGRS=0.08 mag. (Cole priv. com.). In order to check how our photometry compares with that of the 2dFGRS we have matched all galaxies from the two surveys using matches within a 3.5′′ radius. As in Section 4.1.1, we use both our B Landolt and R Kron-Cousins galaxy photometry in conjunction with the colour equation from Pimbblet et al. (2001) in order to convert to the 2dFGRS bJ filter. Figs. 6 and 7 show these comparisons for the NGC and SGC regions respectively. Note that in the NGC only $``$2/3 of our fields overlap with those of the 2dFGRS as opposed to 100 per cent in the SGC explaining the factor of $``$1.5 difference in the number of matching galaxies. We have plotted the 2dF bJ magnitude against the magnitude difference 2dF-CTIO for each field with the mean of the residual illustrated by the squares in each half magnitude bin. The 2dF galaxy photometry is of course based on that of the APM Galaxy Survey (Maddox et al. 1990a) with rms errors on the source magnitudes of 0.2-0.25 mag., and this can be seen via the large scatter around the mean in each magnitude bin. We have shown in Section 3.1 that our CCD photometry is much more accurate than this with rms errors of 0.1 mag. at B=18 and 0.05 mag. at B=16, and therefore we expect any zero-point and scale errors between the two surveys to be contained within the photographic 2dF data. The NGC comparison in Fig. 6 shows clear evidence of both a zero-point and scale error. We find good agreement at bJ=16 but the scale error induces a maximum zero-point error of 0.13 mag. at B=18 with a mean correction of 0.1 mag. for b$`{}_{J}{}^{}>`$16. For the SGC field in Fig. 7 we find a much better agreement with no appreciable scale error and the mean of the residuals agreeing to within 0.05 mag. over the whole magnitude range shown.

## 4 CTIO Number Counts

Figs 8 and 9 show our CTIO galaxy number counts in the B and R bands respectively, which are both corrected for galactic extinction using the Schlegel et al. (1998) dust maps. Our B-band number counts in the NGC agree very well, firstly with the SDSS data in the magnitude range $`16.5<`$B$`<18.5`$ and also with the MGC for $`15.5<`$B$`<18.5`$. It should be noted that all three data sets were taken in overlapping regions of sky - a 32 and 90 deg2 overlap with the MGC and SDSS data respectively in the NGC. Our CTIO data also overlaps with 163 deg2 of the 2dFGRS in the NGC and 297 deg2 in the SGC. Our SGC number counts agree extremely well with the 2dFGRS, which is important given the large overlap, but our NGC data is showing significantly more galaxies than that of the 2dF team. This could be due to real differences in the galaxy number density between the current 740 deg2 2dF field and our 297 deg2 field, or due to zero-point/scale differences in the galaxy photometry.

The shape and normalisation of our NGC number counts also agrees remarkably well with the no-evolution predictions of the low-$`q_0`$ model (see section 4.3 for model details) from B=18 to as bright as B=14. The shape of our B-band number counts in the SGC agrees well with our NGC data set, but the striking difference is that of the normalisation, which is calculated in each 0.5 magnitude interval in Table 2. We a find an average 30.7 per cent deficiency in galaxy numbers for our southern data compared to the north in the magnitude range 14$`<`$B$`<`$18. At B=19 this normalisation discrepancy has dropped to 17.3 per cent , although the galaxy incompleteness is a factor in this magnitude bin. The 2dF number counts also show a north-south difference, but only 20 per cent in galaxy numbers at B=18, with this difference virtually zero at B=19. The survey area of our SGC data was a subset of the 4300 deg2 sky covered by the APM galaxy catalogue and our SGC number counts agree very well with the Metcalfe et al. (1995) corrected APM counts despite the order of magnitude less of sky coverage on our part. Our SGC number counts also agree well with the CCD data of Bertin & Dennefeld (1997), who cover 62 and 83 square degrees at high declinations ($`|\delta |>40^{}`$ in the North and South Galactic Caps respectively. Our CTIO counts, the 2dFGRS and MGC counts and those of Yasuda et al. have all been corrected for dust using the Schlegel et al. (1998) extinction maps. The APM and DUKST counts and those of Bertin & Dennefeld are not corrected for dust and the reason for this was that the vast majority of all these observations were in the SGP where it was orginally thought dust extinction was negligible. In fact Schlegel et al. predict values of 0.03 mag. in the B-band at the SGP and an average of 0.08 mag. over the whole SGC.

Our R-band galaxy counts show a similar trend in the normalisation difference, where the number discrepancies of the SGC counts relative to those in the SGC are tabulated in Table 3. The average percentage discrepancy is 22.2 per cent in the 13$`<`$R$`<`$17 range, and again this normalisation difference drops for fainter magnitudes. At R=18 there is only a 2.5 per cent discrepancy in the galaxy numbers, although galaxy incompleteness is again a factor. The shape of both the data sets in the 13$`<`$R$`<`$17 range is consistent with the no-evolution models, but there is a slight normalisation discrepancy in that all the data sets show less galaxies compared to the models for R$`<15.5`$. There is a noticeable dip in both our number counts compared to the models between about R=13.5 and R=15.5 and there is good agreement with Bertin & Dennefeld, as in the B filter. Our data lies substantially below that of Picard et al. (1991), who cover 386 square degrees in both the NGC and SGC. In Fig. 8 our CTIO number counts have been corrected for dust using the Schlegel et al. (1998) extinction maps but none of the other data sets were de-reddened when originally published. This is because, either it was originally thought that a particular data-set suffered from negligible extinction, as in the case of the Stevenson points in the SGC, or the reddening correction was not known due to the absence of any accurate dust maps at the time of publication eg. Picard (1991).

Fig. 10 shows selected number count data with the appropriate zero-point corrections of 0.10 mag. to the 2dFGRS NGC data (for b$`{}_{J}{}^{}>`$16.0), 0.24 mag. to that of the DUKST and 0.31 mag. to the APMBGC data. The DUKST corrected data is now significantly altered, but the counts are still much lower than the q0=0.05 evolution model. We have also made a small dust correction to the APM and APMBGC data shown in Fig. 10. A standard AB=C(cosec($`b`$)-1) extinction law was originally assumed by Maddox et al. for the APM photometry, which corresponds to AB=0 at the poles and a maximum A$`{}_{B}{}^{}`$0.03 mag. at $`b=50^{}`$, averaging $`0.01`$ mag over the whole APM area. The Schlegel et al. (1998) dust maps, which are used to correct the 2dF counts and our own CTIO data, predict 0.02-0.03 mag. of extinction, even at the poles, and our average CTIO dust correction was 0.08 mag. in the SGC. We have therefore corrected the APM counts and the APMBGC data by an additional 0.08 mag (so the total correction is 0.08 mag. for the APM counts and 0.39 mag. for the APMBGC data). The original APM counts now show no galaxy number deficiency for B$`>`$18, but still indicates a large local hole at brighter magnitudes over a huge 4300 deg2 area.

The 2dF data shown in Fig 10 shows excellent agreement with our CTIO data in both galactic caps. The 0.1 mag. zero-point correction we have applied to the 2dF NGC data now means that their north-south difference agrees with ours almost exactly for B$`<`$18. This agreement with our data in each galactic cap is interesting given the larger angular areas of 740 deg2 and 1094 deg2 of the 2dFGRS NGC and SGC fields respectively. Therefore it would seem that our survey areas of 242 deg2 and 297 deg2 are fairly typical samples of the galaxy distribution in these regions of the Universe.

## 5 Number Redshift Distributions

### 5.1 The Durham/UKST N(z) Distribution

The zero-point discrepancy between the DUKST photographic plate photometry and that of our CCD based data has important consequences in terms of the DUKST number counts and n(z) distributions. We have used the “best sample” from Ratcliffe et al. (1998) using the uncorrected EDSGC photometry, taking into account the magnitude limits and completeness corrections of each of the 60 fields, in order to re-construct the DUKST n(z) distribution shown in Fig. 11. The average magnitude limit for the DUKST “best sample” over all 60 fields is bJ=16.86 and the dotted line shows the n(z) predicted using the Ratcliffe et al. (1998a) luminosity function parameters. This has been calculated using Ratcliffe’s “best sample” magnitude limit, first brightened by 0.05 mag. to take account of the DUKST photometry relative to that of the EDSGC at bJ=19.5, and secondly 0.2 mag is added in order to convert to the B Landolt system. We therefore use $`B_{lim}=16.860.05+0.2=17.01`$. In fact, this curve is equivalent to correcting all of the DUKST magnitudes by 0.24 mag. and the M value in the luminosity function by the same amount as the two effects cancel each other out. The dashed line shows a prediction with the 0.24 mag. photometry correction applied to the magnitude limit, giving B$`{}_{lim}{}^{}=17.010.24=16.77`$, but with no alterations to the original value of the M of Ratcliffe et al. (1998b).

Fig. 12 again shows the DUKST n(z) distribution but this time with predictions using the luminosity functions of Metcalfe et al. (2001) and Norberg et al. (2001). The bJ luminosity function of Norberg et al. (long dashed curve) has been calculated using 110,500 galaxies from the 2dFGRS at z=0, taking account of evolution, the distribution of magnitude measurement errors and small corrections for incompleteness within the 2dF catalogue. We have used their published values of M$`{}_{}{}^{}{}_{b_J}{}^{}`$=-19.66 (M$`{}_{}{}^{}{}_{B}{}^{}`$=-19.46) and $`\alpha =1.21`$. A $`\varphi ^{}`$=2.6x10-2Mpc-3 was chosen so that the predicted galaxy number count from the Norberg et al. luminosity function matched that of the Metcalfe et al. model at B=19.5. We have also shown a prediction from the luminosity function of Metcalfe et al. (2001) using a magnitude limit of Blim=17.01 (dotted curve) and the CTIO corrected magnitude limit of Blim=16.77 (short dashed curve).

The predictions from the two luminosity functions agree reasonably well for z$`<`$0.05 but the Metcalfe et al. model predicts slightly more galaxies at higher redshift. However, both models clearly over-predict the DUKST n(z) at all redshifts. This clear deficiency of galaxies was originally considered surprising by Ratcliffe et al. given the DUKST covers 1500 deg2 but it was thought that, because of the relatively bright magnitude limit of bJ=16.86 and shallow redshift depths, that significant large scale structure at low redshift could explain this. By taking into account the zero-point correction of 0.240 and using the Metcalfe et al. model with B$`{}_{lim}{}^{}=`$17.01-0.240=16.77, we find that this apparent underdensity of galaxies is not as large as first thought. Even so the corrected Metcalfe et al. model still over-predicts the n(z) distribution at virtually all redshifts and so we draw a similar conclusion to that of Ratcliffe et al. in that, even after our photometry correction is applied, significant large scale structure is still observed in the DUKST n(z) distribution. The dashed line in Fig 12 shows this model which indicates that the deficiency of galaxies relative to the Metcalfe et al. model is still apparent until at least z=0.1 Relative to this photometry-corrected Metcalfe et al. model the DUKST n(z) distribution shows three clear holes in the galaxy distribution in the redshift ranges 0.005$`<`$z$`<`$0.025, 0.03$`<`$z$`<`$0.055 and 0.06$`<`$z$`<`$0.09 with number discrepancy percentages of $``$40 per cent, 45 per cent and 50 per cent respectively. In Section 6 we will analyse these apparent under-densities in more detail in conjunction with the results from the 2dFGRS n(z) distribution which we discuss in Section 5.

### 5.2 The 2dFGRS N(z) Distributions

The good photometry agreement in the SGC field between us and the 2dFGRS means that we are able to use the 2dF SGC n(z) distribution in conjunction with the galaxy count model of Metcalfe et al. (2001) described in 4.3, to perform a consistent analysis of the observed deficiency of galaxies in the SGC as a function of redshift. We can also make the appropriate zero-point correction in the NGC so we are then consistent with the 2dF photometry in order to investigate the 2dF NGC n(z) distribution.

We show the 2dFGRS n(z) distributions in Figs. 13 and 14 obtained from the 100k data release (Colless et al. 2001). The mean magnitude limit of the 2dF photographic plates is bJ=19.35 and bJ=19.40 in the NGC and SGC respectively. Due to uncertainties in the effective areas and completeness, we have normalised the data such that the ratio of the predicted to the observed number of galaxies matches the corresponding ratio in the n(B) counts to the same magnitude limits. In Fig. 13 we have plotted three model curves, two of which use the Metcalfe et al. (2001) luminosity function (long dashed and solid curves) and a third which uses the luminosity function of Norberg et al. (2001) (short dashed curve) described in section 3.4.2. The long dashed and short dashed curves were both calculated using the appropriate mean magnitude limit, Blim=19.55, of the APM-based plates in the NGC (assuming B-bJ=0.2). We only show the Norberg et al. model at Blim=19.55, since any photometry correction would be also applied to the Norberg et al. M; this would leave the n(z) virtually unchanged.

The models agree at low redshift but the Blim=19.55 Metcalfe et al. model significantly over-predicts the data, particularly at the higher redshifts plotted. The solid curve, however, takes into account our zero-point correction derived in the previous section, equal to 0.1 mag. for b$`{}_{J}{}^{}>`$16, and we therefore use a magnitude limit, Blim=19.55-0.1=19.45. This model now shows excellent agreement with the observed NGC data and it is interesting that this zero-point correction is vital in order that the Metcalfe et al. model does not over-predict the data. This excellent agreement of the data with the Metcalfe et al. and Norberg et al. models at all redshifts, indicates that the galaxy distribution appears to be fairly uniform in the 2dF NGC field with no evidence for any huge under-densities as we saw for the DUKST field.

Fig. 14 shows the 2dF SGC n(z) distribution along with two model predictions using the Metcalfe et al. luminosity function (solid curve) and that of Norberg et al. (dashed curve). Our photometry comparison with the 2dF SGC data in the previous section showed up no significant zero-point error and so we used the appropriate magnitude limit of Blim=19.60 for each model. The main result of these model comparisons so far has been that the Metcalfe et al. model predicts more galaxies at higher redshift and this case is no different, but the Metcalfe et al. model shows a slightly better agreement with the data than the Norberg et al. model which under-predicts the data for z$`>`$0.14. The most striking feature of the plot however is the two clear “holes” in the galaxy distribution in the ranges 0.03$`<`$z$`<`$0.055, with an under-density of $``$35-40 per cent, and 0.06$`<`$z$`<`$0.1 where the density deficiency is $``$25 per cent. We now analyse these underdensities in the galaxy distribution in more detail.

## 6 Discussion

We have attempted to analyse the depth and angular size of the apparent “hole” in the SGC galaxy distribution using data from the APM, 2dFGRS, DUKST and 2MASS surveys. One of the major discoveries of this paper has been the large galaxy-number deficiency relative to the Metcalfe et al. model in the DUKST and 2dF n(z) distributions. In fact there are distinct similarities between the two n(z) distributions which we attempt to quantify in Fig. 15. There are two panels in these histogram plots where the upper panel shows galaxy number deficiency vs redshift and the lower panel shows Ngal/Nmodel again vs redshift. We define the percentage galaxy number deficiency as:

$$D_g(z)=100\frac{(N_{model}(z)N_{data}(z))}{N_{model}(z)}$$
(9)
where Nmodel is the number of galaxies predicted using the Metcalfe et al. (2001) luminosity function and Ndata is the number of galaxies from the appropriate survey data. In each panel, predictions for the DUKST survey are shown by the solid lines and those of the 2dFGRS by the dashed line. In the upper panel the four holes found in the DUKST n(z) distribution are clearly illustrated by the four peaks in the solid histogram with the two peaks of the dashed histogram showing the under-densities found in the 2dF SGC n(z).

The similarity between the 2dF and DUKST n(z) distributions is quite striking. The two under-densities seen in the 2dF SGC n(z) in the redshift ranges 0$`<`$z$`<`$0.055 and 0.06$`<`$z$`<`$0.1 are also clear features in the DUKST n(z). In fact, the galaxy discrepancy in the range 0$`<`$z$`<`$0.055 for the two surveys is almost the same magnitude in size with the histograms showing a very similar shape. The second under-density in the 2dF SGC n(z) is less pronounced than its DUKST counterpart, but they still cover very similar redshift ranges. The rise in the galaxy number density between the two 2dF SGC under-densities is also seen in the case of the DUKST survey. Given that the 2dF SGC field is entirely contained within the areas of sky observed for the DUKST survey we claim that the the similarities we have described in the two n(z) distributions are both artefacts of the same features in the galaxy distribution. Since the DUKST observes a larger 1500 deg2 region than the 2dFGRS (who have redshifts for galaxies covering $``$300 deg2 in the SGC at present) it could be that there is significant large scale structure observed by DUKST that is not seen by the 2dFGRS explaining the larger and more numerous galaxy number discrepancies in the DUKST n(z) distribution.

But there is evidence for an alternative explanation. For z$`>`$0.06 shown in the lower panel of Fig. 15, the DUKST survey shows significantly larger galaxy number discrepancies than the 2dFSGC n(z) relative to the Metcalfe et al. model. It is possible that galaxies can be biased tracers of the overall mass distribution and that intrinsically more luminous galaxies are then predicted to show stronger clustering properties, with regions of very high and low galaxy number densities. We suspect that the SGC might be under-dense and therefore the fact that the DUKST samples intrinsically brighter galaxies than the 2dF over all redshifts shown in Fig. 15, may mean that we are seeing the effects of bias.

Since this clearly needs further investigation, we have examined how the galaxy distribution varies with apparent magnitude. The results are shown in Fig. 16. In the upper panel we have plotted Ndata/Nmodel for the 2dFGRS SGC data relative to the Metcalfe et al. (2001) model, each with an imposed magnitude limit of bJ=16.81 (B=17.01) to match the DUKST limiting magnitude. The DUKST n(z) is indicated as in Fig. 15 by the dotted histogram for reference. It is very interesting that we now see a better agreement between the 2dF and DUKST n(z) for z$`>`$0.06. Similarly, in the lower panel we have imposed magnitude limits of 18.4$`<`$B$`<`$19.4 on the 2dFGRS SGC data and the Metcalfe et al. prediction in order to examine whether the galaxy number deficiencies seen in the bright data are compensated for by fainter galaxies. We can see from the lower panel of Fig. 16 that there is no significant effect for z$`<`$0.06, while the galaxy distribution for z$`>`$0.06 indicates a significantly reduced galaxy deficiency. This is entirely due to the different galaxy selection criterion we have used via the two magnitude limits of B$`<`$17.01 and 18.4$`<`$B$`<`$19.4, and suggests that the intrinsically brighter galaxies are clustered more strongly on $``$$`>`$ 50$`h^1`$Mpc scales, despite the fact that the amplitude of the underdensities at z$`<`$0.06 remain the same; the galaxies here are sub-L in both samples with approximately similar space densities, and so even this result may be in accordance with the notion of bias.

Fig. 17 shows our B-band galaxy number counts and our corrected APMBGC, 2dF and DUKST counts (see Fig. 10). For clarity, the bright original APM (B$`<`$17) counts have been removed since we have no reliable photometry checks using our CTIO data here, and the faint-end Metcalfe et al. corrected APM counts are shown with our corrected APMBGC counts. These counts are shown with respect to two variable $`\varphi ^{}`$ models where the value of $`\varphi ^{}`$ is a function of redshift instead of the usual constant value defined from the Metcalfe et al. (2001) B-band luminosity function. In these models the $`\varphi ^{}`$ is determined for certain low redshift ranges (z$``$zvar) by multiplying the usual Metcalfe et al. (2001) value by the factor Ndata(z)/Nmodel(z) for the bin widths shown in Fig. 15. For z$`>`$zvar the usual Metcalfe et al. (2001) $`\varphi ^{}`$ is used.

The motivation for these variable $`\varphi ^{}`$ models is to locate the structure in the n(z) which dominates the bright galaxy counts. Recall that the DUKST n(z) indicates three distinct deficiencies in the galaxy distribution below z$``$0.1. The DUKST z$`{}_{var}{}^{}<`$0.09 and z$`{}_{var}{}^{}<`$0.06 models were constructed then in order to see the effect on the predicted number counts, assuming that just the first two and then all three of these holes have significant effects on the galaxy counts.

The z$`{}_{var}{}^{}<`$0.09 model fits the bright Southern CTIO counts well but slightly underpredicts the APMBGC data. The z$`{}_{var}{}^{}<`$0.06 overpredicts the Southern CTIO counts. Since this redshift range appears to be unaffected by bias this indicates that the deficiency in the Southern CTIO counts is caused by structure to z$``$$`<`$$`0.1`$. The fact that the z$`{}_{var}{}^{}<`$0.09 model slightly underpredicts the APMBGC counts indicates that the galaxy deficiency of the hole may be less over the whole SGC than the $``$30 per cent indicated in the counts from the DUKST, 2dFGRS SGC and Southern CTIO strips. A comparison of the APMBGC counts in Fig. 17 with the homogeneous model still indicates a 25 per cent deficiency over the full 4300 deg2 of the APM Southern Survey. The conclusion from these variable $`\varphi ^{}`$ models then is that the Southern B-band counts are most consistent with there being a galaxy number discrepancy for z$``$$`<`$ 0.1, which is of order $``$25 per cent.

These results agree very well with those of Frith et al. (2003), who investigated the angular extent of the hypothesised deficiency of galaxies in the SGC using the publicly available 2MASS second incremental release data. The authors took $`5^{}`$ declination slices in the NGC and SGC with RA ranges similar to that of the 2dF regions, and plotted galaxy number counts in the K-band. In total twelve declination slices of data, each of $`300400`$ deg2, were analysed totalling $`4300`$ deg2 from the two Galactic Caps. Since the APM, 2dF SGC and DUKST survey areas overlap with the three of the southern 2MASS strips, the work done by Frith et al. is extremely relevant to the discussion in this paper. Their conclusions also show strong evidence for a significant under-density of $``$30 per cent in the galaxy number density in their SGC areas. This agrees well with the 25 per cent deficiency that we have shown in this paper is implied by the APM and APMBGC data. The DUKST field then extends further south than is currently sampled by 2MASS, to $`\delta `$=-43 and the APM area even further to $`\delta `$=-70. This then implies a hole in the SGC galaxy distribution of 100x60, which extends to z$``$0.1. This corresponds to a huge volume of space of $``$107 $`h^3`$ Mpc3 and implies significant power on large scales of $``$100-200$`h^1`$Mpc.

To illustrate how excess power on larger scales increases the chances of finding a 25 per cent galaxy deficiency over these volume sizes, we have used the 3-D analogue of equation 45.6 in Peebles (1980). By assuming a power law form of the spatial two-point correlation function out to a given scale length, we can calculate the probability of finding a given number deficiency of galaxies over a volume of space defined by a sphere of radius 150$`h^1`$Mpc. In this simplified scenario, the fluctuation in galaxy number over the expected galaxy number can be written as:

$$\frac{<(N\overline{N})^2>^{\frac{1}{2}}}{\overline{N}}=\frac{\left(1+\frac{4\pi }{V}_0^{r_s}\xi (r)r^2𝑑r\right)^{\frac{1}{2}}}{\overline{N}^{\frac{1}{2}}}$$
(10)
where $`\xi (r)`$ is the two-point spatial function and V is the volume of the sphere over which a particular survey is sampling. If we write $`\frac{<(N\overline{N})^2>^{\frac{1}{2}}}{\overline{N}}`$=$`\frac{\delta N}{\overline{N}}`$ then:

$$\frac{\delta N}{\overline{N}}=\left(\frac{1}{\overline{N}}+\frac{3}{r_s^3}\left[\frac{r_0^{1.8}r_{cut}^{1.2}}{1.2}\right]\right)^{\frac{1}{2}}$$
(11)
where we assume the galaxy correlation length in proper coordinates, r0=5.0$`h^1`$Mpc, rcut in Mpc is the length scale to which we assume the power law form of the correlation function extends, rs=150$`h^1`$Mpc and is the radius of the sphere defining our volume, $`\overline{N}`$ is the expected number of galaxies in this volume and $`\delta N`$/$`\overline{N}`$ is the expected fluctuation in the galaxy number. We assume two cases, each corresponding to a different value of rcut. In the first case rcut=10$`h^1`$Mpc. If we set $`\overline{N}`$=$`(4\pi r_s^3\overline{n})/3`$, where we assume $`\overline{n}`$=0.01 which is the mean galaxy density in units of $`h^3`$Mpc-3, then we find that $`\delta N`$/$`\overline{N}`$=0.016. This corresponds to an expected galaxy number fluctuation over our sphere of radius 150$`h^1`$Mpc of 1.6 per cent and therefore means that finding a galaxy number deficiency of 25 per cent over our sphere is a 15.6$`\sigma `$ result, although it is difficult to judge how much the $`aposteriori`$ selection is affecting this result.

However, if we assume our second case where rcut=150$`h^1`$Mpc then we find an expected galaxy number fluctation of 8 per cent. This is a 3.75$`\sigma `$ result and so a power law correlation function extending to 150$`h^1`$Mpc is starting to be more consistent with the observation of the local underdensity.

We take the $`\mathrm{\Lambda }`$CDM real-space correlation function of Padilla & Baugh (2003), which we have modelled by two power laws of $`\gamma `$=-1.7 and $`\gamma `$=-3.44 with a break at rbreak=30$`h^1`$Mpc. This overestimates the $`\mathrm{\Lambda }`$CDM $`\xi (r)`$ at all scales for r$``$$`>`$$`100`$$`h^1`$Mpc. Even so, integrating this model to rs=150$`h^1`$Mpc, we find that a 1$`\sigma `$ fluctuation is of order 4.8 per cent indicating that our 25 per cent observed fluctuation is a 5$`\sigma `$ deviation.

We conclude that there may be more power required at large scales than in the $`\mathrm{\Lambda }`$CDM $`\xi (r)`$; although biasing the galaxy distribution could improve the agreement, in standard versions of the $`\mathrm{\Lambda }`$CDM model the galaxy distribution is predicted to be unbiased on large scales. The implied excess power is also in contradiction with the currently observed forms for the APM and 2dFGRS correlation functions (Hawkins et al. 2002). Either these correlation functions have underestimated the amount of power at large scales or the galaxy distribution in the SGC is atypical of the overall distribution of galaxies. We are therefore currently reanalysing the correlation functions of the 2dFGRS to check whether any such problem exists (Frith et al., in preparation).

## 7 Conclusions

In this paper we first presented the resulting galaxy counts from our CTIO Curtis Schmidt data. The large deficiency of galaxies seen in the SGC motivated us to investigate the possible existence of a large “hole” in the SGC galaxy distribution. Using our CTIO data, covering 300 deg2 in the SGC, we were able to make the first ever detailed checks of the bright (B$`<`$17) galaxy photometry in the DUKST, 2dFGRS and APM surveys which has crucial implications for the existence of a local hole in the distribution of galaxies. Our conclusions are:

* Our B-band galaxy counts in the NGC agree extremely well with the model of Metcalfe et al. (2001) but our SGC counts shows a significant galaxy deficiency, a mean 30.7 per cent in the magnitude range 14$`<`$B$`<`$18.5.
* Good agreement is found for our NGC data with the SDSS and MGC number counts in the magnitude interval 16.5$`<`$B$`<`$18.5 and likewise for our SGC galaxy counts with the data of Bertin & Dennefeld and the APM survey in the range 16.5$`<`$B$`<`$19.
* We compared our CCD galaxy catalogue to that of the 2dFGRS in the NGC and SGC. In the NGC we find good agreement of the zero-point at bJ=16 but our galaxies are, on average, 0.13 mag. brighter at bJ=18 implying a scale error of 0.065 mag/mag. We find a mean zero-point difference of 0.1 mag. in the range 16$`<`$bJ$`<`$18. In the SGC we find no zero-point or scale errors.
* After applying this 0.1 mag. zero-point correction to the 2dF NGC photometry we then find excellent agreement in both galactic caps between our CTIO data and the 2dFGRS, who’s zero-point corrected results also imply a 30 per cent normalisation difference between the NGC and SGC galaxy counts at B=18.
* After comparing our CTIO photometry to both the DUKST and APMBGC data we found that our galaxies were, on average, brighter by 0.24 mag. and 0.31 mag. respectively.
* Our R-band galaxy counts show a normalisation difference for the NGC and SGC data of $``$22.2 per cent in the range 13$`<`$R$`<`$17.
* The 2dF and DUKST n(z) distributions show striking common structure with regard to 2 “holes” in the galaxy distribution in the redshift ranges 0.03$`<`$z$`<`$0.05 and 0.06$`<`$z$`<`$0.09, with number discrepancy percentages of 35-50 per cent , 25-60 per cent respectively.
* The DUKST survey finds significantly larger galaxy number discrepancies for z$`>`$0.06 than the 2dF survey in the SGC. This is evidence that bright galaxies may be biased tracers of the underlying mass distribution.
* Using variable $`\varphi ^{}`$ models based on the structure seen in the DUKST and 2dFGRS n(z)’s we claim that the galaxy count data is consistent with there being a hole in the SGC galaxy distribution over the whole 4300 deg2 APM area, which extends out to z=0.1.
* Taking together the deficiency in the APMBGC counts and our variable $`\varphi ^{}`$ models we conclude that there is a galaxy number deficiency of $``$25 per cent over a huge angular area of 100x60 in the SGC. The evidence is that this hole extends out to z=0.1 or 300$`h^1`$Mpc. This result agrees well with those of Frith et al. (2003)
* We show, using a simple model, that significant power is required in the two-point correlation function on very large scales if there is to be any reasonable chance of the existence of such a large hole in the galaxy distribution. The $`\mathrm{\Lambda }`$CDM model shows less excess power than required to explain the existence of such a feature in the galaxy distribution unless it is significantly biased on $``$150$`h^1`$Mpc scales. This is not expected in current $`\mathrm{\Lambda }`$CDM scenarios which are supposed to have b=1 for r$``$$`>`$ 1$`h^1`$Mpc.


format: text

text:
# 1 Introduction

## 1 Introduction

Supersymmetric Grand Unified Theory (SUSY GUT) is one of the most promising candidates which describe the interactions among elementary particles in very high energy phenomena . SUSY extension of the Standard Model provides us a solution to the hierarchy problem , and the unification of three gauge interactions in the Standard Model to GUT symmetries explains the charge quantization which is realized in nature . Moreover, these two scenarios are non-trivially consistent with each other since SUSY predicts the coincidence of the gauge coupling constants at the high energy scale , which is a necessary condition to realize the Grand Unification.

Among several candidates for the GUT gauge group such as SU(5), SO(10), and E6, SO(10) has particularly attractive features. One is the matter unification. All the quarks and leptons including right-handed neutrinos in each generation are unified to a single 16 dimensional spinor representation field. Also, SO(10) is the smallest simple GUT group which does not require a particular particle content to cancel the gauge anomaly. Yukawa unification is another interesting point. Because of the matter unification to 16 representation fields, the Yukawa-interaction terms between the matter ($`\mathrm{𝟏𝟔}_i`$) and the Higgs ($`\mathrm{𝟏𝟎}_H`$) fields in the superpotential are restricted to the form of $`W\lambda _{ij}\mathrm{𝟏𝟔}_i\mathrm{𝟏𝟔}_j\mathrm{𝟏𝟎}_H`$. Thus, all the Yukawa matrices for up- and down-type quarks, the charged leptons, and neutrinos are the same since they all originate from the above superpotential. This is not necessarily a desired situation since the above relations result in the identical masses for the fermions in each family and no flavor mixings which are completely different from the observed structure. It is, however, possible to break the above relations naturally through the non-renormalizable interactions between the matter fields and $`\mathrm{𝟒𝟓}_H`$ Higgs fields which break SO(10) by its Vacuum Expectation Value (VEV) such as $`\mathrm{𝟏𝟔}_i\mathrm{𝟏𝟔}_j\mathrm{𝟏𝟎}_H\mathrm{𝟒𝟓}_H`$, because the Yukawa interactions may feel the SO(10) breaking effect .

However, when we take into account the fact that the Yukawa matrices have hierarchical structures and they are quite different among the quark and lepton sectors, SO(10) GUT has a difficulty in reproducing such structures. Many of the recent neutrino oscillation experiments support the Large Mixing Angle MSW solution  (LMA) to the solar neutrino anomaly , and the large mixing in the second and third generation neutrinos is suggested by atmospheric neutrino oscillation experiments  and confirmed by the K2K long baseline experiment . On the other hand, all the mixing angles in the quark sector are small and masses have hierarchical structures. The magnitude of the hierarchy is enormous, e.g., the ratio of the up-quark mass $`m_u`$ to the top-quark mass $`m_t`$ is approximately $`10^5`$. Therefore, a mechanism to explain such a large hierarchy is necessary. The Froggatt-Nielsen (FN) mechanism is an elegant scenario to generate hierarchical structures in the Yukawa matrices , in which the hierarchy comes from the difference between the fundamental scale $`M`$ and a U(1)FN symmetry breaking scale $`\mathrm{\Phi }_{\mathrm{FN}}`$. The Yukawa matrices $`f_u`$, $`f_d`$ and $`f_e`$, and the neutrino mass matrix $`m_\nu `$ are given in terms of a small parameter $`ϵ\mathrm{\Phi }_{\mathrm{FN}}/M`$ and the U(1)FN charges $`Q`$ of the fermions by $`f_u^{ij}ϵ^{Q(q_i)+Q(u_j^c)}`$, $`f_d^{ij}ϵ^{Q(q_i)+Q(d_j^c)}`$, $`f_e^{ij}ϵ^{Q(l_i)+Q(e_j^c)}`$, and $`m_\nu ^{ij}ϵ^{Q(l_i)+Q(l_j)}`$, where $`i`$ and $`j`$ are the generation indices, $`q`$, $`u^c`$, $`d^c`$, $`l`$, and $`e^c`$ represent the corresponding quarks and leptons. It is known that even if we impose the SU(5) GUT relations among the U(1) charges, i.e., $`Q(q_i)=Q(u_i^c)=Q(e_i^c)Q(\mathrm{𝟏𝟎}_i)`$ and $`Q(d_i^c)=Q(l_i)Q(\overline{\mathrm{𝟓}}_i)`$, we reproduce the hierarchical structures for quarks and leptons, and an appropriate neutrino mass matrix accounting for the solar neutrino anomaly by LMA. An example of the charge assignment is $`Q(\mathrm{𝟏𝟎}_1)=3`$, $`Q(\mathrm{𝟏𝟎}_2)=2`$, $`Q(\mathrm{𝟏𝟎}_3)=0`$, $`Q(\overline{\mathrm{𝟓}}_3)=1`$, $`Q(\overline{\mathrm{𝟓}}_2)=0`$, and $`Q(\overline{\mathrm{𝟓}}_1)=0`$ with $`ϵ\lambda 0.22`$ . This type of Yukawa matrices is called lopsided family structure which gives successful masses and mixing patterns especially for the large mixing angles in the lepton sector . However, in SO(10) GUT, the above successful mechanism does not work in a simple way. The hierarchical structures in the fermion masses cannot be lopsided since the SO(10) symmetry requires $`Q(\mathrm{𝟏𝟎}_i)=Q(\overline{\mathrm{𝟓}}_i)`$. The situation does not change even if we introduce the non-renormalizable coupling to $`\mathrm{𝟒𝟓}_H`$ to avoid the SO(10) relation in the fermion masses since the hierarchy is controlled by the charges of the U(1)FN symmetry, which is not related to the SO(10) breaking. There have been attempts to avoid this problem. An often considered way is to introduce new matter multiplets of 10 dimensional representation and mix or flip $`d^c`$ and $`l`$ in the 16 and those in the newly introduced 10 multiplets by the SO(10) breaking effect . In the context of the six dimensional SO(10) GUT , it has been also proposed the introduction of extra matter fields where the hierarchy is a consequence of the volume suppression factors . In these scenario, $`d^c`$ and $`l`$ originate from the 10 representation fields other than 16, and thus the matter unification is spoiled. This is a generic problem in SO(10) GUT. Therefore, in order to solve this problem, the strong connection between the origin of flavor hierarchy and the SO(10) breaking is necessary.

In this sense, the recently proposed mechanism to generate the flavor hierarchy based on the wave-function profiles in the extra dimension is noteworthy . In the simplest scenario, our world has an $`S^1/Z_2`$ compactified extra spacial dimension, and the bulk mass terms for the bulk superfields make them localize on the branes at the orbifold fixed points . Consequently, the values of the wave functions at a brane may be suppressed by a factor of $`\mathrm{exp}(|m|\pi R)`$, where $`m`$ is the bulk mass and $`R`$ is the radius of the extra dimension. The flavor hierarchy is obtained in a similar way as the FN scenario such like $`f^{ij}\mathrm{exp}((|m_i|+|m_j|)\pi R)`$. Hebecker and March-Russell considered the scenario in SU(5) GUT and showed that the hierarchy can be reproduced with a natural parameter sets . Again, in SO(10), it seems that the SO(10) relation, i.e., the same $`m_i`$ for all the matter fields in each family, is too strong to reproduce the hierarchy. However, it is possible to break the SO(10) relations since we have additional contributions to the bulk masses $`m`$’s when SO(10) is broken by the VEV of the bulk adjoint field whose existence is ensured by SUSY in five dimension. The additional contributions are not universal to all the matter fields but proportional to the U(1)X ($``$ SU(5)$`\times `$U(1)X $``$ SO(10)) charges, so that they can change the wave-function profiles without preserving the SO(10) relations.

In this paper, we construct an SO(10) SUSY GUT model in 5-dimensional space-time in which the correct fermion-mass patterns are realized by the above mentioned mechanism. The matter fields in each family are unified to a single 16 representation field. Our scenario is compatible with already proposed doublet-triplet splitting mechanisms such as Dimopoulos-Wilczek mechanism , boundary condition , etc. We also discuss a possibility of the horizontal unification which unifies the three families to the single three dimensional representation of SU(2) group.

This paper is organized as follows: In Section 2, we construct a model and explain the mechanism. The parameter sets to realize the suitable fermion-mass patterns are discussed in Section 3. Section 4 is devoted to discussions and conclusions.

## 2 Model

In this section, we first review SUSY theory on the space-time $`M^4\times S^1/Z_2`$, and show that the 5-dimensional wave function of zero mode has an exponential shape. This mechanism is a key ingredient of the fermion mass hierarchy. Then, we discuss the SO(10) models in this setup.

### 2.1 Zero mode wave function due to bulk mass term

We consider the $`S^1/Z_2`$ compactified 5-dimensional SUSY gauge theory . For our purpose, it is convenient to write down the action in terms of superfields in 4-dimensional superspace . The action for the U(1) gauge theory with a bulk hypermultiplet is given by

$`S`$ $`=`$ $`{\displaystyle }d^4x{\displaystyle _0^{\pi R}}dy[\text{}{\displaystyle }d^2\theta ({\displaystyle \frac{1}{4}}W^\alpha W_\alpha +\mathrm{h}.\mathrm{c}.)+{\displaystyle }d^4\theta (_yV{\displaystyle \frac{1}{\sqrt{2}}}(\varphi +\overline{\varphi }))^2`$
$`+`$ $`{\displaystyle }d^4\theta (H^ce^{2gQV}\overline{H}^c+\overline{H}e^{2gQV}H)+{\displaystyle }d^2\theta (H^c(m+_y\sqrt{2}gQ\varphi )H+\mathrm{h}.\mathrm{c}.)\text{}],`$
where $`W^\alpha `$ and $`V`$ are the field strength and vector superfield associated with the U(1) gauge group, and $`\varphi `$ is the chiral superfield where $`A_5`$ is in its lowest components. The gauge transformation for the superfield is $`\varphi \varphi +1/(\sqrt{2}g)_y\mathrm{\Lambda }`$ with transformation parameter $`\mathrm{\Lambda }`$. Note that the gauge coupling constant $`g`$ has a mass dimension of $`1/2`$. The chiral superfields $`H`$ and $`H^c`$ are the components of the hypermultiplet and their U(1) charges are $`Q`$ and $`Q`$, respectively111The superfields $`H`$ and $`H^c`$ do not mean the Higgs fields here.. To be invariant under the orbifold projection, the $`Z_2`$ parity is assigned to be even for $`W^\alpha `$, $`V`$, and $`H`$, and odd for $`\varphi `$ and $`H^c`$. Also, the bulk mass parameter $`m`$ should be $`Z_2`$ odd such that $`m=m\mathrm{sgn}(y)`$.

By the Kaluza-Klein (KK) decomposition, we can find the existence of a zero mode with a localized wave function. We expand the chiral superfields $`H`$ and $`H^c`$ as follows:

$`H(y)={\displaystyle \underset{n=0}{\overset{\mathrm{}}{}}}H_n(x)f_n(y),H^c(y)={\displaystyle \underset{n=0}{\overset{\mathrm{}}{}}}H_n^c(x)g_n(y).`$ (2)
To obtain the normalized kinetic terms in the 4-dimensional effective action, the functions $`f_n(y)`$ and $`g_n(y)`$ are solutions of the following differential equations:

$`({\displaystyle \frac{d}{dy}}+m)f_n(y)=m_ng_n(y),({\displaystyle \frac{d}{dy}}+m)g_n(y)=m_nf_n(y),`$ (3)
with the normalization conditions:

$`{\displaystyle _0^{\pi R}}𝑑yf_n(y)f_m(y)={\displaystyle _0^{\pi R}}𝑑yg_n(y)g_m(y)=\delta _{nm}.`$ (4)
The zero mode wave function with correct $`Z_2`$ parity is easily found to be:

$`f_0(y)=\sqrt{{\displaystyle \frac{2m}{1e^{2m\pi R}}}}e^{my},g_0(y)=0.`$ (5)
Thus the zero mode wave function $`f_0`$ localizes exponentially at $`y=0`$ $`(\pi R)`$ for $`m>0`$ ($`m<0`$). For $`m<0`$ and $`2m\pi R1`$, since the value of the wave function at $`y=0`$ is exponentially suppressed by a factor of $`f_0(0)\sqrt{2|m|}e^{|m|\pi R}`$, the strength of the couplings between the bulk fields and the brane fields on the $`y=0`$ brane becomes weak. This effect can be the origin of the fermion mass hierarchy if the Higgs fields are confined on the brane at $`y=0`$ .

For the massive modes, the KK masses are given by

$`m_n^2=m^2+\left({\displaystyle \frac{n}{R}}\right)^2.`$ (6)
### 2.2 Gauge symmetry breaking

As we can see in Eq.(LABEL:action), the VEV of the $`\varphi `$ superfield is an additional contribution to the bulk mass $`m`$, and as a consequence the wave-function profile of the zero mode is changed . Note that the additional contribution is proportional to the U(1) charge. We identify the U(1) gauge symmetry as a subgroup of the SO(10) GUT gauge group and $`H`$ as a matter superfield of the 16 dimensional representation of SO(10), and then the different wave-function profiles are realized for the matter fields embedded in 16 since all the components do not have the same U(1) charges.

We consider the vacuum where the $`\varphi `$ superfield acquires the VEV while the matter superfields $`H`$ and $`H^c`$ do not. In order to preserve $`N=1`$ SUSY in the 4-dimensional effective theory (it ensures the configuration to be true vacuum), the VEVs of $`F`$\- and $`D`$-terms must vanish. The vanishing $`F`$-term conditions are satisfied by assuming $`H=H^c=0`$, the $`D=0`$ condition is not the same as the 4-dimensional case but as follows :

$`0=D=_y\varphi ,`$ (7)
where we denote that $`\varphi `$ is its scalar component. The contribution from the $`\varphi `$ field is due to the 5-dimensional kinetic term, i.e., the second term in Eq.(LABEL:action). The above equation is automatically satisfied if $`\varphi =\mathrm{const}.`$, however, as mentioned earlier, the $`\varphi `$ field has odd parity under $`yy`$ transformation which means that the VEV should have the form of $`\varphi =v^{3/2}\mathrm{sgn}(y)`$. Substituting this form in the above equation, we obtain the non-vanishing $`D`$-terms on the branes as follows:

$`D=2v^{3/2}\left(\delta (y)\delta (y\pi R)\right).`$ (8)
The brane localized $`D`$-terms in Eq.(8) have to be canceled by the brane localized terms in the action. The most economical way is to add the Fayet-Iliopoulos $`D`$-terms on both branes. However, it is not applicable in our context since we identify the U(1) symmetry as a subgroup of SO(10)222 The $`D`$-term cancellation by the Fayet-Iliopoulos term is possible if the gauge group is broken explicitly to SU(5)$`\times `$U(1) on the brane by the boundary conditions.. The second way is to add the brane fields charged under the U(1) symmetry. For example, if we assume the presence of chiral superfields $`S_0`$ with charge $`q`$ and $`S_\pi `$ with charge $`q`$ which are localized on the branes at $`y=0`$ and $`\pi R`$, respectively, the $`D`$-term is

$`D=_y\varphi gq|S_0|^2\delta (y)+gq|S_\pi |^2\delta (y\pi R).`$ (9)
The condition $`D=0`$ has a solution with a flat direction:

$`2v^{3/2}=gq|S_0|^2=gq|S_\pi |^2.`$ (10)
This is the usual $`D`$-flat direction in the 4-dimensional effective theory. In the vacuum with $`v0`$, the U(1) symmetry is broken by the VEVs of $`S_0`$ and $`S_\pi `$. This is an important point when we apply this gauge symmetry breaking to the SO(10) GUT breaking. The fact that the U(1) symmetry is broken indicates that the U(1) symmetry should not be included in the Standard Model gauge group. Moreover, the U(1) symmetry must be orthogonal to the Standard Model gauge group, otherwise the VEV of $`\varphi `$ induces the $`D`$-term corresponding to the Standard Model gauge group which cannot be canceled without their breaking. Therefore, the U(1) subgroup of SO(10) is uniquely determined to be so called U(1)X symmetry, and thus SO(10) is broken down to SU(5). As we see later, this U(1)X breaking yields the correct masses and mixing patterns of the fermions through the wave-function deformation as well as the Majorana masses for the right-handed neutrinos.

### 2.3 SO(10) model

We construct an SO(10) GUT model which includes the above setup. The matter fields are unified to three copies of 16 representation, $`\mathrm{𝟏𝟔}_1`$, $`\mathrm{𝟏𝟔}_2`$, and $`\mathrm{𝟏𝟔}_3`$ of SO(10), and they and their charge conjugation fields $`\mathrm{𝟏𝟔}_i^c`$ can propagate into the bulk. Also, the gauge fields including a chiral superfield $`\mathrm{𝟒𝟓}_\varphi `$ as the $`\varphi `$ component in the action Eq.(LABEL:action) are in the bulk. The Higgs field $`\mathrm{𝟏𝟎}_H`$ is introduced as a brane field (on the $`y=0`$ brane) in order to obtain the suppressed Yukawa coupling constants for the first and second generations. To break SO(10) GUT, we need $`\mathrm{𝟒𝟓}_H`$ Higgs field which has VEV along the $`BL`$ direction in the Dimopoulos-Wilczek mechanism , and the Higgs fields of $`\mathrm{𝟏𝟔}_H`$ and $`\overline{\mathrm{𝟏𝟔}}_H`$ are necessary to reduce the rank of gauge group from five to four and give the Majorana masses to the right-handed neutrinos. We put these fields on the $`y=0`$ brane, and $`\mathrm{𝟏𝟔}_H^{}`$ and $`\overline{\mathrm{𝟏𝟔}}_H^{}`$ on the $`y=\pi R`$ brane so that they can play the same roles as $`S_0`$ and $`S_\pi `$ in the above mechanism.

In this setup, the superpotential is given by

$`W`$ $`=`$ $`\mathrm{𝟏𝟔}_i^c(m_i+_y\sqrt{2}g\mathrm{𝟒𝟓}_\varphi )\mathrm{𝟏𝟔}_i`$ (11)
$`+`$ $`{\displaystyle \frac{\delta (y)}{M}}\left[\lambda _{ij}\mathrm{𝟏𝟔}_i\mathrm{𝟏𝟔}_j\mathrm{𝟏𝟎}_H+{\displaystyle \frac{\stackrel{~}{\lambda }_{ij}}{M}}(\mathrm{𝟏𝟔}_i\overline{\mathrm{𝟏𝟔}}_H)_\mathrm{𝟏}(\mathrm{𝟏𝟔}_j\overline{\mathrm{𝟏𝟔}}_H)_\mathrm{𝟏}\right]`$
$`+`$ $`\delta (y)V_0(\mathrm{𝟒𝟓}_H,\mathrm{𝟏𝟔}_H,\overline{\mathrm{𝟏𝟔}}_H,\mathrm{𝟏𝟎}_H,\mathrm{})`$
$`+`$ $`\delta (y\pi R)V_\pi (\mathrm{𝟏𝟔}_H^{},\overline{\mathrm{𝟏𝟔}}_H^{}),`$
where $`i=1,2,3`$ are the generation indices and $`M`$ is a parameter with mass dimension one. The first term is the required form from the SUSY in five dimension. We take a flavor basis where the bulk mass terms $`m_i`$ are diagonalized. The Yukawa coupling constants $`\lambda _{ij}`$ are dimensionless quantities and the interaction terms with $`\stackrel{~}{\lambda }_{ij}`$ induce the Majorana masses for the right-handed neutrinos after the SO(10) breaking. We omit the other possible contractions which are irrelevant for the low-energy effective theory.

The potential $`V_0`$ needs to have an appropriate form for the Dimopoulos-Wilczek mechanism. According to the simplest model in Ref. , it is required to exist another $`\mathrm{𝟏𝟎}_{2H}`$, an additional pair of $`\mathrm{𝟏𝟔}_{2H}`$ and $`\overline{\mathrm{𝟏𝟔}}_{2H}`$, and several singlet fields333It is difficult to forbid the dimension five operators which cause the proton decay in SO(10) model with the Dimopoulos-Wilczek mechanism. The suppression of these operators is discussed in Ref. .. On the $`y=\pi R`$ brane, we can write the arbitrary potential $`V_\pi `$ for the $`\mathrm{𝟏𝟔}_H^{}`$ and $`\overline{\mathrm{𝟏𝟔}}_H^{}`$, and in general, there are a number of vacua where SO(10) is broken. The Standard Model singlet components $`N_H`$ and $`\overline{N}_H`$ ($`\mathrm{𝟏𝟔}_H`$ and $`\overline{\mathrm{𝟏𝟔}}_H`$), and $`N_H^{}`$ and $`\overline{N}_H^{}`$ ($`\mathrm{𝟏𝟔}_H^{}`$ and $`\overline{\mathrm{𝟏𝟔}}_H^{}`$) acquire VEVs after minimizing the potential $`V_0`$ and $`V_\pi `$. Since these singlets are charged under U(1)X subgroup of SO(10), there is a non-trivial condition from the $`D`$-flatness which is given by

$`0=D_{\mathrm{U}(1)_\mathrm{X}}=`$ $`\delta (y)\left[v^{3/2}+5g_X(|N_H|^2|\overline{N}_H|^2)\right]`$ (12)
$`\delta (y\pi R)\left[v^{3/2}5g_X(|N_H^{}|^2|\overline{N}_H^{}|^2)\right],`$
where $`g_X`$ is the coupling constant of the U(1)X gauge interaction normalized such that the charge of $`N_H`$ is $`5`$, and $`v^{3/2}`$ is the VEV of the $`\mathrm{𝟒𝟓}_\varphi `$ field in the U(1)X direction. Along the flat direction in Eq.(12), we obtain the vacuum where $`v0`$ as seen in the previous subsection.

Even with the potential $`V_\pi `$, there are massless modes because of the global SO(10) symmetry in the $`y=\pi R`$ sector. We assume the modes are stabilized by the SUSY breaking effect. The minimal set of the massless modes are a pair of $`\mathrm{𝟏𝟎}`$ and $`\overline{\mathrm{𝟏𝟎}}`$ of SU(5) and a Standard Model singlet since the VEVs of $`N_H^{}`$ and $`\overline{N}_H^{}`$ break SO(10) to SU(5), although it requires additional Higgs fields. The massless modes will obtain the masses of the order of TeV. Alternatively, the singlet mode can be stabilized without SUSY breaking effect. When we gauge the U(1) symmetry under which only $`\mathrm{𝟏𝟔}_H^{}`$ and $`\overline{\mathrm{𝟏𝟔}}_H^{}`$ are charged such as $`\mathrm{𝟏𝟔}_H^{}:1`$ and $`\overline{\mathrm{𝟏𝟔}}_H^{}:1`$, the constraint from the $`D`$-flat condition is given by

$`0`$ $`=`$ $`D_{\mathrm{U}(1)^{}}`$ (13)
$`=`$ $`\delta (y)\left[\xi _0+v^{3/2}\right]\delta (y\pi R)\left[\xi _\pi +v^{3/2}g^{}(|N_H^{}|^2|\overline{N}_H^{}|^2)\right],`$
where $`\xi _0`$ and $`\xi _\pi `$ are the Fayet-Iliopoulos $`D`$-terms of the U(1) symmetry on the branes at $`y=0`$ and $`y=\pi R`$, respectively, $`v^{3/2}`$ is the VEV of the $`\varphi `$ component associated with the U(1) symmetry, and $`g^{}`$ is the gauge coupling constant. If $`\xi _0\xi _\pi `$, we obtain $`|N_H^{}|^2|\overline{N}_H^{}|^20`$, so that $`v0`$ is assured by Eq.(12). The singlet mode is eaten by the U(1) gauge field in this case.

In the vacuum with $`v0`$, the wave functions for the matter fields $`\mathrm{𝟏𝟎}_i`$, $`\overline{\mathrm{𝟓}}_i`$, and $`\mathrm{𝟏}_i`$ are deformed in a different way because of the $`\sqrt{2}g\mathrm{𝟏𝟔}_i^c\mathrm{𝟒𝟓}_\varphi \mathrm{𝟏𝟔}_i`$ terms in the superpotential (11). The bulk mass terms after the SO(10) breaking is modified as $`m_im_i\sqrt{2}g_XQ_Xv^{3/2}`$, where $`Q_X`$ is the U(1)X charges of $`\mathrm{𝟏𝟎}:1`$, $`\overline{\mathrm{𝟓}}:3`$, and $`\mathrm{𝟏}:5`$. This is the direct connection between the origin of the flavor hierarchy and the SO(10) breaking. As we see in the next Section, this modification gives a realistic fermion-mass and mixing hierarchy. The violation of the unwanted SU(5) relation in the Yukawa matrices $`f_d=f_e^\mathrm{T}`$ is obtained from the effect of $`\mathrm{𝟏𝟔}_i\mathrm{𝟏𝟔}_j\mathrm{𝟏𝟎}_H\mathrm{𝟒𝟓}_H`$ operators .

We can also consider the 5-dimensional models where the gauge symmetry is broken by the boundary conditions . First, we discuss the 5-dimensional models on the space-time $`M^4\times S^1/(Z_2\times Z_2^{})`$, where the $`Z_2`$ and $`Z_2^{}`$ break SO(10) down to SU(4) $`\times `$ SU(2) $`\times `$ SU(2) gauge symmetry and 4D $`N=2`$ SUSY down to $`N=1`$, respectively. In this scenario, the SO(10) gauge symmetry is explicitly broken down to SU(4) $`\times `$ SU(2) $`\times `$ SU(2) on the brane at $`y=0`$ . We do not have to worry about the doublet-triplet splitting problem since we can introduce the Higgs field as $`H:(\mathrm{𝟏},\mathbf{2},\mathbf{2})`$ representation field and thus there is no color-triplet Higgs from the beginning . The SO(10) breaking Higgs fields $`\mathrm{𝟏𝟔}_H`$ and $`\overline{\mathrm{𝟏𝟔}}_H`$ also need not to be the full multiplets and we assume the existence of $`\mathrm{\Phi }_H:(\overline{\mathrm{𝟒}},\mathbf{1},\mathbf{2})`$ and $`\overline{\mathrm{\Phi }}_H:(\mathrm{𝟒},\mathbf{1},\mathbf{2})`$, and their VEVs break the gauge symmetry down to the Standard Model gauge group. The matter fields are also decomposed to $`\mathrm{\Psi }_L^i:(\mathrm{𝟒},\mathbf{2},\mathbf{1})`$ and $`\mathrm{\Psi }_R^i:(\overline{\mathrm{𝟒}},\mathbf{1},\mathbf{2})`$ which are required to originate from the different $`\mathrm{𝟏𝟔}`$ dimensional fields in the bulk due to the $`Z_2`$ projection. By SU(4) $`\times `$ SU(2) $`\times `$ SU(2) invariant but SO(10) non-invariant interactions, we obtain the violation of the unrealistic SU(5) relation of $`f_d=f_e^\mathrm{T}`$ between the Yukawa matrix of the down-type quarks $`f_d`$ and that of the charged leptons $`f_e`$. For example, the following SO(10) non-invariant interaction terms do not give a contribution to $`f_d`$ but to $`f_e`$ after the Standard Model singlet components of $`\mathrm{\Phi }_H`$ and $`\overline{\mathrm{\Phi }}_H`$ acquire VEVs:

$`W{\displaystyle \frac{\delta (y)}{M}}{\displaystyle \frac{a_{ij}}{M^2}}(\mathrm{\Psi }_L^i\mathrm{\Phi }_H)_{(\mathrm{𝟏},\mathrm{𝟐},\mathrm{𝟐})}(\mathrm{\Psi }_R^j\overline{\mathrm{\Phi }}_H)_{(\mathrm{𝟏},\mathrm{𝟏},\mathrm{𝟑})}H_{(\mathrm{𝟏},\mathrm{𝟐},\mathrm{𝟐})},`$ (14)
where $`a_{ij}`$ are the dimensionless coupling constants. Including these contributions, we do not have any relation between the down-type-quark and lepton Yukawa matrices. Since the contributions should be comparable to the SO(10) invariant terms of $`\mathrm{\Psi }_L\mathrm{\Psi }_RH`$ in order to reproduce the fermion masses, the parameter $`M`$ is required to be the same scale as the VEV of $`\mathrm{\Phi }_H`$. Therefore, we take all the dimensionfull parameters to be of the order of the GUT scale $`10^{16}`$ GeV.

In addition, we can discuss the scenario where the $`Z_2`$ and $`Z_2^{}`$ break 4D $`N=2`$ SUSY down to $`N=1`$ and SO(10) down to SU(5) $`\times `$ U(1)X, respectively. Because of the $`Z_2^{}`$ projection, we introduce six copies of $`\mathrm{𝟏𝟔}`$ fields in the bulk. In this model, $`\mathrm{𝟏𝟎}_i`$ and ($`\overline{\mathrm{𝟓}}_i`$ and $`\mathrm{𝟏}_i`$) come from the different $`\mathrm{𝟏𝟔}`$’s, and then, can have different bulk masses. So, similar to SU(5) models in Ref. , one can generate the fermion-mass hierarchy by choosing a parameter set $`m_i`$ ($`i=16`$) for six $`\mathrm{𝟏𝟔}`$ representation matter fields. The doublet-triplet splitting problem can be solved by Dimopoulos-Wilczek mechanism. Essentially speaking, this is an SU(5) model but not SO(10) model, and then the matter unification is spoiled. Thus, we will not discuss it in this paper.

## 3 Fermion masses

We consider the fermion masses in this scenario with the assumption that all the components of the Yukawa matrices on the brane are of order unity. We also assume that neither SO(10) nor SU(5) relations among the Yukawa matrices maintain because of the symmetry breaking effects on the brane as discussed in Subsection 2.3. The hierarchical structures in the observed fermion masses and mixings are realized by means of the wave-function profiles of the zero modes discussed in Subsection 2.1. The lopsided family structure is given through the effect of non-vanishing $`v^{3/2}=\mathrm{𝟒𝟓}_\varphi `$ in the bulk.

The observed hierarchy can be approximately expressed by the power of the Cabibbo angle $`\lambda 0.22`$. The CKM matrix is given by

$`V_{\mathrm{CKM}}\left(\begin{array}{ccc}1& \lambda & \lambda ^3\\ \lambda & 1& \lambda ^2\\ \lambda ^3& \lambda ^2& 1\end{array}\right).`$ (18)
The ratios of the masses at the GUT scale  are given by

$`m_u:m_c:m_t\lambda ^7:\lambda ^4:1,`$ (19)
$`m_d:m_s:m_b\lambda ^4:\lambda ^2:1,`$ (20)
$`m_e:m_\mu :m_\tau \lambda ^5:\lambda ^2:1.`$ (21)
Also, the ratio of the neutrino masses in the second and third generations is estimated from the ratio of the $`\mathrm{\Delta }m^2`$ of the solar and atmospheric neutrino oscillations. If we assume the hierarchical mass pattern, the ratio is given by

$`m_{\nu _2}:m_{\nu _3}\sqrt{{\displaystyle \frac{\mathrm{\Delta }m_{\mathrm{solar}}^2}{\mathrm{\Delta }m_{\mathrm{atm}.}^2}}}\lambda ^{12}:1,`$ (22)
where we used the LMA solution for the solar neutrino oscillation which requires nearly bimaximal mixing in the Maki-Nakagawa-Sakata (MNS) matrix  as follows:

$`V_{\mathrm{MNS}}\left(\begin{array}{ccc}1/\sqrt{2}& 1/\sqrt{2}& ϵ\\ 1/2& 1/2& 1/\sqrt{2}\\ 1/2& 1/2& 1/\sqrt{2}\end{array}\right),`$ (26)
where $`ϵ`$ is a small parameter.

The Yukawa couplings in the effective theory are the products of the Yukawa couplings and the values of the wave functions on the brane at $`y=0`$ ($`f_0(0)`$). From Eq.(5), the $`f_0(0)`$’s are given by

$`f_0(0)^{i,(r)}=\sqrt{{\displaystyle \frac{2|M|a_i^{(r)}}{1e^{2a_i^{(r)}c}}}}\{\begin{array}{cc}\sqrt{2|Ma_i^{(r)}|}e^{|a_i^{(r)}|c}\hfill & (a_i^{(r)}<0)\hfill \\ \sqrt{2|M|a_i^{(r)}}\hfill & (a_i^{(r)}>0)\hfill \\ \sqrt{|M|/c}\hfill & (a_i^{(r)}=0)\hfill \end{array},`$ (30)
where the superscript $`r`$ stands for the belonging SU(5) representations such as $`\mathrm{𝟏𝟎}`$, $`\overline{\mathrm{𝟓}}`$, and $`\mathrm{𝟏}`$, and $`i`$ is the generation index. The parameter $`c`$ is $`|M|\pi R`$. The $`a_i^{(r)}`$ parameters are dimensionless quantities expressed in terms of $`\mu _im_i/|M|`$ and $`kg_Xv^{3/2}/|M|`$ as follows:444 In the case where SO(10) is broken down to SU(4) $`\times `$ SU(2) $`\times `$ SU(2) by the boundary condition as discussed in the previous section, the left-handed multiplet $`\mathrm{\Psi }_L^i`$ and right-handed one $`\mathrm{\Psi }_R^i`$ may have different bulk masses $`m_i^L`$ and $`m_i^R`$. However, as we see later, only the case with $`m_i^Lm_i^R`$ gives the correct mass patterns, and this is suitable to the horizontal unification and the gauge coupling unification.

$`a_i^{(\mathrm{𝟏𝟎})}=\mu _i+k,a_i^{(\overline{\mathrm{𝟓}})}=\mu _i3k,a_i^{(\mathrm{𝟏})}=\mu _i+5k.`$ (31)
The parameter $`k`$ represents the effect of SO(10) breaking and the SO(10) relation of $`a_i^{(\mathrm{𝟏𝟎})}=a_i^{(\overline{\mathrm{𝟓}})}=a_i^{(\mathrm{𝟏})}`$ restores in the limit of $`k0`$. This parameter plays an important role to reproduce the fermion-mass hierarchy. Hereafter, we take $`|\mu _3|=1`$ which can be done without loss of generality. Defining the dimensionless suppression factors $`n_i^{(R)}f_0(0)^{i,(r)}/\sqrt{|M|}`$, we obtain the formulae of the Yukawa matrices:

$`f_u^{ij}=\lambda _u^{ij}n_i^{(\mathrm{𝟏𝟎})}n_j^{(\mathrm{𝟏𝟎})},f_d^{ij}=\lambda _d^{ij}n_i^{(\mathrm{𝟏𝟎})}n_j^{(\overline{\mathrm{𝟓}})},f_e^{ij}=\lambda _e^{ij}n_i^{(\overline{\mathrm{𝟓}})}n_j^{(\mathrm{𝟏𝟎})}.`$ (32)
The matrices $`\lambda _u`$, $`\lambda _d`$, and $`\lambda _e`$ are the Yukawa matrices on the brane at $`y=0`$ and their components are of order unity. For neutrinos, the Yukawa matrix and the Majorana mass matrix for the right-handed neutrinos are given by

$`f_\nu ^{ij}=\lambda _\nu ^{ij}n_i^{(\overline{\mathrm{𝟓}})}n_j^{(\mathrm{𝟏})},M_R^{ij}=\stackrel{~}{M}_R^{ij}n_i^{(\mathrm{𝟏})}n_j^{(\mathrm{𝟏})}.`$ (33)
Again $`\lambda _\nu `$ and $`\stackrel{~}{M}_R`$ are the Yukawa matrix and the Majorana mass matrix ($`\stackrel{~}{M}_R=\stackrel{~}{\lambda }\overline{\mathrm{𝟏𝟔}}_H^2/M`$) on the brane at $`y=0`$, respectively. Since the neutrino masses are given through the seesaw mechanism $`m_\nu =f_\nu M_R^1f_\nu ^\mathrm{T}`$ , the suppression factors of $`n_i^{(\mathrm{𝟏})}`$ cancel in this formula and thus the neutrino mass matrix is expressed by

$`m_\nu ^{ij}=(\lambda _\nu \stackrel{~}{M}_R^1\lambda _\nu ^\mathrm{T})^{ij}n_i^{(\overline{\mathrm{𝟓}})}n_j^{(\overline{\mathrm{𝟓}})}v_2^2,`$ (34)
where $`v_2174`$ GeV is the VEV of the Higgs field which couples to the up-type quarks and neutrinos.

The formulae in Eqs.(32) and (33) are the same as those of the FN mechanism except the relations in Eq.(31). The number of the free parameters is reduced to be four which is $`\mu _1`$, $`\mu _2`$, $`c`$, and $`k`$ compared to the case of FN mechanism in which six charges (three $`Q(\mathrm{𝟏𝟎}_i)`$ and three $`Q(\overline{\mathrm{𝟓}}_i)`$) and a parameter $`ϵ=\mathrm{\Phi }/M`$ are adjustable. Interestingly, we see in the following that the four parameters are sufficient to reproduce the Yukawa structures.

Our goal is to reproduce the lopsided family structures given as follows:

$`f_u\left(\begin{array}{ccc}\lambda ^6& \lambda ^5& \lambda ^3\\ \lambda ^5& \lambda ^4& \lambda ^2\\ \lambda ^3& \lambda ^2& 1\end{array}\right),f_df_e^\mathrm{T}\left(\begin{array}{ccc}\lambda ^4& \lambda ^3& \lambda ^3\\ \lambda ^3& \lambda ^2& \lambda ^2\\ \lambda & 1& 1\end{array}\right),m_\nu \left(\begin{array}{ccc}\lambda ^2& \lambda & \lambda \\ \lambda & 1& 1\\ \lambda & 1& 1\end{array}\right).`$ (44)
We can see that this form of Yukawa matrices reproduce the hierarchical structures in Eqs.(18)-(26) when we take into account the $`O(1)`$ ambiguities. The large mixing in the 1-2 generation of the neutrinos is realized when the determinant of the 2-3 submatrix in $`m_\nu `$ is $`O(\lambda )`$, which is also consistent with the mass relation in Eq.(22). As mentioned in Introduction, the form is suitable to the FN mechanism. Similarly, in our mechanism, the above structures are given when the suppression factors have the form of

$`n_i^{(\mathrm{𝟏𝟎})}(\lambda ^3,\lambda ^2,1),n_i^{(\overline{\mathrm{𝟓}})}(\lambda ,1,1).`$ (45)
The above type of suppression factors is obtained in the following way. First, we consider the $`n_i^{(\mathrm{𝟏𝟎})}`$ factors. The last component in $`n_i^{(\mathrm{𝟏𝟎})}`$ of $`O(1)`$ is given if $`\mathrm{𝟏𝟎}_3`$ is localized on the $`y=0`$ brane. From Eq.(30), this is achieved by

$`a_3^{(\mathrm{𝟏𝟎})}=\pm 1+k0,`$ (46)
where $`\pm `$ represent the undetermined sign of $`\mu _3`$. The second component in $`n_i^{(\mathrm{𝟏𝟎})}`$ of $`O(\lambda ^2)`$ is, in contrast, given by the $`\mathrm{𝟏𝟎}_2`$ localization on the $`y=\pi R`$ brane such that

$`a_2^{(\mathrm{𝟏𝟎})}c=(\mu _2+k)c\mathrm{log}\lambda ^23.0.`$ (47)
The first component in $`n_i^{(\mathrm{𝟏𝟎})}`$ of $`O(\lambda ^3)`$ determine the relative size of the $`\mu _1`$ parameter as follows:

$`(a_1^{(\mathrm{𝟏𝟎})}a_2^{(\mathrm{𝟏𝟎})})c=(\mu _1\mu _2)c\mathrm{log}{\displaystyle \frac{O(\lambda ^3)}{O(\lambda ^2)}}1.5.`$ (48)
Next, the $`n_i^{(\overline{\mathrm{𝟓}})}`$ factors constrain the parameters. The second and last components in $`n_i^{(\overline{\mathrm{𝟓}})}`$ of $`O(1)`$ give

$`a_3^{(\overline{\mathrm{𝟓}})}=\pm 13k0,a_2^{(\overline{\mathrm{𝟓}})}=\mu _23k0.`$ (49)
The suppression of the first component in $`n_i^{(\overline{\mathrm{𝟓}})}`$ is given by

$`a_1^{(\overline{\mathrm{𝟓}})}=(\mu _13k)c\mathrm{log}\lambda 1.5.`$ (50)
From the above conditions, the sign of $`\mu _3`$ is determined to be $`\mu _3=+1`$ and the $`kc`$ parameter is approximately given by

$`kc0.75.`$ (51)
Once we postulate the value of $`k`$ or $`c`$, we can determine the values of $`\mu _1`$ and $`\mu _2`$ by Eqs.(47) and (48). For example, for $`k0.75`$ and $`c1`$, we find $`\mu _13.75`$ and $`\mu _22.25`$. We schematically explain this mechanism in Fig.1. Without SO(10) breaking, $`\mathrm{𝟏𝟎}_i`$ and $`\overline{\mathrm{𝟓}}_i`$, of course, have the same profiles. However, once the SO(10) breaking effect is turned on, the profiles are deformed in such a way that the Yukawa structures become the currently observed form.

When we think about the origin of the bulk mass parameters $`m_i`$, there is a possibility that the origin is the spontaneous breaking of an SU(2) horizontal symmetry555Similarly, one can consider the U(1) flavor symmetry.. In fact, the suppression of the $`n_1^{(\overline{\mathrm{𝟓}})}`$ is not necessarily important. Even if $`n_1^{(\overline{\mathrm{𝟓}})}=O(1)`$, the Yukawa matrices are in a good shape if we take into account the $`O(1)`$ ambiguities in the Yukawa couplings on the brane. The neutrino masses and mixings are explained by the anarchy scenario . In that scenario, the form of the suppression factors is given by

$`n_i^{(\mathrm{𝟏𝟎})}(\lambda ^4,\lambda ^2,1),n_i^{(\overline{\mathrm{𝟓}})}(1,1,1).`$ (52)
Although this type of suppression factors gives a small value $`\lambda ^2`$ to the Cabibbo angle, we think that the observed large value of the Cabibbo angle is due to the $`O(1)`$ ambiguity. An interesting possibility arises in this case. We find a parameter set of $`k1`$, $`c3`$, $`\mu _11`$, $`\mu _20`$, and $`\mu _3=1`$, which gives above suppression factors. With the relation $`\mu _1=\mu _3`$ and $`\mu _2=0`$, there is a possibility that the origin of the bulk mass parameters $`m_i`$ are the VEV of the bulk adjoint field of the SU(2) horizontal symmetry in which the three generations of the $`\mathrm{𝟏𝟔}`$ representation matter fields are unified to $`\mathrm{𝟑}`$ representation. As discussed in the previous section, the VEV of the bulk SU(2) adjoint field gives rise to the $`D`$-terms on the branes and they are canceled by the $`D`$-terms given by the VEVs of the localized fields in the fundamental representation on the branes. The VEVs break the SU(2) symmetry completely and the couplings to the matter fields may induce off diagonal components of the Yukawa couplings which are absent in the SU(2) symmetric limit. This possibility is a special feature in SO(10) GUT and can not be realized in SU(5) GUT. The existence of the U(1)X breaking is essential to give the different bulk masses to $`\mathrm{𝟏𝟎}`$ and $`\overline{\mathrm{𝟓}}`$ fields.

Finally, we comment on the neutrino masses. If we assume that all the dimensionfull parameters are of the order of $`10^{16}`$ GeV, we find the mass of the heaviest neutrino in Eq.(34) is one order of magnitude smaller than the favored value by the atmospheric neutrino data of $`0.05`$eV. However, this is not a serious problem because the $`O(1)`$ ambiguities easily reduce the Majorana masses of the right-handed neutrinos and/or increase the Dirac neutrino Yukawa couplings.

Although the right-handed neutrino masses do not affect the neutrino masses, they might be important from the viewpoint of leptogenesis . In the scenario with Eq.(51), our predictions on the lightest right-handed neutrino masses are given by

$`M_R^1M_R^{11}10^7M10^9\mathrm{GeV}.`$ (53)
This value is compatible with the leptogenesis scenario . Of particular interest is the case with the horizontal symmetry. With the values of $`k1`$ and $`c3`$, we obtain the right-handed neutrino masses $`M_R^1110`$ GeV, $`M_R^21001000`$ GeV, and $`M_R^310^{56}`$ GeV. Since the lightest right-handed neutrino does not enter the thermal equilibrium before the electroweak phase transition, the partial lepton asymmetry carried by the particles except for the right-handed neutrino is converted to the baryon number by the sphaleron process even if there is no net $`BL`$ asymmetry .

We assumed in the superpotential in Eq.(11) that the right-handed neutrinos acquire masses from the VEVs of $`y=0`$ brane fields. Alternatively, we can change the source of their masses to the $`y=\pi R`$ brane. However, in this case, Eq.(34) is not applied, and then it leads too small neutrino masses or non-suitable forms for the LMA solution.

## 4 Discussions and Conclusions

We consider the scenarios with the lopsided family structure in SO(10) GUT by using the 5-dimensional wave function profile. We can reproduce the fermion masses and mixings including the LMA solution without introducing extra matter fields. The key point is that we directly connect the SO(10) breaking effect to the origin of fermion-mass hierarchy, which is difficult in the usual FN scenario.

We used the bulk mass terms as a origin of the hierarchy. The SO(10) breaking effect gives additional contributions to the bulk mass terms so that the Yukawa matrices do not obey the unrealistic SO(10) relation $`f_uf_df_ef_\nu `$. Moreover, we find the possibility of SU(2) horizontal unification of three generations, and that is possible in SO(10) rather than SU(5) GUT.

“SU(5) or SO(10)?” is an interesting question. From the viewpoint of the doublet-triplet splitting problem, SO(10) might have an advantage since it has an intrinsic possibility to realize the doublet-triplet splitting by the Dimopoulos-Wilczek mechanism, but it is not easy. Since we need to reduce the rank of gauge group, the Higgs sector becomes complicated. It is, therefore, difficult to judge from this viewpoint. The fermion-mass hierarchy is another interesting viewpoint. We can say that SU(5) is better than SO(10) in this case. The SU(5) relation of $`f_df_e^\mathrm{T}`$ is approximately good and the hierarchy is compatible to the FN mechanism. On the other hand, SO(10) symmetry is too strong and it is required to introduce extra-matter fields. However, the concept of the matter unification is too beautiful to be thrown away. In this sense, the wave-function localization is an attractive possibility which enables a matter unified SO(10) GUT to be realistic and opens up a further possibility of horizontal unification.

Considering the SUSY breaking in this scenario is also interesting. Severe experimental limits of the processes with Flavor Changing Neutral Current (FCNC) and CP violation restrict the sfermion mass matrices to be some specific forms. In our scenario, there is an interesting possibility if SUSY is broken on the $`y=\pi R`$ brane. Because of the wave-function profiles, only the first and second generation fields can feel the SUSY breaking effect strongly. This is the desired situation for the effective SUSY scenario  in which the sfermions of the first and second generations have large soft masses of the order of 10 TeV so as to avoid the constraints from the FCNC and CP violating processes, and the third generation sfermions are as light as O(100) GeV for naturalness. Consideration in this direction is interesting, although we need the mechanism to suppress the gaugino masses and to generate the $`\mu `$-term. The gauge mediation  is another candidate where the $`\mathrm{𝟏𝟔}_H^{}`$ and $`\overline{\mathrm{𝟏𝟔}}_𝐇^{}`$ fields on the $`y=\pi R`$ brane may be the messenger fields.

## Acknowledgments

We would like to thank Hooman Davoudiasl, Carlos Pena-Garay, and Scott Thomas for useful discussions. The work of R.K. was supported by DOE grant DE-FG02-90ER40542. The work of T.L. was supported by the National Science Foundation under Grant No. PHY-0070928.