In the early stages of the Church's residence the land was subdivided into one lot containing the gardener's house (originally known as 'The Bungalow') and another lot comprising the main house, other outbuildings and grounds. The Bungalow was held in trust by The Perpetual Trustee Company Ltd with the title intended to revert to the Catholic Church on the death of the Resch's gardener. Some alterations were made to the house for the Catholic Church by architects S. G. Hirst & Kennedy. The former ballroom was converted for use as a chapel-of-ease (named St Edmunds in honour of Edmund Resch Jr) that seated 200 and had standing room for about 60 people. Alterations were made to improve services in 1965 and some redecoration but throughout this period the furniture appears to have changed little from what the Resch family had left. That same year, 1964, Cardinal Gilroy moved into Swifts as the official residence of the Archbishop of Sydney. During the first-ever Papal visit to Australia in 1970, Pope Paul VI was accommodated at Swifts as the guest of Cardinal Gilroy. Pope John Paul II also stayed at Swifts during his 1986 visit (hosted by Cardinal Clancy).
