The church that was built in 1732 stood at Grytten for about 100 years. During this time, most of the forested areas around the church and the local rivers was cut down which caused some of the river banks to shift and the erosion of the areas along the river. Over time, the church began to be threatened by shifting sands along the river banks. Shifting sand was a major problem in the area and large parts of the vicarage (which according to tradition included substantial farmland) was turned into a sand desert. In 1824, a sand dune "9 cubits tall" (about ) was located just east of the church and the church yard was largely covered in sand. The building itself suffered damages from sand penetrating cracks in the walls. So, in 1828, the church building was dismantled and the church site was moved to Veblungsnes, about to the northwest, along the fjord. The church was rebuilt at the new site using some building materials from the old church as well as some new materials. The new church was not based on the old church, but rather it had a new octagonal design with an onion dome, looking nothing like the old church. The building has an altarpiece from the previous church building which dates back to the 13th century. The new church was consecrated on 27 September 1829. In 1868, the church porch was repaired and enlarged. In 1965–1967, the church was enlarged using plans by Torgeir Suul.
