Pevsner doubts the building having been a true castle or indeed having been fortified at all. Few remains survive on the site now occupied by a farmhouse and farmbuildings called "Colcombe Abbey Farm". (The appellation "Abbey" appears to have no historical significance as no Abbey is known to have occupied the site.) Only one small building, now converted to a house with Grade II listing, displays signs of ancient construction, namely its mullioned stone window. Hoskins reported this to contain "a splendid kitchen with a fireplace nearly 20 feet across with a private room above". A wide and shallow leat runs under the present farmhouse, suggesting either a former moat, as tentatively proposed by Pevsner,Pevsner, N., & Cherry, B., Buildings of England: Devon, London, 2004, pp. 273–74 or perhaps a mill. The top Tudor-shaped arch of an ancient stone window frame with heraldic escutcheons on either side, from which the armorials have been worn away, survives set into a low parapet wall on a bridge spanning the leat, in front of the front door of the farmhouse. The site is a terrace with a panoramic view of the town of Colyton, situated on a low hill on the opposite side of a shallow valley. The Devon historial Polwhele stated it to have been ruinous in his own time. The topographer Rev. John Swete visited Colcombe on 26 and 27 January 1795 and painted two watercolours and made a description in his journal as follows:Gray, Todd (Ed.), Travels in Georgian Devon: The Illustrated Journals of the Reverend John Swete, 1789–1800, 4 Vols., Vol.2, 1998, Tiverton, pp. 110–111 Having dined here (i.e. Colyton) about five I quitted it for Colcombe Castle, which lay about half a mile from the town. Having by a bridge crost the River Coly, I turned to the left and quickly came to the ruins of this once capital mansion. Perhaps little if any of the edifice wherein once the Courtenay Earls of Devon resided is now to be seen...Of this house of Colcombe, which Sir William (Pole) called a Castle are the present ruins. The whole of the are seen in the preceding sketch (i.e. dated 26 January 1795) and consisting of the interior part of the building are nearly the same as the print given in the "History of Devon" (i.e. by Polwhele). What was once the Forum at Rome is now become a cow-market, and the castle of the Earls of Devon and of the antiquary Sir Wm Pole is now converted into a farmer's dung-court, such is the transition from splendour to meaness. And from the pride and honour of strength to the lowest degradation and decay! On the western side, separated by a lane, is the house in which the farmer dwells who rents the estate of Colcombe of Sir J. de la Pole and which was probably built from the ruins of the castle. Standing by the door of it I took the preceding sketch (i.e. dated 27 January 1795) which will give some notion of the front and which seems to have been the principal one with an aspect to the west commanding the beautiful scenery of the expanse of pasture beneath through which ran the River Coly, of the hilly woodlands in Shute Park to the right and the beautiful object of Colyton to the left the situation had been happily chosen. It had in view many picturesqu beauties and placed sub aprico colle (Ovid, Elegy V;Text i.e. "under a sunny hill") was well calculated for a family residence".
