The archaeological and architectural remains in the parish of East Down attest to its varied history. Ancient occupation is indicated by the tumuli on the higher land. The Church and some farms date back to Mediaeval times and the current Manor was built in the eighteenth century. The diverse industries of the countryside such as malting and milling can be seen in the buildings around the area, though rarely, of course, in their current uses.

In many ways the community reached its peak of vitality in the nineteenth century. Around 1800 the Church house was rebuilt, in 1823 the National School started and a Sunday School was added in 1827. In 1851 the population peaked at 455. The Board School , now the village hall, was opened in 1877, the Church was extensively "restored" in 1888 and in 1901 the Methodist Chapel and School was built. East Down includes the hamlets of Churchill, Shortacombe, Brockham and Clifton.

East Down was also home to the Northcote family, linked to the Earls of Iddesleigh. Northcote is now a farmhouse, but history tells us it was once a Domesday manor. East Down Manor House which can be seen from the Pyne Arms car park, was formerly the seat of the Pine family, later known as the Pine-Coffins. It is a beautiful early Georgian house, still a private residence, and the pub is named after this heritage link.

The church within the grounds of East Down Manor House is dedicated to St. John the Baptist and has a fascinating history and is worth a visit.

The pub used to be the village Post Office until the 1950s when it opened as a public house.