No manor is mentioned here in the Domesday Survey. The parish was at that time included in Wissett, and was then held by Alan, Earl of Brittany and Richmond.

It no doubt shortly after the Conquest was included in the lands given to Ralph Guader (de Waer or Wayer), Earl of Norfolk and Suffolk, who forfeited his earldom or treachery to the Conqueror, whom he attempted to depose, being joined in his design by Walthorp, the great Earl of Northumberland, Roger de Britolis, Earl of Hereford, whose sister he had espoused, and others of distinction. He took the opportunity on his wedding day, and when it is said they were heated with wine, to disclose to the conspirators his plans. When the convivial party had recovered from their intemperance, a good number repented, and refused to join the rising, which was quickly suppressed by Odo, Bishop of Bayeux, and Geoffrey, Bishop of Worcester. The Earl of Norfolk deserted his followers, and fled into Brittany, leaving them to their fate. The Castle of Norwich was subsequently besieged, and his Countess obliged to surrender, but she was suffered to go beyond the sea. The Earl ultimately assumed the Cross, and joined the expedition under Robert Curthose to Jerusalem against the Turks, where it is said, " he afterwards became a pilgrim, and died a great penitent."

He left issue two sons, Ralph and Alan, and a daughter Amicia, married to Robert, Earl of Leicester, son of Robert, Earl of Mellent, but the property at Rumburgh was confiscated and granted to Alan, Earl of Brittany and Richmond, surnamed Rufus of Ferganut, by reason of his red hair. He was the grandson of Geoffrey, Duke of Bretagne, and commanded the rear of the army at the Battle of Hastings, being joint Commander of the Second Division of the Norman Army.

He married Constance, daughter of King William the Conqueror, but died without issue in 1089, and was succeeded by her brother Alan Niger, 2nd Earl of Richmond and Earl of Brittany. Alan Niger died in 1093, without issue, and was succeeded by his brother Stephen, 3rd Earl of Richmond and Earl of Brittany.

He married Havise, Countess of Guincamp, and dying the 13th April, 1137, was succeeded by his eldest son, Alan, surnamed the Savage, 4th Earl of Richmond and Earl of Brittany, who took an active part in the contest between King Stephen and the Empress Maud on the side of the former. In 1142 he took the Castle of Lincoln with considerable treasure from Ranulph, Earl of Chester, by scaling the walls at night. He also garrisoned the Castle of Hotun in Yorkshire, then part of the Bishop of Durham's possessions, and made great spoil at Ripon upon the demesnes and tenants of the Archbishop of York.

He founded Jorvaulx Abbey, Richmondshire, and the Abbey of Coetmaloen, Quimper, Bretagne, probably to atone for his wickedness, for he is described as a most deceitful, wicked individual. He married Bertha, daughter of Conan III., " the fat " Duke of Bretagne, and dying 30th March, 1146, was succeeded by his eldest son, Conan le Petit, 5th Earl of Richmond, also styled Duke and Count of Bretagne.

He was patron of St. Mary's Abbey, York, and founder of Rownay Nunnery, co. Hertford. He married Margaret, daughter of Henry, Earl of Huntingdon, and sister of Malcolm IV., King of Scotland, by whom he had an only daughter, Constance, who married Geoffrey Plantagenet, 4th son of Hen. II., and was mother of the unfortunate Arthur, said to have been put to death by his uncle, afterwards King John.

Constance died 4th Sept. 1201, but the lordship of Rumburgh, according to the Davy MSS., passed about this time to the family of de Vallibus or de Vaux. Certainly in 1286 Sir John de Vallibus had view of frankpledge here, but there seems no evidence that he held this particular lordship. The probability is that he held the advowson only, and that this holding has been confounded with the manor. Sir John de Vallibus's daughter, Petronilla, wife of Sir William de Nerford, held the advowson in 1316, and it was vested in 1364 in John de Nerford, the son of Thos. de Nerford, and probably grandson of the above Petronilla.'He died in 1364, and his inquisition p.m. shows that he held the advowson of the Priory Church of Rumburgh and the Manor of Wysete, with the appurtenances of the King in capite by knight's service. The probability is that the Manor of Rumburgh was included in the grant towards the foundation of Rumburgh Cell and Priory made by Alan, the 4th Earl of Richmond, who died in 1146. Though it may be questionable when the lordship passed to the priory there can be no doubt it was held by the priory until 1528, when it was one of the small priories suppressed before the general dissolution, and was granted by the King,30th Dec. 20 Hen. VIII. to Cardinal Wolsey for his college at Ipswich. Inquisitions upon lands of the suppressed monastery at Rumburgh will be found amongst the State Papers, 19 Hen. VIII.

Cardinal Wolsey, by a conveyance in 1529, granted the manor to St. Mary's College, Ipswich. On the disgrace of Wolsey, the grant was resumed, and the manor went to the Crown. It seems to have been granted to Robert Downes, who had licence in 1530 to alienate the same to Thomas, Duke of Norfolk,' who forfeited it on his attainder in 1546, and the following year the lordship was granted to John, Earl of Warwick, who had licence the same year to alienate to Sir Edward North. The manor is included in the inquisition p.m. of Lionel Talmash, who died 25th June, 1552, leaving Lionel his son and heir.

Thomas, Duke of Norfolk, was restored in 1553, when the lordship again vested in him, and passed on his death 18th July, 1554, to his grandson, Thomas, 4th Duke of Norfolk, who was beheaded in 1572, for his communication with Mary, Queen of Scots. His son Philip, Earl of Arundel, however, seems to have succeeded to the lordship, for in 1585 he had licence to-alienate the manor to John Holland and others. The sale was carried into effect the same year. Amongst the Suffolk Charters in the Bodleian is an assignment II Eliz. by the Duke of Norfolk to John " Hollande " of the remainder of a lease granted by one Chandler of the manor and church, and an additional 21 years. John Holland had licence to alienate the manor in 1611 to Paul Bayning, sen., one of the Sheriffs of London in the time of Eliz., who died in 1616, and was succeeded by his son and heir, Sir Paul Bayning, created Baronet 24th Sept. 1612, elevated to the peerage 27th Feb. 1627, as Baron Bayning, of Horkesley-Bentley, in the county of Essex, and advanced to the rank of Viscount Bayning, of Sudbury, in the county of Suffolk, on the 8th March in the same year. He married Anne, daughter of Sir Henry Glemham, of Glemham, and dying at his house in Mark Lane, in the City of London, 29th July, 1629, was succeeded by his son and heir, Paul Bayning, 2nd Viscount, who married Penelope, only daughter and heir of Sir Robert Naunton, Knt., master of the Court of Wards and Liveries.

He died 11th June, 1638, leaving two daughters only, Anne married to Aubrey de Vere, Earl of Oxford, and Penelope, married to the Hon. John Herbert, youngest son of Philip, 4th Earl of Pembroke, 1st Earl of Montgomery.

The manor appears to have been taken by the elder daughter and her husband, the Earl of Oxford. It does not appear who became lord of the manor later, but the Court Rolls show that William Peck held a court on the 22nd Feb. 1722, and Richard Gipps held his first court 27th Nov.1728, and his last court 5th May, 1731.

In 1731 the manor was held by William Cobbold, a wealthy Quaker, of Brandon, who purchased from Richard Gibbs, and held his 1st court 12th March, 1731, and other courts 10th Nov. 1732 and 14th Sept. 1739.

William Cobbold held his last court 21st Nov. 1746, and his will is dated the 29th April same year. Under it the manor passed to Samuel Jessup the elder, a Quaker, of Leiston Abbey (who held his first court 16th May, 1747), for life, with remainder to his son, Samuel Jessup the younger, and his (Samuel the elder's) daughter Sarah in fee. On the death of Samuel Jessup the elder the manor passed to his son and heir, Samuel Jessup the younger, and then to his sister Sarah Jessup, and after her to her brother Daniel Jessup, who held courts 1st Feb. 1787, 5th March, 1787, 17th Dec.1790, and died in 1813. Sarah Jessup held her first court 29th Nov. 1813, and her last court 28th Nov. 1827. The 28th May, 1828, Phillis Weebing, widow, John Manby, and John Grimsey held the lordship, and had their first court 28th May that year and their last court 5th Aug. 1841, since which date all proceedings relating to properties in the manor have been out of court. In 1830 on the 17th June, the manor was offered for sale at the Great White Horse, Ipswich. The fines and quit rents were then stated to be annually £40 or thereabouts. With it was offered the capital farm called Rumburgh Abbey Farm, being in Rumburgh and All Saints, comprising 348a. 2r. 16p., of which about 115a. were old pasture. The whole was stated to be free from great and small tithes, and held under a lease at the rent of £510 per annum.'The property evidently did not sell, for in 1855 the manor was vested in the executors of Mrs. Weebing, and in 1877 it was sold to Charles Henry Capon, whose mortgagees sold it in 1887 to Joseph Beaumont, on whose death in 1889 it vested in George Frederick Beaumont, of the Lawn, Coggeshall, Essex.