Moyse's Hall was built in the latter half of the 12th century, probably in about 1180. It is not known who built it, but the builder must have been wealthy. At this period, most houses were built of timber with wattle and daub infill. The use of a combination of stone and flint indicates a well-to-do builder, especially in Bury St Edmunds where stone had to be transported from some distance.

There are two main theories about who built Moyse's Hall. The first is that it was built for commercial and domestic use. Its situation on the marketplace suggests that the owner may well have been involved in commerce, with a stone-built undercroft providing security for goods and money. There is a tradition, first recorded in 1771, that Moyse's Hall was built by a Jew, and this is possible for a number of reasons. Firstly, there was a Jewish community in Bury St Edmunds for about fifty years until 1190. On Palm Sunday 1190, fifty-seven Jews were killed in riots in the town, and later that year Abbot Samson expelled the Jewish community from the town. Secondly, Christians in that period were forbidden to lend money at interest, but they often needed to borrow it and turned to the, Jewish community to provide it. Some Jews grew wealthy by lending money, wealthy enough to build stone houses. Thirdly, it has been suggested that the name Moyse comes from Moses. However, Mose and Moyse are common Suffolk surnames: two of the men who took part in the riots of 1327 had the surname Mose. There has been a general tendency to attribute Norman stone town houses to Jewish builders despite the absence of any direct evidence. Moyse's Hall might equally well have been built by a Gentile merchant.

The other theory is that Moyse's Hall was built by the abbey of Bury St Edmunds. Bury St Edmunds was a monastic borough, and the abbey exercised a high degree of control over the town. Jocelin of Brakelond's chronicle of the abbey describes how, in around 1198, Abbot Samson built some stone houses and bought others, which he gave to the schoolmaster to use as lodgings for poor clerks. It is possible that Moyse's Hall is one of these stone houses. Also, Moyse's Hall dates from the same period as the completion of the West Front of the abbey church and uses the same stone, Barnack. However, there is no other evidence that Moyse's Hall was built by the abbey, so the identity of the builder remains a mystery.