Does anyone know of the origin of this word?
It feels like it must be a loan word from another language as I can't find the semantic connection to 名 and 堂。Any ideas?
Although the exact origin is not determined, it's considered an inherently Chinese compound word.
1) 明堂 (mainstream view)
The mainstream view is that 名堂 comes from 明堂. In short, 明堂 is where ancient kings do important things, most importantly ancestor and gods worships. 堂 is the name for this type of building and 明 is used attributively.
明堂 has many different names. "夏后氏曰世室，殷人曰重屋，周人曰明堂". The specification of the building is also not clear from historical documents. People have a lot of contradicting guesses. Because 名堂 has so many names and specifications, people associate it with 名目 and 花样, which are two of the main meaning of modern 名堂.
In Tang Dynasty, 武则天 used a huge amount of resources to build a 明堂. People complained about it, which is probably why 名堂 is often given a negative tone. Also because the blueprint is unfounded, 名堂 is sometimes associated with 乱来. Current 搞什么名堂 may have root in 造什么明堂.
The mixed use of 明 and 名 has a long history and is also seen in many other words.
Another view goes against the first one. The main reason is two-fold. Firstly, the modern use of 名堂 came much later than the building 明堂. Secondly, the place where 名堂 is first recorded to become widely used is 江浙 Wu area rather than the traditional ruling area of 中原.
By this view, 名堂 came from 堂名 in Kun Opera.
This view tries to align the rising time and area of 名堂 with that of Kun Opera. I find it quite far-fetched, though.
This is found on the Baike page. However, no source is added.
People used to have many "names", 名、字、号, and they also tend to call themselves by the names of their 居室 house, 书房 study, 堂屋 the room to worship ancestors, etc. This gives rise to the meaning of 花样多、有讲究、繁琐不理解.
While this view is good for mnemonics, I won't give it much etymological credit unless a source is added.