Let's break it down:
不按套路
套路 usually indicates a frame of reference, a set of rules ("routine" may be correct for some sports, like martial arts, or weight-lifting exercises; or the "every day routine"), and the term is often associated to playing cards or games. So 不按套路 means literally "not according to the rules", but not in the sense of cheating; more in the sense of "out of the box" as in "thinking out of the box", "unconventional", "unexpected".
As if when playing Monopoly (大富翁) you land on one of the expensive lots and even if you could, you don't purchase it, contrary to what other players would do in the same situation.
出牌
And as a matter of fact we have 出牌，so 不按套路出牌 is something like "playing unconventionally". It doesn't refer to a specific (card) game. It's just used metaphorically here.
的
Just the noun modifier.
脑回路
As you correctly note, 回路 means "circuitry", "circuit", as in 电回路。 The variant 脑回路 here is also a relatively common term to indicate, by extension, the "circuits in your brain"
Putting it all together:
不按套路出牌的脑回路
I like your translation "unconventional brain circuitry". The meaning is that Zhang Kankan 张侃侃 the woman, is kinda "unconventional" in her line of though.
As a matter of fact the woman says “你的衣服很香” to Tan Jin 谈晋，who is a male.
And Tan Jin is taken by surprise, he thinks that this sentence is unconventional, coming from her, because it's normally said by a male to a female, and not the other way around (he says 不要随便说男生的台词).
So he feels a little bit 无奈，or “我服了你了” toward Zhang Kankan unconventional approach, which is elegantly rendered as 甘拜下风, "whatever", "I raise white flag".
