Here are some pairs which frequently appear:
三 and 二/两: 三三两两, 三心二意, 三天两头, 三长两短.
三 and 五:三令五申, 三五成群, 三番五次, 三纲五常.
四 and 八:四平八稳, 四通八达.
三 and 六/九:三头六臂, 三六九等, 三教九流, 三公九卿, 三跪九叩.
五 and 十:五光十色, 一五一十.
千 and 百/万:千奇百怪、千方百计、千辛万苦、千恩万谢.
You may notice that:
(1) 三 is often used. In classical Chinese (文言文), 三 not only means the exact number of "three", but also "several". It's vague and thus widely used to describe several or some.
(2) Very often, a number-pair consists of a number and its multiple (四 & 八, 五 & 十, 三, 六 & 九).
As a former Chinese elementary school student with rich experience of being forced into playing "name as many as possible chengyu with numbers!" games, I would say that in most cases, there are just no rules to follow. That's because, first, some chengyu are originated from historical stories or incidents, e.g. 过五关,斩六将 (from a story of Guan Yu. He literally went through five passes and killed six), 七擒七纵 (story of Zhuge Liang). These numbers are not changeable.
Second, there are just too many chengyu. Let's welcome 三 again: 举一反三, 三心二意, 三生三世, 朝三暮四, 三五成群, 三头六臂, 三七二十一, alas none with 八 (though there're 三八妇女节, and gossip woman, 三八), 三教九流.
It simply goes well with everyone.
BTW, though I didn't really enjoy the game mentioned above, it is good practice if you are interested in chengyu. You see, I can still recall loads of them without referring to dictionaries. :)