All Mangekyou skills are genjutsu

Why wasn't the Sharingan's genjutsu ability used much in the 4th Ninja War?

Using genjutsu on an enemy is pointless if they have allies nearby who can infuse the chakra to remove them from it. Genjutsu wouldn't work on Bee as his bijuu Gyuki would easily get him out, as he did when Bee was trapped in Sasuke's genjutsu for a moment.

If Tobi applied it to Naruto, Bee would get him out easily. Additionally, Naruto already showed during the Kazekage Rescue arc that he knows how to cancel genjutsu (although it didn't work). Now he was much stronger after learning the sage mode and the bijuu mode. In short, the chances of a genjutsu being successful on Naruto are very slim.

Also, Tobi now had a Rinnegan who he thought was much more powerful and wanted to try him out in combat, which he did with the so-called "Jinchuriki of Six Paths" technique.

As for Madara, his reincarnation did not go according to his plan. He wasn't meant to catch the tail animals, but even if he did, he didn't have the Gedo Mazo to seal them in.

Since he was reborn, Madara has spent more time demonstrating his powers and complaining that the current generation of ninja are so weak compared to himself and Hashirama than actually fighting. Casting a genjutsu on "weak" ninja is probably not as rewarding for one's pride as dropping meteors with mokuton techniques, susanoo, and the like. :) :)