Orders of Magnitude adds depth to this episode by giving Kirk and Scotty a shared history that lets Scotty know that Kirk would never give this order lightly. When considering the morality of the order in this instance, you have to remember that both planets are engaging in large-scale slaughter (if not genocide) already, but have become desensitized to it due to how "civil" it has become. Under such conditions, the war could easily continue another 500 years or more, and claim not just hundreds of millions but BILLIONS of lives.

One could further argue that by wiping out life on one planet, Kirk automatically saves countless lives on the other, simply by finally bringing the war to an end. So even if the "bluff" (Kirk's not really bluffing, but he IS hoping he won't have to follow through on his threat) gets called, General Order 24 would still result in saving at least as many lives as it ends.