It was a commercial flop (probably because it cost $700 in today's money and could do 4 only somewhat-interesting things), so even though it looks crazy today, safety wasn't the reason it was pulled off the market. That said, the U-238 samples came in tiny glass vials that you were instructed not to break. But if you did break them and then ate one of the chunks of Uranium it could do some pretty serious damage inside of you if it wasn't expelled quickly. But that point you'd also have broken glass and a rock-swallowing habit and probably other problems that would kill you before any kind of radiation-induced cancer would.

IIRC it was not inert and over time with enough exposure, was radioactive enough to potentially cause health detriments.