[test][math] Lower precision for polynomial test

When the roots are very close together the inversion becomes ill-conditioned. Exclude these cases from the random generation, test some corner cases with close roots and decrease the tolerance for root comparison.

Needs backport to releases 3.4