[test][math] Lower precision for polynomial test
When the roots are very close together the inversion becomes ill-conditioned. Exclude these cases from the random generation, test some corner cases with close roots and decrease the tolerance for root comparison.
Needs backport to releases 3.4