[test][math][invertCubicPoly] Increase tolerance. If fails try once with more...

[test][math][invertCubicPoly] Increase tolerance. If fails try once with more iterations to increase precision.
4 jobs for !2599 with fix/invert-poly-doc-and-test
latest merge request