[fix][bboxtree] Fix point in tetrahedron method

* The normal vector has to be normalized to compute the right distance
* The distance epsilon needs to be scaled with a characteristic
  length, now the edge length, not the distance of the point from
  the origin (is possibly zero or very large).