Different ways to estimate relative pose during visual servoing

In order to obtain relative pose (pdHp) between current and desired pattern pose (cHp and cHpd) in camera frame, we can:

Method 1: Find rigid transformation between two set of pattern points (X_p and X_pd) in camera frame directly using a findRigidTransform() function described here.

Method 2: Calculate cHp and cHpd using the findRigidTransform() as the pattern points in local coordinate is known. The relative pose pdHp is calculated as:

pdHp = inv(cHpd) * cHp

Method 3: Calculate cHp and cHpd using the cv::solvePnP() function which finds an object pose from 3D-2D point correspondence. The relative pose is then calculated same as method 2.

According to the results, Method 1 is least stable and accurate as the estimated relative pose varies a lot and very different from the other two methods.
Method 2 and 3 have similar results in which Method 3 seems better since the rotation part is closer to Identity when the relative pose should be the same.

Summary: Method 3 is the best for the time being.