computer vision - Align depth image to RGB image -

computer vision - Align depth image to RGB image -

i trying generate point cloud using images captured kinect python , libfreenect, couldn't align depth info rgb info taken kinect.

i applied nicolas burrus's equation 2 images turned farther away, there wrong code:

cx_d = 3.3930780975300314e+02 cy_d = 2.4273913761751615e+02 fx_d = 5.9421434211923247e+02 fy_d = 5.9104053696870778e+02 fx_rgb = 5.2921508098293293e+02 fy_rgb = 5.2556393630057437e+02 cx_rgb = 3.2894272028759258e+02 cy_rgb = 2.6748068171871557e+02 rr = np.array([ [0.999985794494467, -0.003429138557773, 0.00408066391266], [0.003420377768765,0.999991835033557, 0.002151948451469], [-0.004088009930192, -0.002137960469802, 0.999989358593300 ] ]) tt = np.array([ 1.9985242312092553e-02, -7.4423738761617583e-04,-1.0916736334336222e-02 ]) # uu, vv indices in depth image def depth_to_xyz_and_rgb(uu , vv): # z value in meters pcz = depthlookup[depths[vv , uu]] # compute x,y values in meters pcx = (uu - cx_d) * pcz / fx_d pcy = (vv - cy_d) * pcz / fy_d # apply extrinsic calibration p3d = np.array( [pcx , pcy , pcz] ) p3dp = np.dot(rr , p3d) - tt # rgb indexes p3d should match uup = p3dp[0] * fx_rgb / p3dp[2] + cx_rgb vvp = p3dp[1] * fy_rgb / p3dp[2] + cy_rgb # homecoming point in point cloud , corresponding color indices homecoming p3d , uup , vvp

is there did wrong? help appreciated

first, check calibration numbers. rotation matrix approximately identity and, assuming calibration frame metric, translation vector says sec photographic camera 2 centimeters side , 1 centimeter displaced in depth. approximately match setup? if not, may working wrong scaling (likely using wrong number characteristic size of calibration target - checkerboard?).

your code looks right - re-projecting pixel of depth photographic camera @ known depth, , projecting in sec photographic camera @ corresponding rgb value.

one think check whether using coordinate transform in right direction. iirc, opencv produces [r | t], using [r | -t], looks suspicious. perhaps meant utilize inverse, [r' | -r'*t ], utilize apostrophe mean transposition.

computer-vision kinect openni depth openkinect