Depth quality - flat surface is curved


Hello everyone,

Recently, I was trying to use depth information to reconstruct a point cloud. As far as I know, it should be pretty straightforward: you read the depth files as a matrix 240x320 of unsigned 16-bit integers.

I am modifying Scanner app to get depth information. I am using STStreamConfigDepth320x240 and storing depthFrame.shiftData.

Then in python I am doing this:

import numpy as np
import sys
from cv2 import imwrite

def main():
    if len(sys.argv) < 3:
        print('Usage: python ./ FILE OUTFILE.PNG')
     srcpath, outpath = sys.argv[1:]
    x = np.fromfile(srcpath, dtype=np.uint16)
    x[x == 2047] = 0
    x = x.reshape((240, 320))
    imwrite(outpath, x)

if __name__ == '__main__':

I can assure you, that I double checked the cv2’s imwrite and it saves the PNG image with bitdepth 16.

Then I used this script:

(overall I’m hoping to use PCL library for my project)
providing the camera angles for the structure sensor: vertical 45 and horizontal 58, according to this topic: Structure Sensor’s Angle of View

My problem is shown below: even for one frame there are some very weird angles of the items I scanned. In the first example u can see that a floor I pictured turns out to be heavily bent in the point cloud. Basically, there is a can of coke on the floor.

I was moving camera from right to left and I didn’t apply any transformation to point cloud. Here it is how it looks when multiple frames are combined (all of them are bent):

This is data captured by sensor(uint16 png image):

With high contrast:

The brighter, the further it is. The first depth picture is very dark because all the values are somewhere around ~700 whereas obviously 16-bit PNG file make it just about 1/100 of the brightness (700/65535).

I suspected that there may be something wrong with the angles I provide or focal lengths (I’ve also tried using focal length) etc needed to calculate the correct placements of points in the 3d space. I still suspect it should have something to do with that, since I believe that the depth itself is properly calculated. Is there a way to sensibly debug it?

Do you suspect what may be the problem? Could someone provide any guidance on what errors did I actually make and how to fix this?