I am desperately stuck and would appreciate some help and guidance from the community, and apologise in advance for the lengthy post I hope you persevere in reading through it!
What I am trying to achieve
The ultimate goal of my application is to analyse surfaces, looking for anomalies and then performing calculations on these anomalies in order to make decisions. I first tried to use the Scanner app to simply scan a surface with a defect and then look at the resultant mesh, however the mesh just doesn’t have the resolution I need. To do this analysis I am going to need a dense Point Cloud of the surface and therefore need to roll my own app.
My first attempt…
…was kind of successful, in that I was able to use code posted in these forums to extract the data from a depth frame and save to a Point Cloud file. Using the 640x480 resolution depth image gives a potential maximum number of points 307,200 (if all are not NaN). However, this does not give me a sufficiently detailed Point Cloud dataset.
My second attempt…
…was a complete failure. I am capturing multiple frames of depth data (from different angles) and then processing them into separate Point Cloud files. When I load these Point Cloud files into an application they all appear on top of each other and create a total mess. I am assuming this is because the extract routine is creating the 3D points in the reference of the camera (using the intrinsics posted in these forums) and so will be the same for every depth frame.
So, I think I need to be able to supply some form of translation to the processed depth frame 3D points so they get correctly oriented to the viewpoint of the camera. Trouble is I don’t know where to start:
- What role does STDepthFrame.glProjectionMatrix play in applying transformations to the depth frame data, or does it?
- What role does STDepthFrame.colorCameraPoseInDepthCoordinateFrame play in applying transformations to the depth data, or does it?
- I tried to check these values for each frame I captured, but they never changed - why not?
- Is it because I need to use the STTracker in order for the STDepthFrame.colorCameraPoseInDepthCoordinateFrame to be updated for each frame based on how the camera has moved over time?
Apologies for all the questions but I’m really struggling to figure all this out. I am thinking that I need to take the following approach:
- Capture a depth frame along with the camera pose matrix
- Capture another depth frame along with the camera pose matrix
- Calculate the difference in camera poses from 1 and 2 to get a transformation matrix
- Apply the transformation matrix calculated to the depth data from the second frame so that it is positioned correctly, relative to the first depth frame
I just don’t know how to get the necessary data from the structure sensor to do the calculations!