Announcing Structure SDK 0.8.1: Improved Trackers and Turntable Support


#1

Update (August 1, 2018):

We’re announcing Structure SDK 0.8.1 - a follow-up maintenance update to Structure SDK 0.8. Thank you to community members who identified a few things that only required quick fixes.

Issues Addressed:

  • Fixed a bug with STDepthFrame and now developers can correctly access shiftData.

  • Fixed an issue with StructureUnityUBT and StructureUnityAR packages where sample scenes would display a “Tracking Lost” Message.

  • Updated the sample apps with new camera intrinsic conventions to fix an issue where STColorFrame was reported with a height and width of 0.

    • Please note, If AVCaptureConnection.cameraIntrinsicMatrixDeliveryEnabled is not used for newer devices, you will see the following exception thrown:
      *** Terminating app due to uncaught exception AVCameraCalibration, reason: We want to use AV Intrinsics, but we're not getting them! Make sure that you have set conn.cameraIntrinsicsMatrixDeliveryEnabled = YES on the AVCaptureConnection object (before you commit the AVCaptureSession configuration). ***
    • For more details, please see line 145-153 in the ViewController+Camera.mm file within the Scanner sample project.

Known Issues:

  • Mesh export creates MTL + OBJ even when there is no texture to export

We’re pleased to announce the release of Structure SDK 0.8!

This is our second major update for 2018 with a focus on the newly enhanced tracker capabilities including new scene-specific tracking options that adds turntable support!

With one additional line of code, Structure SDK 0.8 allows you to easily configure your own Structure Sensor application to allow for objects to be scanned while the Structure Sensor is stationary, such as when the object is on a turntable.

Structure SDK 0.8 also includes the following enhancements:

  • Fixed exporting issue where the meshes were imported upside down or Y-side down.
  • Sample Apps are now all currently using autolayout.
  • Added the ability to see camera intrinsics through STIntrinsics.
  • Added an easy way to see which SDK version is currently being used with currentSDKVersion(void).
  • Added the ability for scene-specific tracking options in STTracker.

For a full list of changes in this update, please see the Full Release Notes further below.

Download

Download Structure SDK 0.8.1

Full Release Notes

New and Legacy Trackers

Through our rigorous tests, the new tracker has been shown that this improves tracking in most cases.

To make things easy the improved tracking capabilities are automatically turned on, so there is nothing that you need to do extra to see these improvements with your application.

Legacy (pre 0.8.0) tracking behavior can be specified by setting kSTTrackerLegacyKey to YES in the tracker options.

Scene specific Tracking options and Turntable Support

We’ve added a new key to STTracker called kSTTrackerSceneTypeKey. Use STTrackerSceneTypeObjectOnTurntable specifically to enable turn table support.

We have also added STTrackerSceneTypeObject and STTrackerSceneTypeRooms as options to use for this key.

Below are two examples of objects scanned on a turntable. The first example shows the quality of a scan with the key STTrackerSceneTypeObject, while the second shows the quality of a scan with the key STTrackerSceneTypeObjectOnTurntable.

STTrackerSceneTypeObject VS STTrackerSceneTypeObjectOnTurntable:

For a video example of scanning with turntable support enabled versus scanning with turntable support disabled, please have a look here.

New Export Key to Enable Y-up Mesh View

You are now able to export meshes in the more natural way with X is right, Y is up, and Z is inward. This makes it much easier to edit your scans when exported into a 3D Graphics Program, like MeshLab.

Default STMesh writeToFile:_
VS
kSTMeshWriteOptionUseXRightYUpConventionKey = YES:


Viewed in Meshlab

API Changes

  • Improved trackers are now available. Legacy (pre 0.8.0) tracking behavior can be specified by setting kSTTrackerLegacyKey to YES in the tracker options.
  • STIntrinsics are now available in the public SDK. These can be called on STColorFrame and STDepthFrame objects by calling [frame intrinsics].
  • Added kSTMeshWriteOptionUseXRightYUpConventionKey to be able to export meshes with the convention that X is right, Y is up, and Z is inward, which is a more natural way to view meshes (they are normally X right, Y down, Z outwards). Note that this is still a right-handed coordinate frame.
  • All sample apps (sans the unity ones, of which the UI is controlled in the Unity settings) now use autolayout. All the applications can be converted to work on iPhone displays, however, may require some work to do so.
  • Adds a function currentSDKVersion(void) to the SDK, which allows users to query what version of the SDK they are currently using. It returns a struct with four members (Major, Minor, Patch, version) which give the major number, minor number, patch number, and version string of a given SDK release.
  • Adds a key to STTracker called kSTTrackerSceneTypeKey. This allows for scene-specific tracker settings. Currently supports three modes: STTrackerSceneTypeObject, STTrackerSceneTypeObjectOnTurntable, STTrackerSceneTypeRooms. The key defaults to STTrackerSceneTypeObject.This text will be hidden

Scanning with a turntable?
#5

Hi,

Thanks for this release.

It seems that the Room Capture sample app is not working on an iPad Pro 10.5" with iOS 11.3.1 installed.

Any idea why?


#6

I’ll be taking a look into this as soon as I can.

Could you be a little more descriptive of the issues you are having with your 10.5-inch iPad Pro when you try to build the Room Capture on it?

Does it happen on other devices or just on your 10.5-inch iPad Pro? What about other versions of iOS?

Thanks!


#7

Hi Anthony,

I only have 10.5-inch iPads at hand with this version right now to check.
When starting a scan, the green mesh is not displayed, and when pressing stop scan the app crashes.
In our own app, we noticed that the STColorFrame that is delivered with a synced depth frame from the SDK has a width/height of 0, even though its sample buffer is a valid image.

Thanks!
Bar


#8

Thanks for the additional info! We, obviously, try to make sure that all of our sample applications can work on all of our supported devices, but sometimes things slips through the cracks.

I’ve made a note to our developers of this issue.

Thanks!


#9

I think this is a bug in the 0.8 SDK since it happens for us as well in our own codebase.

Thanks!


#10

Our developers will take a look at it, but it appears to be an issue with the sample code and not the SDK directly.

Once we do make a fix (along with others on our list), we will plan a new release of the SDK.


#11

Thanks anthony.

Any updates on this issue?


#12

We have made it a task to fix the issue in our next release.

Please let us know if you have any more questions!


#13

Hi,

thanks for the release! Does this version support OpenGLES3? When using the renderMesh function with OpenGLES3, my app crashes. When switching to OpenGLES2 everything works fine.
However, in the future, I would like to use the improved OpenGLES3.

thx!


#14

Unfortunately, we currently only support OpenGLES2 in the Structure SDK 0.8

I have noted this feature request to our developers, however!


Structure Unity AR Application crash.
#15

#16

#17

#18

Hi,
Im getting thecameraIntrinsicsMatrixDeliveryEnabled error in a unity3D (2017.3.1f1) IOS build,
lines 145-153 in ViewController+Camera.mm contain this code

FourCharCode fourCharCode = CMFormatDescriptionGetMediaSubType(formatDesc);

    CMVideoFormatDescriptionRef videoFormatDesc = formatDesc;
    CMVideoDimensions formatDims = CMVideoFormatDescriptionGetDimensions(videoFormatDesc);
    
    if ( 2592 != formatDims.width )
        continue;
    
    if ( 1936 != formatDims.height )
        continue;

Im building to IOS 11, and I have this code in the" structureAR.mm" that tires to turn it on but it does not

#if __IPHONE_OS_VERSION_MAX_ALLOWED >= 110000
    if (@available(iOS 11_0, *))
    {
        if (conn.isCameraIntrinsicMatrixDeliverySupported)
            conn.cameraIntrinsicMatrixDeliveryEnabled = YES;
    }
#endif

I 'm currently stuck on this issue. Is there a project setting to enable this? Or a line of code that i need to set? or a post that explains a fix for this issue?


#19

Can you tell me which version of Xcode you are using and what iOS device you are building to?


#20

Anthony,
I’m building an xcode project from unity3d(2017.3.1f1)
Target Device: ipad Air (A1474)
xcode: 9.4.1 (9F2000)
This is the complete error dialog I get on xcode
Uncaught exception: AVCameraCalibration: We want to use AV Intrinsics, but we’re not getting them! Make sure that you have set conn.cameraIntrinsicsMatrixDeliveryEnabled = YES on the AVCaptureConnection object (before you commit the AVCaptureSession configuration).


#21

Got some good news, I got it to build and work on our iPad Pro (a1670) :slight_smile:
It seems the old ipad Air are not compatible.
is there a way to disable that feature for the old iPad Air models?
Is there an older SDK version you recommend to use instead of 0.8.1 that works with IOS 10 or 11?


#22

The line, AVCaptureConnection.cameraIntrinsicMatrixDeliveryEnabled, is supposed to only be used on newer devices.

The code used in the StructureAR.MM file should be smart enough to detect older and newer devices, but if it isn’t working correctly, you can always just block that but of code off as a comment from #if to #endif.

I’ll run some tests on the iPad Air we have in the office sometime soon. Thanks for the info.