Announcing Structure SDK 0.6: New & Improved Object Scanning


#1

Update (Jan 23, 2018): Structure SDK v0.7 - Major Update is now available. See the forum announcement for more info.


Update (Dec 22, 2016): Structure SDK v0.6.2 - Maintenance Update is now available. See the announcement for more.


Update (Sept 28, 2016): Structure SDK v0.6.1 - Maintenance Update is now available. See the announcement for more.


Today we’re pleased to announce the release of **Structure SDK 0.6**!

Today’s update is considered a major update to the Structure SDK with a focus on improved object scanning. From all of us at Occipital, we hope you enjoy this latest and greatest and fastest Structure SDK.

And a Big Thanks! to the 16 organizations that participated in testing the Beta for Structure SDK 0.6. Your feedback helped us get this version ready for public release.

Download

Download Structure SDK 0.6

New & Improved Object Scanning

SDK 0.6 features a brand new mapper. Its performance is a dramatic improvement over the old mapper, and will allow for much more frequent model updates and/or finer resolutions. Moreover, it enables finer detail when scanning larger volumes (up to 10x10x10 meters). This new mapping mode is enabled by default. (See the STMapper section in the Release Notes below for how to continue using the legacy 0.5 mapper).

“WOW! The new mapper is really impressive for our use case! I was able to support the volume size that we have been using at a much higher resolution and it looks great!”

                                                    **Aaron Bryden, Praxik**

##A Summary of Scanning Performance Improvements You'll Notice * Vast improvements of speed over previous mapper. **Performance tests show that the new mapper in SDK 0.6 produces 60% more resolution - in each XYZ axis - at the same speed when compared to the previous mapper.** That's a _~2.3x speed improvement at the same resolution!_ * Structure SDK 0.6 now features accuracy-enhancing tech in part due to our [Lynx Laboratories acquisition][3]. This means your 3D scans will be more true-to-reality than ever. * Support for a **wider variety of color input resolutions** _(Thanks to beta tester feedback!)_. * New ability to define the **far depth integration range**, up to 8 meters. * Clean surface scans with less extraneous, "floating" geometry captured from the environment. Cleanup is handled after mesh is processed post-scan.

(Notice the finer detail in the scanned Caesar bust with the new Structure SDK mapper; With approximately 3X more faces & vertices in the mesh. Results may vary based on the object scanned.)

## Density Comparisons of New High Resolution Mesh [][1]
(Run on iPad Mini 2, with A7 chip.)


--------------------- # Complete Release Notes ## New & Improved Object Scanning This new version features a brand new mapper. Its performance is a dramatic improvement over the old mapper, and will allow for much more frequent model updates and/or finer resolutions. The new mapping mode is enabled by default. You can keep using the legacy 0.5 mapper by specifying the `kSTMapperLegacyKey` option when initializing a `STMapper` instance. ## High-resolution Color Capture The Structure framework now supports the following image resolutions in `STColorFrame` instances: - 640x480 - 2048x1536 _(New)_ - 2592x1936 - 3264x2448 _(New)_ - 4032x3024 _(New)_ Please note that capturing higher resolution images will not necessarily improve the detail of the output textures computed by `STColorizer` instances. ## API Changes ### `STSensorController` * The `error` parameter of the sensor controller initialization method `- (instancetype)initWithOptions:(NSDictionary *)options error:(NSError* __autoreleasing *)error` was dropped, resulting in this new signature: `- (instancetype)initWithOptions:(NSDictionary *)options;` * The sensor controller initialization option `kSTHoleFilterConfigKey` was renamed to `kSTHoleFilterEnabledKey`. * Similarly, the `kSTHighGainConfigKey` was renamed to `kSTHighGainEnabledKey`.​ ### `STSensorControllerDelegate` - The sensor delegate method `[STSensorControllerDelegate sensorDidOutputSynchronizedDepthFrame:andColorFrame:]` was renamed to `[STSensorControllerDelegate sensorDidOutputSynchronizedDepthFrame:colorFrame:]`. - Similarly, its infrared sibling `[STSensorControllerDelegate sensorDidOutputSynchronizedInfraredFrame:andColorFrame:]` was renamed to `[STSensorControllerDelegate sensorDidOutputSynchronizedInfraredFrame:colorFrame:]`. - The `sensorDidEnterLowPowerMode` delegate callback method was removed. ### `STDepthFrame` - Reconstruction accuracy can be improved on a per-frame basis via a new method called `applyExpensiveCorrection`. This is recommended for long-range depth, as it will avoid geometry distortions in the reconstructed model. However, it has a significant runtime cost. For instance, on iPad Air 2 it can take up to 5 milliseconds to complete. To observe improvements in accuracy, perform a reconstruction and compare the resultant model with and without this new feature enabled. ### `STMapper` - The `(void)setHasSupportPlane:(BOOL)hasIt` method was moved to a boolean initialization option called `kSTMapperVolumeHasSupportPlaneKey` which defaults to `NO`. - The `STMapper` initialization method can now be passed additional options, described below: - The `kSTMapperLegacyKey` allows you keep using the pre-0.6 mapper in your applications. It defaults to `NO`, because you really should be using the awesome new mapper, unless you have a very good reason not to! - The `kSTMapperVolumeBoundsKey` allows you to specify the extents of the mapped volume as a numeric array of 3 integral values specifying the number of cells in each dimension. - The `kSTMapperEnableLiveWireFrameKey` determines whether the STMapper should automatically build a wireframe mesh in the background when new depth frames are provided. It is set to `NO` by default. - The `kSTMapperDepthIntegrationFarThresholdKey` option key enables tuning of the depth integration far threshold. This is an advanced option, and the default 4 meters should fit most use cases. - The `volumeResolution` property has been removed. Instead, its value is now directly specified as an option when initializing a `STMapper` instance. ### `STTracker` - The `error` parameter of the tracker initialization method `[STTracker initWithScene:options:error:]` was removed, resulting in this new signature: `- [STTracker initWithScene:options:]`. - Similarly, the `error` parameter of the `- (BOOL)setOptions:(NSDictionary *)options error:(NSError* __autoreleasing *)error` method was dropped, resulting in this new signature: `- (void)setOptions:(NSDictionary *)options`. - The `STTrackerStatus` enum is no more. Instead, two `STTracker` properties replace it and clarify its semantics: - The `trackerHints` property, of type `STTrackerHints`, aggregates 3 boolean values set when the tracker gets lost, gets too close or when the model gets out of view. - The `poseAccuracy` property, an enumeration value of `STTrackerPoseAccuracy` type, which qualifies the confidence of the tracking as unavailable, very low, low, approximate or high. - The `predictCameraPoseAtTimestamp` has been removed. You can use the `lastFrameCameraPose` instead. ### `STCubeRenderer` - The `adjustCubeSize` method does not need a volume resolution anymore. ### `STErrorCode` The following error codes are not in use anymore and were removed: - `STErrorOptionNotRecognized` - `STErrorOptionInvalidValue` - `STErrorOptionMissingValue` - `STErrorOptionCannotBeUpdated` - `STErrorFileNoSuchFile` - `STErrorTrackerLostTrack` - `STErrorTrackerColorSampleBufferFormatNotSupported` - `STErrorTrackerColorSampleBufferMissing` - `STErrorTrackerTrackAgainstModelWithoutLiveTriangleMesh` - `STErrorTrackerPoorQuality` ## Platform Upgrade - This release features full iPad Pro 9.7‑inch and 12.9-inch support, together with the [latest Calibrator 1.2 update][2], already available on the App Store. - `armv7s` support was dropped. This reduces the size of the binaries, doesn't affect performance and matches Apple's latest default architecture list. - Support for iSense sensors is discontinued, and they will not be recognized by the SDK. Please contact `support@structure.io` if you need to deploy a Structure SDK app for iSense devices. - Error reporting was improved and error messages clarified. - Documentation was clarified and improved. - As with [Structure SDK 0.5.5][4], this release requires **iOS 8.0** or above. ## Sample Code Changes - The Viewer Sample app now streams color sample buffers in YCbCr format. RGBA sample buffer support is deprecated. - The Scanner sample app can now perform high-resolution color streaming on iPad Air 1 (at 24 FPS), iPad 4 (at 15 FPS) and other older iOS devices unable to capture 2592x1936 at 30 FPS. - The Unity packages were updated for the latest API changes and tested with Unity 5.3.4. - Sample app projects were upgraded for Xcode 7.3. - Sample apps icons have been revamped with a shiny new look. [1]: https://youtu.be/EhqZau5R0Gk [2]: https://itunes.apple.com/us/app/structure-sensor-calibrator/id914275485?mt=8 [3]: http://occipital.com/2015/lynx-acquisition [4]: http://forums.structure.io/t/announcing-structure-sdk-0-5-5-improvements-for-ios-9-2/4951

Paired Output (Depth and Color) and Swift.
Announcing Structure SDK 0.6.2
#4

#5

And be sure to check out the updated Room Capture - Structure Sensor Sample & Scanner - Structure Sensor Sample apps - available now in the iTunes App Store. Made with our latest mapper available in SDK 0.6.


#6

Great to see a new SDK update, but we’ve been having some trouble integrating some of the changes, and I was wondering if someone could help shed some light on what might be going on.

In version 0.5.5, when scanning objects on the ground the mapper would capture the floor along with the object, so we translated the tracker’s initialCameraPose property by a small amount (0.0025 meters) to make sure we didn’t scan the floor. Upon switching to 0.6, we noticed that scans were cutting off things above the ground, so we removed that translation and now it usually doesn’t capture the floor, but it sometimes does get parts of it and the bottom of the scan comes out jagged. What I mean by jagged is that there are some triangles that stick out from the bottom almost like teeth (see screenshot).

Another issue we’ve been having is with the live rendering lagging. Only started happening when we migrated to 0.6, was wondering if anyone has idea why that might be.

Lastly, I was wondering if you could shed some light on the nature of the new mapper changes. The video you showed with the higher polygon count seems similar to what would happen if you just increased the resolution on the mapper. Does the new mapper simply run faster and therefore allow higher resolution processing at the same speed or is there a technical reason (don’t need details) for an underlying accuracy improvement?

Thanks in advance for any insights anyone might have. We tried using the legacy mapper and that didn’t fix these problems so we’re probably going to revert to version 0.5.5 until we can get these problems sorted out. Also curious if anyone else is having some of the same problems?

Sam


#7

Hi Sam,

I would think the best way to get a complete scan of shoes / sandals would be to somehow suspend them. If you do scan them on the floor, I would think the best results would be achieved by making sure the floor was captured, e.g., playing with the offset you mentioned in the opposite direction. Then you could clip the floor out using whatever CAD program you use for post-processing.

Re: live rendering, what device are you testing on? I’m not experiencing any lag on an iPad Air 2.

Best,

Jim


#8

Jim,
The shoes were just an example, we need to scan on the floor because we’re scanning human feet in a weight-bearing position. You are correct that capturing the floor would be helpful if we were post-processing with a CAD program but we’re interesting in showing and analyzing the mesh immediately after the scan is taken on an iPad (Air 2, same one here), so taking out the floor is more work than it’s worth.

As for live rendering, it’s almost certainly not a device issue since it doesn’t using 0.5.5, even with a much higher resolution, and does lag using 0.6 on the same device. Here’s a scan of the same shoes using 0.5.5 (with a slightly lower resolution).

Notice how the bottom is smooth across? With the new SDK it’s almost as if it misses mapping sections of the object initially and then never goes back and adds them even if you look at the missing section from multiple angles.


#9

Hi Sam,

I think maybe the bottoms were smoother because the offset you used was actually including just enough of the floor to produce that nice edge. Using SDK 0.6, could you try adding the offset in the other direction and see if there’s a “sweet spot” that gives the same results?

I posted about this previously, but make sure your offset is in the proper coordinate system, or your results will vary depending on the angle you start scanning.
http://forums.structure.io/t/closed-cubeplacement-in-scannerstatescanning/5233/4?u=jim_selikoff

Is the floor highly reflective? When I scan a pair of sandals on a wood floor I don’t see the described jagged triangles.

Also, would it be possible to post a screen recording demonstrating the live rendering lag?

Best,

Jim


#10

Hi! Perhaps a silly question, but how can I upgrade the device to new SDK upgrade?


#11

@zahradnik.ondrej, the sample apps on the AppStore have been updated to use the latest SDK.

-jim


#12

So if I am using the “Scanner” it is just enough to have the updated version? No need to install some new firmware into the scanner hardware? Btw same with Scant PC version? Thanks a lot for the clarification!


#13

No problem!

Your sensor is likely on the latest firmware version. Run the Structure app on the iPad and click on the Gear icon to check. It should be at 2.0.

If you mean “Skanect”, the Structure SDK only affects iOS apps.

Hope this helps,

Jim


#14

How can I download SDK .5.5 so I can compare differences?


#15

@paul5 - We don’t have SDK 0.5.5 available to download anymore. You can see the comparison in the video above. You can also check out Scanner app which will allow you to toggle from older SDK features and the newer ones.


How can I download of the 0.5.5 SDK