Converting to STCaptureSession using Scanner Sample App as an example.


#1

How to Convert from STSensorController to STCaptureSession

SDK (iOS) 0.9, introduced the STCaptureSession API. STCaptureSession consolidates many features common across all Structure SDK applications, and provides a unified interface for dealing with both Structure Sensor and your Apple devices. This new API represents a paradigm shift from the STSensorController API and introduces a more extensible way to write Structure Sensor apps. The flexibility and unified control of STCaptureSession enables developers to do even more with Structure Sensor.

We presented a webinar about converting from STSensorController to STCaptureSession on August 8th, 2019. You can view a recording on this webinar here:

STCaptureSession VS STSensorController

Structure SDK (iOS) 0.9 adds support for the new STCaptureSession API which represents a shift from the original STSensorController paradigm. The API is a direct replacement for STSensorController and contains a superset of features found in the STSensorController API. STCaptureSession provides better control, and maps more directly to how the sensor operates, making it easier for developers to configure and get running.

For example:

  • Control over the iOS sensors and Structure Sensor is now handled through the same API enabling:
    • Internal handling of the lens state within the capture session, unlocking the ability to use our wide-vision lens module in developer applications
    • Handling of the calibration state for a given lens module within the capture session, giving developers awareness of when users need to run calibrator for an optimal experience
    • Lens detection on the color camera and automatic detection of wide-vision lens modules attachment
  • Unified control of iOS and Structure sensors means that when the app is interrupted / resumed (e.g. in app backgrounding), we can track the applied settings or configuration and restore it automatically for the user without any additional code in the application itself. This is especially important for e.g. setting iOS camera properties like camera focus, exposure, ISO, and white balance.
  • Many of the delegate methods have been pared down and formalized into a smaller set of delegates that do more. This means less code duplication and makes it easier to find out where you should look when trying to find out something about your sensor state.

The capture session models an event-based system as opposed to a polling-based system. This makes the threading model of most apps easier to understand, and also means that developers need to decide how they respond to events (e.g. sensor is ready, new frame came in, etc) rather than trying to produce a model of how to connect, interact with, and handle the sensor. This is important because it means that the capture session will operate consistently across many applications. Application-specific setup of the Structure Sensor and iOS camera are no longer necessary, developers just need to respond to data and events as they arrive. This translates to most of the boilerplate code for handling setup being condensed into the capture session. For more information about event-driven programming, please have a look here.

Many of the features provided by STCaptureSession are also possible using the STSensorController API; however, the control provided by the STSensorController API is significantly narrower than what is provided STCaptureSession. The drawback of using STSensorController is that all the code for handling Structure Sensor setup, iOS sensor setup, and sending frames to frame sync are tightly coupled. This means that if you start changing your app even a little bit, or if a new feature (e.g. Apple SDK updates) comes out then it is necessary to update every app with app-specific code in order to reap any benefits. It also means that when you start refactoring code or moving parts around, you have a higher likelihood of breaking the setup, and thus having very hard-to-track and very critical bugs.

STSensorController has some event-based capabilities, however not every API falls into this category. You have to query for things like sensor connection ready state and iOS camera connection / ready state. There are also certain features that aren’t possible for us to provide with STSensorController, such as Wide Vision Lens support and detection.

Examples:

  • Query sensorBatteryNeedsCharging or when the sensor leaves low power mode, – sensorDidLeaveLowPowerMode.
  • Query the sensor with sensorDidStopStreaming or sensorDidConnect.

Using Scanner Sample app as an example of the STSensorController to STCaptureSession upgrade process

High-level overview of the steps needed to use STCaptureSession

Broadly, an app using STCaptureSession will need to do the following in order to stream frames and pass those frames to our SLAM code:

  1. An STCaptureSession object will need to be allocated and configured for monitoring. Monitoring mode is where the STCaptureSession looks for sensor connection events, but doesn’t yet stream frames.

  2. In our latest Scanner Sample application, we initialize the STCaptureSession in the viewDidLoad method of our ViewController.

  3. Sensor configuration is passed to the capture session via the startMonitoringWithOptions method. This will tell the capture session to monitor the status of the sensor (e.g. disconnected, waking up, connected, ready) and will signal the captureSession:sensorDidEnterMode: delegate when the sensor changes states

  4. Outside of the streaming config itself, it’s a really good idea to configure the preferred lens and lens detector mode here as well.

  5. Sensor events will come in over the sensorDidEnterMode:, colorCameraDidEnterMode:, and sensorChargerStateChanged: delegates.

  6. From here, the app’s UI should reflect the state of the sensor, so your users know what’s going on.

  7. Once the sensor enters ready state, it’s a good idea to check the STCaptureSession for the calibration type. If a user’s sensor needs calibration, they should be informed.

  8. Once the sensor is in a valid ready state, you should start streaming. This is done by calling captureSession.streamingEnabled = YES; Don’t worry, streaming is independent of whether or not the sensor is monitoring, so any sensor disconnections or reconnections will be detected, and will signal the appropriate delegates.

  9. Once streaming starts you will receive frames and accelerometer / gyroscope data via the captureSession:didOutputSample:type: delegate

  10. If you are streaming iOS color camera frames, the captureSession:didStartAVCaptureSession: delegate will be notified.

  11. From here, sensor data should be used, modified, or sent to our SLAM system the same way it’s always been done.

Specific instructions for upgrading your app:

Convert your app from STSensorController to STCaptureSession to take advantage of all upcoming new features. This example uses the generic settings, like low-resolution color, but these options can be changed by looking at the documentation.

  • STSensorController has ViewController+Sensor.mm and View Controller.Camera.mm
    • STCaptureSession has only ViewController+CaptureSession.mm
  • Go to ViewController.mm
    • Scroll down to STSensorController* _sensorController;
      • Replace with STCaptureSession* _CaptureSession;
    • Head to - (void)enterCubePlacementState
      • Under _trackingLostLabel.hidden = YES, add _captureSession.properties =STCaptureSessionPropertiesSetColorCameraAutoExposureISOAndWhiteBalance();
      • Delete [self setColorCameraParametersForInit];
      • Replace self setColorCameraParametersForInit] with _captureSession.properties = STCaptureSessionPropertiesSetColorCameraAutoExposureISOAndWhiteBalance();
    • Head to -(void)enterViewingState
      • Under self.resetButton.hidden = YES;, add _captureSession.streamingEnabled - NO;
    • Head to -(void) appDidBecomeActive
      • Under if ([self currentStateNeedsSensor]), delete [self connectToStructureSensorAndStartStreaming];
      • Add [self setupStructureSensor];
      • Change if (_slamState.scannerState == ScannerStateScanning) to if(_captureSession.streamingEnabled)
    • Go to - (void)initializeDynamicOptions
      • Comment out _dynamicOptions.highResColoring = [self videoDeviceSupportsHighResColor]; and _dynamicOptions.highResColoringSwitchEnabled = false;
    • Head to - (IBAction)enableHighResolutionColorSwitchChanged:(UISwitch*)sender
      • Change to
        • [self setupStructureSensor];
        • [self onSLAMOptionsChanged];
    • Head to -(void)updateIdelTimer
      • Change if ([self isStructureConnectedAndCharged] to if ([self isStructureConnected]
    • Head to -(void)meshViewDidDismiss
      • Change [self connectToStructureSensorAndStartStreaming] to [self setupStructureSensor];
    • Scroll back up to the top of ViewController.mm and #import Structure/StCaptureSession.h
    • remove #import ViewController+Camera.h; from ViewController
    • Also rename #import ViewController+Sensor.h; to ViewController+CaptureSession.h;
  • Rename ViewController+Sensor.h and ViewController+Sensor.mm to ViewController+CaptureSession.h and ViewController+CaptureSession.mm in project manager
    • remove #import ViewController+Camera.h from ViewController.h and ViewController.mm
    • Also rename #import ViewController+Sensor.h; to #import ViewController+CaptureSession.h;
  • Go to ViewController+CaptureSession.h
    • rename the interface as @interface ViewController (CaptureSession) STCaptureSession;
    • Remove (STSensorControllerInitStatus)connectToStructureSensorAndStartStreaming;
    • Rename (BOOL)isStructureConnectedAndCharged to (BOOL)isStructureConnected;
  • Go to ViewController+CaptureSession.mm and rename @implementation ViewController (Sensor) to @implementation ViewController (CaptureSession)
  • Delete ViewController+Camera.h and ViewController+Camera.mm because STCaptureSession handles this.
    • Note that - (bool) videoDeviceSupportsHighResColor method could be moved and implemented in ViewController+CaptureSession if you want high resolution color.
  • Go to ViewController+CaptureSession.mm
    • head to - (void)setupStructureSensor
      • Add if (_captureSession != nil) { _captureSession = nil;}
      • Change _sensorController = [STSensorController sharedController] to _captureSession = [STCaptureSession newCaptureSession];
      • Add STCaptureSessionColorResolution resolution = STCaptureSessionColorResolution640x480;
      • Add NSDictionary* streamConfig = @ { Add Options here};
        • Ex: NSDictionary* sensorConfig = @{ kSTCaptureSessionOptionColorResolutionKey: @(resolution), kSTCaptureSessionOptionDepthSensorVGAEnabledIfAvailableKey: @(NO), kSTCaptureSessionOptionColorMaxFPSKey: @(30.0f), kSTCaptureSessionOptionDepthSensorEnabledKey: @(YES), kSTCaptureSessionOptionUseAppleCoreMotionKey: @(YES)};
      • add _captureSession.lens = STLensNormal and _captureSession.lensDetection = STLensDetectorOff; to set the lens used to normal and to set the Wide Vision Lens Detector to off.
      • Change _sensorController.delegate = self to _captureSession.delegate = self;
      • Add [_captureSession startMonitoringWithOptions:sensorConfig];
    • Head to - (BOOL)isStructureConnectedAndCharged
      • Delete AndCharged from the end of the method.
      • Replace line with return _captureSession.userInstrictions & STCaptureSessionsUserInstrictionNeedToConnectSensor;
    • Comment out the rest of the methods present in ViewController+CaptureSession.mmto help reduce warnings and error messages in Xcode as we build.
    • Add the following functions:
      • (void)captureSession:(STCaptureSession *)captureSession colorCameraDidEnterMode:(STCaptureSessionColorCameraMode)mode
      • (void)captureSession:(STCaptureSession *)captureSession sensorDidEnterMode:(STCaptureSessionSensorMode)mode
    • Under colorCameraDidEnterMode method
      • Add switch statement to handle all of the color camera modes with the following cases:
        • case STCaptureSessionColorCameraModePermissionDenied:
        • case STCaptureSessionColorCameraModeReady:
          • Break;
        • case STCaptureSessionColorCameraModeUnknown:
          • Throw Error Message
        • Default:
          • Throw Error Message
      • Add [self updateAppStatusMessage];
        • Note, we don’t need to do this, and instead just throw these within the delegate directly instead of calling updateAppStatusMessage
          • See updateAppStatusMessage in ViewController.mm in captureSession scanner app sample for details on how this is implemented.
    • Under sensorDidEnterMode method
      • Add switch statement to handle all of the sensor modes with the following cases:
        • case STCaptureSessionSensorModeReady:
          • If ([self currentStateNeedsSensor]) {
          • captureSession.streamingEnabled = YES;
          • Break;
        • case STCaptureSessionSensorModeWakingUp:
          • _appStatus.sensorStatus = App Status::SensorStatusNeedsUserToConnect;
          • Break;
        • case STCaptureSessionSensorModeStandby:
          • Break;
        • case STCaptureSessionSensorModeNotConnected:
          • Break;
        • case STCaptureSessionSensorModeBatteryDepleted:
          • Break;
        • case STCaptureSessionSensorModeUnknown:
          • Throw Error Message
        • Default:
          • Throw Error Message
      • Add [self updateAppStatusMessage];
    • Write new method, - (void)captureSession:(STCaptureSession*)captureSession didOutputSample:(NSDictionary*)sample type:(STCaptureSessionSampleType)type
      • Write a switch (type) statement with the following cases
        • case STCaptureSessionSampleTypeSensorDepthFrame:
          • STDepthFrame* depthFrame = [sample objectForKey:kSTCaptureSessionSampleEntryDepthFrame];
          • if (_slamState.initialized) {
            • [self processDepthFrame: depthFrame colorFrameOrNil:nil];
            • [self renderSceneForDepthFrame: depthFrame colorFrameOrNil:nil]; }
          • Break;
        • case STCaptureSessionSampleTypeIOSColorFrame:
          • Break;
        • case STCaptureSessionSampleTypeSynchronizedFrames:
          • STDepthFrame* depthFrame = [sample objectForKey:kSTCaptureSessionSampleEntryDepthFrame];
          • STColorFrame* colorFrame = [sample objectForKey:kSTCaptureSessionSampleEntryIOSColorFrame];
          • if (_slamState.initialized) {
            • [self processDepthFrame:depthFrame colorFrameOrNil:colorFrame];
            • [self renderSceneForDepthFrame:depthFrame colorFrameOrNil:colorFrame];}
          • Break;
        • case STCaptureSessionSampleTypeDeviceMotionData:
          • CMDeviceMotion* deviceMotion = [sample objectForKey:kSTCaptureSessionSampleEntryDeviceMotionData;
          • [self processDeviceMotion:deviceMotion withError:nil];
          • Break;
        • case STCaptureSessionSampleTypeUnknown:
          • Throw Error
        • Default:
          • Throw Error

Tips & Tricks

  • You can optimize dynamic properties such as the iOS color camera exposure, ISO, and white balance by setting the capture session properties (see the captureSession:didStartAVCaptureSession: delegate in our latest Scanner Sample app)
  • If you are willing to go rogue? You can find and replace all of your STSensorController objects from your app and replace with STCaptureSession. (Don’t forget to add the header)

listed #2