Facial Mocap with Structure Sensor


Has anyone experimented with facial motion capture using the Structure Sensor? At 60fps, it should be able to record facial positions. Can’t seem to find anything online, though.


I would recommend seeing if there are any OpenNI or OpenCV projects that are about face detection, as those Open-Source libraries can be used with the Structure Sensor.


Have you seen what they’re doing at USC ICT?


come ON dudes!
https://github.com/ageitgey/face_recognition this is awesome
https://github.com/zhoushiwei/irisfaceRGBD this is awesome
https://github.com/sergeytulyakov/FaceCept3D this is awesome
https://stackoverflow.com/questions/14012304/face-recognition-using-kinect this is some basic background material
https://gist.github.com/jpanganiban/3844261 this is awesome if you can digest code (i can’t but it looks clean and good to me heh)
https://www.docs.opencv.org/2.4/modules/contrib/doc/facerec/index.html this is what you need

some old forum links: Detecting Face while scanning
and the one Jim Selikoff posted ^^^above^^^

also - 60fps is the rate that the camera data is coming in. It is being processed. I feel like it is not exactly like the right number you’re referring to… unless you are able to include a particular resolution to specify the level of detail yr trying to indicate, it’s like saying you can only see a ripple at a certain framerate when in reality there’s always gonna be a circle on the water and you (or the software) has to determine how quickly it’s moving. so dude, Think About Space. I mean if the Queen of England is doing a very insubstantial golf clap and you’re scanning from a Tiger Woods distance, that’s one thing… you will need a high framerate to calculate the line of best fit for the wan smile of a bumblebee as it flirts with its departing lover, i mean, but speaking generally… the framerate is cool to know but not of primary importance here in the world of facial recognition, except to the extent that it’s mandated by your mocap needs, i.e. a body of such-and-such size at such-and-such a distance to so-and-so precision for your purposes (did they smile at all? Did they smile bigly? Did they, perchance, curl their lip in an oddly aggressive smile?). be aware that the maxxed-out specs might not be the functional software specs for any given application – you can’t see the flicker on your monitor, I hope – AND that they might not even need to be maxxed-out for your purposes!

last thing, quick thing: Structure Sensor is a superlatively well-constructed piece of hardware, but generic depth cameras will relate to your interests, if you have even the slightest experience on the command line. your question is a little vague, but … read around. openCV will suck to compile or whatever … but then you will be free. just do it.

p.s.: skanect is optimized to start at a face (you can find out more about that at https://groups.google.com/forum/#!searchin/skanect/face|sort:date/skanect/vITB7enui6c/o4Eih4dlBQAJ – who knows what those crazy guys were thinking! :P), itSeez3D has a special face/bust setting, and you can always get away with good color on a lousy depth map at an angle where nobody realizes your scan has no nose but merely a nose-lookin’ lump. honestly facial motion capture is nothing but a particular kind of motion capture, and motion capture is nothing but plain old color video with a bunch of geeks in the room. so think about workarounds or ways others may or may not have approached what you may or may not be considering approaching. what depth gives you is the ability to select blobs easier. so maybe consider doing that - chop out the room and leave nothing but a colorful face-object. (and the ability to have a night-vision ipad, but I digress.)

you might want to post about whatever you intended to do with this magical face capturing tech once you found it online, BTW - am sure many are seeking something similar and could react theoretically positively in a helpful manner. spin up an animoji generator, yo – killer app