31

I'm implementing a face tracker on Android, and as a literature study, would like to identify the underlying technique of Android's FaceDetector.

Simply put: I want to understand how the android.media.FaceDetector classifier works.

A brief Google search didn't yield anything informative, so I thought I'd take a look at the code.

By looking at the Java source code, FaceDetector.java, there isn't much to be learned: FaceDetector is simply a class that is provided the image dimensions and number of faces, then returns an array of faces.

The Android source contains the JNI code for this class. I followed through the function calls, where, reduced to the bare essentials, I learned:

  1. The "FaceFinder" is created in FaceFinder.c:75
  2. On line 90, bbs_MemSeg_alloc returns a btk_HFaceFinder object (which contains the function to actually find faces), essentially copying it the hsdkA->contextE.memTblE.espArrE array of the original btk_HSDK object initialized within initialize() (FaceDetector_jni.cpp:145) by btk_SDK_create()
  3. It appears that a maze of functions provide each other with pointers and instances of btk_HSDK, but nowhere can I find a concrete instantiation of sdk->contextE.memTblE.espArrE[0] that supposedly contains the magic.

What I have discovered, is a little clue: the JNI code references a FFTEm library that I can't find the source code for. By the looks of it, however, FFT is Fast Fourier Transform, which is probably used together with a pre-trained neural network. The only literature I can find that aligns with this theory is a paper by Ben-Yacoub et al.

I don't even really know if I'm set on the right path, so any suggestions at all would undoubtedly help.

Edit: I've added a +100 bounty for anybody who can give any insight.

Paul Lammertsma
  • 37,593
  • 16
  • 136
  • 187
  • libFFTEm isn't referenced from Android's face detection layer, actually it IS the compiled output of all sources in external/neven – ognian Jul 28 '10 at 15:35
  • That explains why I couldn't find the source code! Thanks; this suggests that if it indeed uses Fast Fourier Transforms, there should be some evidence of it in `neven`. However, nothing really jumps out at me. – Paul Lammertsma Jul 28 '10 at 16:12
  • 2
    It looks like the "Em" in `libFFTEm` may refer to "expectation-maximization," a statistical method used in color-based skin detection. – Justin Aug 12 '10 at 21:52
  • 1
    There is a project, which extracted the neven face detector library from Android: [https://github.com/lqs/neven](https://github.com/lqs/neven). – bytefish Aug 27 '14 at 21:03

3 Answers3

4

I Found a couple of links too...Not sure if it would help you...

http://code.google.com/p/android-playground-erdao/source/browse/#svn/trunk/SnapFace

http://code.google.com/p/jjil/

http://benosteen.wordpress.com/2010/03/03/face-recognition-much-easier-than-expected/

DeRagan
  • 22,827
  • 6
  • 41
  • 50
  • Thank you for your response, and those are interesting projects indeed, but I'm looking for some insight into how Android's *built-in* face detector works (`android.media.FaceDetector`). – Paul Lammertsma Aug 13 '10 at 00:03
3

I'm on a phone, so can't respond extensively, but Google keywords "neven vision algorithm" pull up some useful papers...

Also, US patent 6222939 is related.

Possibly also some of the links on http://peterwilliams97.blogspot.com/2008/09/google-picasa-to-have-face-recognition.html might be handy...

Jonathan Ellis
  • 5,221
  • 2
  • 36
  • 53
Stobor
  • 44,246
  • 6
  • 66
  • 69
  • It appears that Google acquired Neven Vision, an object recognition technique based upon various patents filed by Hartmut Neven. I've not found a detailed description of the algorithm, but following up on the patent gives a little insight. – Paul Lammertsma Aug 16 '10 at 14:58
  • The [various patents listed on this site](http://www.seobythesea.com/?p=267) give some insight into the technique. It seems to suggest that it heavily relies on [Gabor Wavelets](http://en.wikipedia.org/wiki/Gabor_filter#Wavelet_space). – Paul Lammertsma Nov 25 '10 at 10:59
0

have a look at this: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=1562271

I think I once saw some matlab code doing this in a presentation. Maybe it's somewhere online.

Greetings, Lars

Lars
  • 1,136
  • 7
  • 16
  • I googled this link here: Maybe it's the code. I don't know, I don't have matlab or scilab here to try it out. Maybe you have: http://download.cnet.com/Face-Recognition-in-Fourier-Space/3000-2053_4-10878312.html – Lars Aug 13 '10 at 11:29
  • I'm afraid I don't find any evidence of this being the technique that Neven Vision is based on. – Paul Lammertsma Nov 25 '10 at 11:27