I Built a Pokédex for Real Animals (Where the AI Never Leaves Your Phone)

Adrian Chrysanthou15 min read
capture a cute puppy at the park.

For anyone who’s ever pointed a phone at a lizard and whispered, “What ARE you?”

An orange cat patrols my street like he pays the mortgage. I've taken maybe forty photos of him. They sit in my camera roll doing absolutely nothing.

That bugged me.

Every animal ID app I tried did the same thing: upload my photo to a server, wait, get back a species name, the end. Nothing to collect, nothing to show for it, and now my photos live on someone else's computer.

So I asked the big question. Could I run the whole thing on the phone? Detection, classification, background removal, depth, all of it, with zero server calls?

Well, Yes. It took ten TFLite models, one research-grade teacher model, and a breed classifier that started life at 0.96% accuracy. This is how SnapPetz got built.

What SnapPetz Actually Does

Three taps, start to finish:

  1. Point. The live scanner draws a paw box around any critter in frame. Cats, dogs, horses, cows, sheep, birds, and a big wildlife bucket that covers everything from foxes to beetles.
  2. Catch. Tap, and the phone identifies the animal, cuts it out of the background, and reads what it's doing, like a wagging tail, a yawn, or a slow blink.
  3. Collect. The catch becomes a spinnable 3D emblem built from the animal's real silhouette, with a rarity tier from Common to Legendary. Drag to rotate it. Flick it to keep it spinning.

You can also pin a catch to the World Map so friends can see the heron you found at the retention pond. Sharing is opt-in and off by default.

It's free on iOS and Android.

Making that work takes a lot of machine learning crammed into your pocket.

The Architecture

On-device catch pipeline

A single catch runs through a chain of small models, each doing one job.

StageModelSizeJob
Live detectionEfficientDet-Lite04.6 MBDraws the paw box, one per animal, 320x320 input
Safety gateMobileNetV2 NSFW classifier8.7 MBRejects anything that isn't a critter
SpeciesMobileNetV3-Small (ImageNet)10 MB1,000 classes folded into 5 species
BreedDistilled EfficientNet-Lite013.6 MBCat and dog breed specialist
CutoutRMBG-1.4, MobileSAM fallback44 MBPixel-clean silhouette of the animal
DepthDepth Anything V2 Small27 MBDepth map for the 3D emblem
TraitsSigLIP-base image encoder94 MB768-d embedding scored against 16 behaviors
Table containing every model used

The whole bundle is about 240 MB and ships inside the app. Everything runs through react-native-fast-tflite, with a delegate fallback chain of GPU, then NNAPI, then plain CPU.

The Species Trick

I don't have a labeled dataset of every animal on Earth. ImageNet sort of does...

So instead of training a 5-class species head from scratch, I take a pretrained 1,000-class ImageNet model and collapse its output. 308 (exact) of those classes are animals I care about. I sum their probabilities into five buckets and renormalize:

// 1,000 ImageNet probs in, 5 species out
// cat: 5 classes, dog: 74, farm: 18, bird: 29, wildlife: 182
const species = aggregateImagenetToSpecies(probs, mapping);

The other 692 classes (toasters, sports cars, pool tables) just drop out. I never had to train anything or build a dataset. EZ PZ.

Training Off Research Models

Initial Attempt:

My first breed model was a MobileNetV3-Small trained on Stanford Dogs. One epoch on CPU. Validation accuracy: 0.96%.

There are 120 breeds. Random guessing gets you 0.83%. I had built a very expensive coin flip.

Ten epochs with ImageNet pretraining got it to 8.24%. ughhhh. Then I fixed the actual recipe (SGD with Nesterov momentum, weight decay, cosine learning rate), swapped the backbone to EfficientNet-Lite0, and ran it on my RTX 4090. 74.96%. A 9x jump from one afternoon of reading my own config.

The Teacher: BioCLIP

Dogs worked because ImageNet is weirdly obsessed with dogs. It gave me 74 dog classes to work with and only five for house cats.

So I went looking for a model that had actually studied biology. BioCLIP is a research vision model from the Imageomics team, and I used the variant pretrained only on iNaturalist data. Photos of real organisms, labeled by people who care a lot about the difference between two sparrows.

I ran a clean comparison on Oxford Pets (37 cat and dog breeds). Frozen backbone, same training recipe, only the classification learns:

BackbonePretrained onTop-1
EfficientNet-Lite0ImageNet83.03%
BioCLIP ViT-B/16iNat2187.96%

BioCLIP wins by almost 5 points, but it has roughly 86 million parameters, which is way too big for a phone.

The Student: Distillation

Knowledge distillation lets a small model learn from a big one. The student gets the label ("this is a Siamese") plus the teacher's full opinion: mostly Siamese, a little Birman, definitely not a pug.

Training and distillation pipeline
uv run crittercatch-train distill \
  --teacher-backbone bioclip-vit-b-16-inat \
  --student-backbone efficientnet-lite0 \
  --student-pretrained \
  --dataset oxford-pets \
  --epochs 30 --lr 1e-3 --temperature 4 --alpha 0.5

But Why Did This Work?:

  • Temperature 4: Softens the teacher's output so the student can see which breeds look alike.
  • Alpha 0.5: Half the loss comes from real labels, half from matching the teacher.
  • Pretrained student: Starting from ImageNet weights beats random init by a few points.

The result: a 5 million parameter EfficientNet-Lite0 that hits 90.80% top-1 on Oxford Pets. That's 7.77 points better than the same architecture trained the normal way, in a 13.6 MB file.

Traits, Rarity, and the Emblem

A species name alone gets boring fast. I wanted every catch to record what that particular animal was doing when I found it.

Reading Behavior With Zero Training

SnapPetz has 16 traits: wagging tail, smiling, mid-yawn, mid-pounce, alert stance, and so on. I did not train a classifier for any of them.

I pre-computed SigLIP text embeddings for every trait, conditioned on species (a cat mid-pounce looks nothing like a horse mid-pounce), and baked them into the app. At catch time, the phone runs the image through the SigLIP encoder once, gets a 768-dimensional vector, and takes a dot product against each baked trait.

That gets me zero-shot behavior detection on a phone without shipping a text encoder. (as best as possible!)

Thresholds were the hard part. Every trait needs its own cutoff, and SigLIP scores live in a much lower range than CLIP's (0.03 to 0.18 instead of 0.15 to 0.35). I calibrated each one against 2,343 hand-verified photos, maximizing precision while setting a floor for recall. (Pro-Tip, leverage LLM to build the scripts to grab the photos, but you’ll still need to manually verify)

Traits drive rarity. A dog just sitting there is Common. A dog mid-yawn with a wagging tail scores a lot higher.

From Photo to 3D Emblem

  1. RMBG-1.4 cuts the animal out of the background at 1024x1024.
  2. Depth Anything V2 estimates a depth map of the cutout.
  3. A Skia shader extrudes the silhouette into 24 depth slices and wraps your photo around it.

The result is a gold-rimmed emblem shaped like the animal you caught. Mine is the orange cat, attitude included, and I can spin him around with my thumb.

Lessons Learned

Getting ten models running inside a React Native app broke in a lot of ways. I almost went native Swift and native Kotlin

  • TFLite conversion: My PyTorch to ONNX to TFLite pipeline produced models that ran fine on my desktop and died on device with a helpful "Status: error.". Thankfully exporting straight from Keras fixed it.
  • Image decoding: For a while, every photo came back as a random animal. I was feeding raw JPEG file bytes, headers and all, into the model and calling them pixels. :/
  • Tensor layouts: MobileSAM's encoder outputs NHWC and its decoder wants NCHW. I had to do a ton of trial and error with frontier LLMs to figure out a solution.
  • Benchmarks: BioCLIP tied ImageNet on Stanford Dogs, and I almost dropped it. Switching to a dataset where ImageNet had no home-field advantage showed the real gap.
  • Distillation: The distilled student beat the same architecture trained normally by 7.77 points. The file size and speed didn't change at all.

Your Photos Never Leave Your Phone

I did all this so the app is private by default.

Recognition never uploads a photo. Your collection lives in an encrypted database on your device (AES-256-GCM, with the key in the iOS Keychain or Android's encrypted storage). No ads and no analytics SDKs.

The World Map is the one place data can be uploaded, and only when you flip the switch on a specific catch. You can share the heron and keep your backyard to yourself.

Try It Yourself

SnapPetz is free on the App Store and Google Play. Grab it at snappetz.com, walk outside, and point it at the first thing with a pulse.

Then go find something Legendary and pin it on the map.

I did finally catch the orange cat on my street. Epic tier. He remains unimpressed. :P

If you know someone with a camera roll full of animals they don't own, send them this. And if you catch something weird, I want to see it.