Pet segmentation + breed classification
A multi-task U-Net (EfficientNet-B0 encoder) running entirely in your browser with ONNX Runtime Web. Your photo is processed on your device and is not uploaded anywhere.
Top-3 breeds (click one to see its heatmap)
About the model
One network predicts both a pet/background mask and one of the 37 breeds in the Oxford-IIIT Pet dataset. On the held-out test set (1,109 images): Pet IoU 0.924, breed accuracy 92.4%. The ONNX file used here was checked against the original PyTorch model on that full test set before release.
The heatmap is Grad-CAM for the selected breed: warmer regions raised that breed's score. Because the classifier head is a small pooled MLP, the Grad-CAM gradient can be computed exactly from its weights, so the heatmap is built into the model itself (checked against PyTorch autograd Grad-CAM) and needs no backpropagation in the browser.
An out-of-domain check flags photos that look unlike the training data: it warns when the top breed logit is below 1.69 or less than 8.37% of the image is predicted as pet (thresholds set on validation data). On held-out data this kept 94.9% of real test pets, flagged 96.8% of non-pet photos (COCO), but flagged only 52.6% of dog breeds the model was never trained on (Stanford Dogs).
Limitations
- It only knows 37 breeds and always returns three of them. The out-of-domain warning catches most non-pet photos but only about half of unfamiliar dog breeds, and unfamiliar cat breeds were not tested.
- The mask is predicted at 224×224 and scaled up to your image, so edges are soft.
- The heatmap is a coarse 7×7 map scaled up to your photo. It shows where evidence for a breed came from; it is not a proof of why the model decided.
- The browser resizes your photo slightly differently from the Python pipeline used for evaluation, so results can differ a little from the reported test setup.