AI On-Device: Cooking from your fridge with Gemini Nano, MobileCLIP and Gemma
What happens when you commit to running every piece of AI in your app on the phone itself — no cloud, no API keys, no server logs of your users' kitchens? We built AI Pantry, an Android app that scans your fridge with the camera, recognizes the ingredients and generates recipes — 100% on-device. Our first choice was the obvious platform one: Gemini Nano, Android's new built-in on-device model. One dependency, zero megabytes of weights in the APK — perfect, except it only runs on brand-new flagship phones (Pixel 9/10, Galaxy S25/S26), not on the devices most users actually own. So we pivoted to plan B: bundle a single multimodal LLM (Gemma 4 E2B) and use it for both vision and text. It worked. It also took 5–10 minutes per scan on a mid-range phone (Pixel 7). This session is the story of fixing that, with real code and real measurements: - Gemini Nano via ML Kit GenAI Prompt API: the model lives in AICore, an Android system service — not in your app. You get multimodal prompts (photo + text in, text out) against weights you never ship, download, or update: zero MB in your APK, hardware acceleration included. The integration is one dependency and one rule: availability is a runtime question, so gate every single call with checkStatus() — the same phone can answer differently over time (model not yet downloaded, or evicted). Used this way, Nano becomes a free accelerator on supported devices, with your own detector as the everywhere-fallback — never the foundation the app stands on. - One model to rule them all — and why it crawled: we bundled Gemma 4 E2B and asked it to look at the fridge photo and write the ingredient list as JSON. The catch is how multimodal LLMs work: first the photo becomes ~1,000 image tokens that must all be processed before a single word of output (the prefill), then the answer is generated one token at a time (the decode) — hundreds of sequential passes through a 2-billion-parameter model. On a mid-range phone, that arithmetic adds up to a 5-to-10-minute scan. Know this two-phase cost model before promising any multimodal UX. - Zero-shot ingredient detection with MobileCLIP-S2 on LiteRT: recognition doesn't need a model that writes — it needs one that compares. CLIP-family models use two small encoders that turn an image and a phrase into vectors in the same space, so "is this a watermelon?" becomes a cosine similarity instead of a generated sentence: one fixed forward pass of a ~140 MB encoder, no prefill, no word-by-word decode. To make it work we built the vocabulary ourselves — a curated list of hundreds of ingredient labels — embedded it offline on a laptop, and bundled just the vectors (~4.5 MB): the phone only runs the image encoder, and the "classifier" is an editable text file. Add an ingredient? Add a line. Result on the same phone: minutes → seconds. - Where the LLM earns its keep: recipes are creation, not recognition — the one job no closed vocabulary can do, so Gemma stays, via LiteRT-LM, and we make it feel fast: two-stage prompting (first a few-token list of recipe ideas, the full recipe only when you tap one — shorter prompts, less waiting), defensive JSON parsing with silent retries (small models may break format contracts, plan for it), and text-only generation streaming at 15–25 tok/s — on a phone, watching the recipe write itself is a feature, not a wait. - Shipping it: each model travels differently. Gemini Nano costs you zero megabytes — AICore owns it. MobileCLIP's ~140 MB encoder ships inside the APK like any asset. But Gemma's 2.6 GB of weights fit in no APK, and hand-rolling a downloader for multi-GB files (resume, integrity, metered connections) is a project in itself — so they travel as AI packs via Play for On-device AI, riding Google Play's own delivery pipeline. The user just installs the app; distribution stops being your problem. Key takeaways: 1. Match the model class to the task: perception ≠ generation. Recognizing what's in a photo is a comparison problem; writing text is a generation problem. Using a generative model for perception cost us minutes per scan — swapping in a discriminative one brought it down to seconds, on the same phone. 2. Platform AI is a capability, not a foundation. Gemini Nano/AICore gives you a free, zero-MB multimodal model — but only on recent flagships, and availability changes at runtime. Detect it with checkStatus() on every use, exploit it when present, and always have your own fallback underneath. 3. The cheapest ML you'll ever ship is a text file. With CLIP-family models, precomputing label embeddings offline turns zero-shot classification into "an editable list of ingredient names + ~4.5 MB of vectors" — no dataset, no training run, no ML infrastructure. Extending the classifier is adding a line. 4. Learn the two-phase cost model before promising any multimodal UX. A photo becomes hundreds of image tokens that must all be processed before the first output word (prefill), then the answer comes out one token at a time (decode). That arithmetic — not the model's quality — decides whether your feature takes seconds or minutes. 5. With small on-device models, defensive parsing IS the API contract. They will occasionally break your output format: plan for lenient JSON parsing and silent retries from day one, so the user sees results, not errors.