AI evaluation
Can you fool our AI with a fake bin?
Every bin on SmartBin must pass an AI photo check before it reaches the map. Below is our labelled test set, real dustbins and look-alikes picked to trick the model, with the verdict and the model's own reason for each. Nothing here is hand-picked after the fact: an admin can re-run the whole set live.
Nine layers between a fake bin and the map
The AI is one layer. The API enforces all of them, so skipping the app doesn't skip the checks.
- 1
Sign-in required (Amazon Cognito)
Stops: Anonymous spam
- 2
Live camera only in the app
Stops: Gallery uploads, old or downloaded photos
- 3
Upload links only for JPEG, PNG or WebP, into your own folder, for 5 minutes
Stops: Other file types, writing into someone else's folder
- 4
The photo must be yours
Stops: Reusing another person's upload
- 5
The file must really be an image, checked from its bytes
Stops: A text file renamed .jpg (deleted on the spot)
- 6
AI check: only a purpose-made bin in a public place passes
Stops: Buckets, post boxes, hydrants, barrels, garbage piles, drawings (photo deleted)
- 7
No other bin within 15 m
Stops: Registering the same bin twice
- 8
Monthly limit per user, set by admins
Stops: Bulk registration from one account
- 9
Missing reports only from within 20 m
Stops: Removing real bins from home
How we test, and what it can't catch
- Test set: 20 photos from Wikimedia Commons, each labelled by hand. "Hard" cases are genuinely borderline and don't count toward pass or fail. Photo credits are listed here.
- Pass bar: no fake accepted, and every clear dustbin accepted.
- Same code as production: the evaluation calls the exact check the API runs on every new bin. Rerun it yourself with
npm run eval:vision. - The trade-off we chose: when we let the model accept improvised bins (drums, barrels) if it saw waste in them, it imagined a bin liner in an empty barrel and accepted a bag of rubbish. So only purpose-made bins pass. The cost: a real bamboo basket bin is rejected.
- Limits: one clear photo of a real bin passes wherever it was taken, and phone GPS can be faked. That's why the AI is backed by the live camera, the 15 m and 20 m rules and the monthly limits.