Bookstore listings are noisy. Hiki’s Python back end turns them into a catalog that readers can filter and search.
Every day, Google Cloud Scheduler starts Hiki’s scrapers. They are built on aiohttp, with rate limiting and exponential backoff, and they collect product data from several bookstores: names, prices, discounts, stock status and images. Change detection flags new arrivals, price drops and stock changes, so readers never miss an update.
A raw product name from a bookstore is often noisy and inconsistent. Hiki’s enrichment pipeline orchestrates several APIs to find the manga behind it. AniList is the primary source for scores, genres, tags, synopsis and cover art. Shikimori translates titles for product names written in Cyrillic, and Jikan and Kitsu are fallbacks for the edge cases. The pipeline generates title variants, handles both Cyrillic and Latin characters, and resolves even badly mangled names to their canonical manga entries. Bookstores list no genres, so this step is what makes Hiki filterable: the 21 genres and 100+ tags that readers filter by come from AniList.
The API is FastAPI with async MongoDB through the Motor driver. MongoDB stores the catalog as flexible documents with compound indexes. The Next.js front end reads from this API and renders the catalog in every supported language.
The back end runs on Google Cloud Run, next to the front end, and Cloud Build deploys it on every push to main. Cloud Armor sits in front of it with geo-restrictions and an allowlist for crawlers.