Visual semantic search for a million NFTs with CLIP

nftopia

It was Spring ‘22. The snow was meltin’, the birds were singin’, and my fellow ape Xi Chen was deep in the rabbit hole of crypto & NFTs.

As he navigated this labyrinth, he often found himself screaming “ooooo-oo-ah-ah-oooo”, which is ape-speak for “Yo, why can’t I simply search for NFTs by describing what’s in the image?”

Why? I can only speculate, but I presume he wanted to search for something like “ape driving a lambo”. I mean, I know that’s what I’d do!

As a bonus, if there were none, when we did get our first Lambo trading NFTs, we could sell a picture of us driving it as an NFT to get a second Lambo! An ape can dream!

Xi’s ape noises increasingly drew the ire of his wife. Luckily for him, he was friends with another ape, Reza Sohrabi, who was deep into NLP. He recalled Reza’s passionate howls, “ooooo-oo-oo-ooooooo”, which roughly translates to “Yo, this OpenAI’s CLIP model is pretty neat and lets you embed images & text in the same space with a few lines of code.”

Intrigued, he decided to try it out on a few thousand NFT images. He was so impressed with how easy it was that, rumor has it, his howls that night rivaled those coming out of Oakland Zoo. Needless to say, he slept on the couch that week.

When I met Xi & Reza back in April and saw how simple it was to build visual semantic search for NFTs, I jumped up on the chair like Tom Cruise on Oprah and screamed “ah-ah-oooo-oo-ah-ah”, which, in this case, translates to “Yo, I can easily scale this up to a million images and put it behind a web app.”

Immediately, the three of us started to thump our chests and run around in circles, howling and screaming at each other. After a good fifteen minutes, we finally settled down to chart our course and got to work.

And, in just a couple of weekends, we stood up NFTopia. It let users search roughly 1.1 million NFTs by simply describing the image (like Google Photos), and also browse visually similar NFTs.

We expected self-driving Lambos to magically show up in our living room the following day. They never did, but until then… Ape. Together. Strong. 🦍 🦍 🦍

Where Lambo?

On a serious note, while there’s some emphasis on NFTs in this post, I mostly use them as a use case to outline how to build a simple visual semantic search engine. You can easily swap NFTs out for an image set of your choice, like an e-commerce catalog or AI-generated images.

The post is split into three sections.

  1. getting a million NFT images using Alchemy
  2. embedding them with CLIP
  3. powering search with Pinecone

Along the way, I’ll also cover the techniques, tooling & infrastructure to build & scale this, and how easy it is thanks to the rapidly maturing tech stack underneath!

1. alchemy + nfts = Apegasm

The first step on our road to NFTopia was to get NFT images. And for a meaningful glimpse into this world, we wanted a sizeable number. Now, I had no idea how many NFTs there were in the world, but that didn’t stop me from instinctively yelling out “one million NFTs” while holding my pinky like Dr. Evil, cos obviously!

pictured: visual approximation of me

1.1. now what the hell’s an nft?

Before diving into the logistics of getting a million NFTs, I’ll say just enough about them for what follows to make sense. Now, I’m by no means an expert on NFTs, but I presume a lot of folks who talk about them aren’t either, so I’ll fit right in.

The obvious starting point on this topic is Pete Davidson’s SNL video.

If you watched it and still have questions, here’s more mumbo-jumbo to add to it. To add some cringe, I’ll also frame it as a hypothetical conversation between you and me. Again, this is my partially informed perception of NFTs, so don’t quote me on it!

The last bit to add here is that, at its highest aspiration, an NFT doesn’t even need a URI and can point to physical objects like cars, houses, and perhaps, planets and galaxies!

How exactly this gets enforced in the real world is beyond me, since I haven’t been smoking what they’ve been smoking. But I do know that Dennis Hope has already bought most of the solar system and sold several lunar plots. So if someone is trying to sell you a 256-bit integer representing a slice of the moon, do your due diligence and make sure it doesn’t conflict with a piece of paper already issued by the Lunar Embassy.

I mean, no one wants a uint256 vs. cellulose fiber supreme court case!

1.2. whatevs, just gimme nfts.

In case you’re wondering if you wasted a minute of your life reading the previous section, you’d be mostly right, especially with that last bit.

But remember why you’re here? To build a visual semantic search engine for NFT images! And where are these images? We just found out that their URIs are in the token metadata!

Now, you probably just cracked open a can of Red Bull, ready to yank this data out directly from the blockchain. Brave, you are, but a lot to learn, you have!

In classic crypto-contrarian fashion, we’ll instead use a centralized, for-profit organization to chew through all that raw blockchain data and feed us some easy-to-digest crypto-nft-goo. And for hungry baby birds like us, there’s no momma quite like Alchemy. In this centralized mommy, we trust!

To their credit, Alchemy makes this process comically simple with a single endpoint, getNFTsForCollection, that returns all NFTs & metadata for a given collection. On top of being simple, it’s also free & fast, with a generous free tier and capacity for high concurrency.

So, a seed set of about 22K collections translated to roughly 370K requests that pulled in metadata for a whopping 34 million NFTs in just a few hours!

Collection IDs

You can download the full set of ~22K collection ids from here (~1.5MB).

1.3. downloading a million images!

Thanks to Alchemy, I now had 34 million NFT locations, 26 million of which pointed to media assets.

IPFS, a distributed file store, was the most popular host, followed by base64 strings as inline data, and a mix of different servers for the rest. Here’s a quick look at the top sources. For a more granular view, you can download the raw file here (~35KB).

Source distribution for NFT media - mobile view

Source distribution for NFT media

At this point, the obvious instinct was to scream #yolo and start downloading. But before that, it was important to be mindful of a few things that I’ll briefly cover here.

  1. The process environment: When pulling a lot of files with little knowledge of what they are or do, knowing the environment the process runs in is pretty important. At the very least, the process should not have sudo access, and preferably, it should run in an isolated environment. It’s also important to understand how exactly these downloaded bytes are processed.
  2. Mime type & file size: NFTs can point to anything from small images (.png, .jpg) to large media (.gif, .mp4, .wav, etc.), so it’s best to check both mime type and file size before downloading. I mean, it’s no fun to accidentally download 1GB videos thinking they’re 1MB images, especially when yolo-ing with a million files!
  3. Retry strategy: Servers can be fickle, especially when you’re hammering them, so a retry strategy is a must.
  4. Storage: From a quick sample, the average file looked like a few MBs, which meant a million files would roughly translate to a few TBs. So, resizing & compressing images before saving to disk was a no-brainer.
  5. Concurrency: Perhaps the most important thing to consider was concurrency. While it goes without saying that downloading a million files needs it, a bit of thought on “how” would go a long way to optimize throughput. But, as an eager ape, I simply decided to throw a LOT of threads at it.

my concurrency strategy

And with that, I fired away my code and the logs went brrr, making me feel like one of those operators from the Matrix.

Logs that made me feel like cool stuff was happening.

… and here’s my machine humming on all cylinders cores.

My machine humming

After a few initial hiccups, I pulled 1.1 million images, amounting to 1.6TB of data, in just a few days. In fact, I hit my Xfinity data cap of 1.2TB first, but luckily, I only had to wait a couple of days for my next billing cycle to start.

Along the way, I learned a few things that I think are worth sharing.

  1. Xfinity’s “Advanced Security” was actually throttling my downloads because it likely thought my machine was part of some botnet. I found this out much later and had to disable it to get my full speed back.
  2. Decompression bombs are apparently files that, when loaded into memory, can take up several orders of magnitude more space than the original file and crash your system. Since I used Pillow to read downloaded bytes as images, I hit this error several times. I never dug in to see if it was indeed a malicious file masquerading as an NFT, but it was interesting nonetheless!
  3. IPFS, at first look, reminded me of torrents! Also, probably not a surprise, but it was almost an order of magnitude faster at night than during the day (consider this an anecdote at best).
  4. Since a large chunk of NFTs were cartoonish avatars, resizing (800x800px) & compression significantly reduced the storage footprint. In this case, the downloaded 1.6TB translated to just ~65GB of compressed jpgs on disk!
  5. Initially, to optimize CPU usage, I had async calls inside multiple thread pools inside a process pool (one per core). But it didn’t seem to do all that much given the heavy I/O skew, so I switched to a single thread pool for simplicity. It’s very likely this bit me later, when I bumped the thread count to several thousand, due to the GIL & threading overhead.

2. CLIP, a heavenly union between text & images.

By the following weekend, I had over a million images ready to be embedded by CLIP, a boringly straightforward process! In fact, this is all it takes to embed an image (source here).

import clip
import torch
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

with torch.no_grad():
    embedding = model.encode_image(preprocess(Image.open("nft.jpg")).unsqueeze(0).to(device))
    embedding /= embedding.norm(dim=-1, keepdim=True)

# pickle embeddings
# ...

And just like that, 1.1 million NFT images were batched, embedded and pickled in about half a day on my old 980Ti GPU!

Visual semantic search would now be as simple as embedding a user’s query and computing its dot product with the NFT embeddings! Better yet, retrieving NFTs similar to a given NFT would be just as simple. It would just mean using that NFT’s embedding as the query. Further out, it would be just as trivial to retrieve NFTs similar to arbitrary images!

clip?

I won’t go into the details of CLIP but at a high-level, it is a model that maps both images & text into the same vector space. It does this pretty well after having chewed through a LOT of images & associated captions from the web. To learn more, here’s a good reference from Roboflow.

3. thou shalt easily find thy neighbor with pinecone

Computing similarity over all NFTs and retrieving nearest neighbors is somewhat bearable with thousands, or even tens of thousands, of NFTs. But when you have millions, things start to slow down.

Enter approximate nearest neighbor (ANN) search.

At a very high level, this can be thought of as “bucketing” the entire embedding space and computing similarity over a handful of representative points from these buckets instead. Then, by only considering buckets close to the query, the search space shrinks to just a fraction of what it would’ve been otherwise.

There are several off-the-shelf solutions for this, with varying degrees of accuracy and speed. An early hero (also my first introduction) from almost a decade ago is Annoy from Spotify, while the current popular choice is Faiss from Facebook. Using either usually means managing both indices & deployment, which I had zero interest in doing.

So, while on the prowl for managed services, I stumbled onto Pinecone, which did just that while offering a simple interface and a generous free tier!

Here’s how simple it is to add vectors once you’ve created a project & an index.

import pinecone

pinecone.init(api_key="api_key")
index = pinecone.Index("nftopia")

# load embeddings
# ...

index.upsert(
    vectors=vectors,
    namespace="image-embeddings"
)

To make things even better, it supports both batching & async calls, so loading 1.1 million embeddings was very quick!

Once the index is ready, querying it is as simple as this.

# embed query
# ...

results = index.query(
    namespace="image-embeddings",
    top_k = 10,
    queries = [query_embedding],
    include_metadata=True,
)

… and retrieving similar NFTs is as simple as this.

# get anchor nft id
# ...

nft_embedding = index.fetch([anchor_nft_id], namespace="image-embeddings")

results = index.query(
    namespace="image-embeddings",
    top_k = 10,
    queries = [nft_embedding],
    include_metadata=True,
)

4. putting it all together

With the core pieces in place, it was time to put it all together. Since most of my familiarity was with Google Cloud Platform, I used a subset of their services for deployment. Functionally, this meant three pieces.

  1. A web GUI that would let users (i) type in text and view relevant NFTs, and (ii) browse visually similar NFTs. For this, I used Django on App Engine, which may have been overkill, but was fast to get up & running given my familiarity.
  2. A service to embed user queries with CLIP in real-time. For this, I simply wrapped CLIP with FastAPI and deployed it on Cloud Run.
  3. A storage solution to host & serve the NFT images. For this, I used Cloud Storage.

While (1) & (2) could have been bundled, I kept them separate for simplicity.

That’s it! So when you head over to NFTopia and type in a query, Django first fetches its embedding from the CLIP service. Then, it hits Pinecone for the nearest neighbors and renders them. For similar NFTs, Django hits Pinecone to first fetch the anchor NFT’s embedding and then retrieves its nearest neighbors.

All this happens pretty fast. CLIP takes about 150-200ms (GCR only supports CPU), and Pinecone takes another 100-150ms.

Also, both App Engine & Cloud Run automatically scale with traffic and are pretty cheap to run, while Pinecone requires moving to a paid tier to scale up, and needs manual management.

If you have any questions, feedback or want any data/code, feel free to hit me up on X or LinkedIn!

← all posts