Enhancing the Rarible NFT Indexer: A Journey to Deliver Web3 Speed

In the world of decentralized applications (dApps), fast and reliable data access is paramount to providing users with a seamless experience. As we delved into the exciting realm of non-fungible tokens (NFTs), we quickly realized that creating a robust and efficient NFT indexer was essential to meet the demands of our users. In this article, we'll discuss our journey towards optimizing the Rarible NFT indexer, why indexer delay is crucial for dApps, and how we approached and solved this challenge.

Why We Need an Indexer

The Data Challenge

Web3 applications rely heavily on data presentation to provide users with a rich and interactive experience. However, blockchain APIs often do not provide data in a format that can be readily used on the user interface. In the NFT domain, there is a wealth of data associated with NFTs, including orders, metadata, and aggregated data like floor prices and volumes. Unfortunately, this valuable information is not readily available on the blockchain and needs to be added.

The Solution: Combining On-Chain and Off-Chain Data

Our approach was to create an indexer that combines on-chain and off-chain data to serve requests for the user interface. This enables us to provide users with a comprehensive and engaging NFT experience by seamlessly integrating essential data that would otherwise be challenging to obtain directly from the blockchain.

Why Indexer Delay Matters for dApps

In the transition from web2 to web3, one key difference is the time it takes for transaction confirmations. While web2 actions are nearly instant, web3 transactions require confirmation time, which can lead to delays in data availability. Users, however, expect web3 applications to offer similar responsiveness. This is where indexer delay becomes crucial for web3 apps. To create a truly exceptional web3 application, the indexer must be as fast as possible.

The Frustration of Delay

We began to notice that user actions sometimes took an excessive amount of time, resulting in user frustration. To illustrate this, let's examine the data flow in a typical web3 app:

The user sends a transaction using their wallet.
The wallet submits the transaction, which is then confirmed on the blockchain.
Data is indexed, and only then is it ready to be consumed by the frontend.

How We Approached the Problem

To address the issue of indexer delay, we took a methodical approach. We started by creating a simple application that monitored newly minted NFTs on various chains. We measured the delay between block confirmation and the time when NFT data could be fetched from different sources, including our competitors.

Our investigation revealed that on Ethereum, our indexer was on par with the competition, and in some cases, even slightly faster. However, on Polygon, our competitors had a significant advantage, and indexer delay on Polygon was highly volatile.

How We Solved the Task

To tackle this challenge, we implemented several key optimizations:

Introducing Time Marks

We introduced a series of metrics called "time marks" to monitor every step in our data pipeline. These time marks helped us pinpoint where delays were occurring. For example:

Source to Scanner-In: The time between block confirmation and the initial event processing step in our data pipeline.
Enrichment-In to Enrichment-Out: The time needed to add additional off-chain information to the NFT.

Optimization Strategies

Our optimization efforts included:

Minimizing message resending delays within our Kafka-based pipeline.
Optimizing read requests for specific NFTs, reducing delays from seconds to milliseconds.
Addressing indexer delay degradation on Polygon, mainly caused by spam NFTs. This involved a complex optimization process, balancing improvements in one part of the pipeline with potential delays in others.
Switching to higher-performing RPC nodes on Polygon, as node delays varied significantly (from 200ms to several seconds).

What We Achieved

Our efforts bore fruit in several ways:

Enhanced Observability: We gained greater visibility into our pipeline, allowing us to identify areas that needed improvement early in the process.
Stable Indexer Delay: Indexer delay became more constant and typically stayed within the required interval.
Competitive Edge: Our indexer is now faster than our competitors, providing a superior user experience.

What We Recommend

Use Rarible API: Our indexer is now the fastest on the market, ensuring your dApp provides a top-tier experience.
Choose Nodes Wisely: Node selection is critical; not all nodes are created equal.
Measure, Measure, Measure: Continuous monitoring and measurement are essential for maintaining control over your application's performance.

In conclusion, our journey to optimize the Rarible NFT indexer highlights the importance of indexer delay in Web3 applications and the dedication required to deliver a seamless user experience. By implementing meticulous optimizations and embracing innovative solutions, we've not only met but exceeded user expectations, setting a new standard for NFT indexers in the market.