At Personal the Second, our mission is to drive the following era of sports activities fandom – NFTs (non-fungible tokens) of professional athletes. Participant NFTs are far more than the equal of digital baseball playing cards, they’re the way forward for the sports activities collectibles market.
We’re serving to to paved the way. Followers and traders can observe real-time market values for NFL and NBA participant NFTs by way of our service. We additionally present an on-line market for purchasing and promoting NFTs. Consider us like Ameritrade or Coinbase, however on your sports activities NFT property.
Most significantly, we have now additionally created a fantasy sports activities platform known as The House owners Membership that debuted with a fantasy soccer league for the 2021-22 season. We gave out $1.5 million in prizes to rivals primarily based on their NFL fantasy groups, composed of the participant NFTs they owned.
Whereas proudly owning different sorts of NFTs has change into like amassing artwork, we’re gamifying sports activities NFTs so as to create much more utility and pleasure round them. This additionally creates larger alternatives for savvy merchants to make cash shopping for and promoting participant NFTs.
This makes my job as CTO of a small startup extraordinarily fascinating, as I needed to oversee constructing a knowledge infrastructure that supported:
- A fantasy sports activities league the place each knowledge ingestion and concurrent utilization spikes throughout sport days
- A participant leaderboard with real-time outcomes
- Safe, environment friendly and quick knowledge change with the Ethereum blockchain the place participant NFT knowledge is saved
- Extra standard use circumstances akin to inside monetary reporting
It’s a tall order. Contemplating how rapidly the Net 3.0 house has been evolving, it’s no shock that the primary model of our knowledge infrastructure didn’t help all of those calls for. Happily, we had been in a position to rapidly pivot after we found a real-time analytics database tailored for our quick evolving wants.
DynamoDB: Analytics Limitations Revealed
I joined Personal The Second in 2021 whereas we had been nonetheless in stealth mode. I rapidly found that to construct our fantasy sports activities league and NFT market, we would wish two primary sources of knowledge:
- Actual-time sport scores and participant statistics, each provided by an exterior knowledge supplier
- Blockchain nodes akin to Alchemy that enable us to each learn and write details about NFTs and customers’ crypto wallets to the blockchain
I constructed the primary model of our knowledge infrastructure wholly round Amazon’s DynamoDB database within the cloud. As our database of report, DynamoDB was nice at ingesting exterior knowledge, which we saved inside a single desk in DynamoDB. We additionally had smaller DynamoDB tables storing our consumer data and the mechanics of our fantasy sports activities contests. Apart from our common weekly prime staff contests and cumulative participant leaderboards, we ran contests akin to worst staff, in order that customers with dangerous NFT playing cards nonetheless had an opportunity to win.
To run these contests, we would have liked to run advanced, large-scale queries utilizing the DynamoDB knowledge tables. And due to the variety of contests, we had plenty of totally different queries. That’s the place DynamoDB’s analytical limitations reared their ugly heads.
As an example, to make any DynamoDB question run fairly quick, we first wanted to create a secondary index with a form key tailor-made for that question. Additionally, DynamoDB, as a NoSQL database, doesn’t help SQL instructions akin to JOINING a number of tables. As a substitute, we needed to denormalize our primary DynamoDB desk by importing the entire consumer info that was saved in separate DynamoDB tables. This had main downsides, akin to difficulties protecting knowledge precisely up to date throughout sport days, as effectively needing additional storage for a lot redundant knowledge in our primary desk. It’s all deeply-technical work that requires a developer expert in DynamoDB analytics. And they’re a uncommon and costly bunch.
Thrown into the combo was the object-relational mapping (ORM) software we had deployed known as Dynamoose. Dynamoose supplies helpful options together with a programmatic API and a schema for the schemaless DynamoDB. Nonetheless, the tradeoff for that further knowledge modeling is plenty of further latency for our queries. In our case, that resulted in a question latency of three seconds.
All in all, making an attempt to make DynamoDB help quick analytics was a nightmare that might not finish. And with the NFL season set to start out in lower than a month, we had been in a bind.
A Sooner, Friendlier Resolution
We thought of just a few options. One was to create one other knowledge pipeline that might mixture knowledge because it was ingested into DynamoDB. This could require creating a brand new desk, which might’ve required just a few additional weeks of dev time. One other was to scrap DynamoDB and discover a conventional SQL database. Each would have required plenty of work.
After discovering Rockset by way of an AWS weblog on creating leaderboards, we wasted no time in beginning to construct a brand new customer-facing leaderboard primarily based on Rockset. One of many first issues my staff and I observed was how straightforward Rockset was to make use of. I’ve labored with virtually each database on the market prior to now twelve years. Rockset’s UI is actually the perfect I’ve labored with.
The SQL question editor is top-notch, monitoring question historical past, saving queries and extra. It made my six builders, who all know SQL, instantly productive. Simply primarily based on skimming the SELECTs and JOINs in just a few pattern Question Lambdas, they understood what sort of knowledge that they had and find out how to work with it. By the top of the day, that they had actually constructed functioning SQL queries and APIs with none exterior assist. And with Rockset’s Converged Index™ and [automatic query optimizer](https://rockset.com/weblog/sql-query-planning-for-operational-analytics/, all queries are quick and failure proof. We don’t need to construct a customized index for each question like we do with Dynamo.
Through the use of Rockset, we saved weeks of man-hours making an attempt to beat and compensate for DynamoDB’s analytical limitations. We had been in a position to roll out a complete new participant leaderboard in simply three weeks.
Developer productiveness is nice, however what about question efficiency? That’s the place Rockset actually shined. As soon as we moved the entire queries feeding our leaderboards to Rockset – 100 Question Lambdas in whole – we began with the ability to question our knowledge in 100 milliseconds or much less. That’s not less than a 30x pace improve over DynamoDB.
Rockset’s serverless mannequin additionally made scalability very easy. This was essential to optimize each efficiency in addition to value, since our utilization is so dynamic. In the course of the first season, our peak concurrent utilization throughout sport occasions – Monday and Thursday nights, and all day Sundays – was 20x greater than throughout off peak occasions. I’d merely flip a swap and bump up the scale of our Rockset occasion throughout sport days and never fear about any bottlenecks or time outs.
We gained a lot confidence in Rockset’s pace, scalability, and ease of use that we rapidly moved the remainder of our analytical operations to Rockset. That features ten knowledge collections in all, the most important of which holds 15 million information, that retailer key knowledge, together with:
- 65,000 NFT transactions value $1 million in our first season
- the 23,000 present customers in our system together with information of the 160,000 NFTs they personal
- our largest knowledge assortment – 400,000 information ingested from blockchains for NFT transactions associated to our good contracts
DynamoDB stays our database of report, connecting to microservices syncing with the blockchain and streaming knowledge feeds. However actually each knowledge retrieval and analytical calculation now goes by way of Rockset, from loading the participant NFT market and viewing the entire pricing statistics and transactions, to the consumer playing cards. Rockset syncs with DynamoDB continually, pulling new sport scores each 5-10 seconds and syncing with the blockchain the place NFT and consumer pockets knowledge is saved, and writing all of that into an listed assortment.
We additionally do all of our inside administrative reporting by way of Rockset. Rockset JOINs market, consumer, and funds info from separate DynamoDB tables to supply mixture experiences that we export as CSV information. We had been in a position to produce these experiences in mere minutes utilizing the Collections tab in Rockset.
Constructing this in DynamoDB, against this, would have required scripts and handbook becoming a member of of information, each of that are fairly error-prone. We most likely saved days if not weeks of time utilizing Rockset. It additionally enabled us to run giveaways and contests for customers who had full set collections of NFTs in our system or spent X {dollars} within the market. With out Rockset, aggregating our ever-expanding assortment of DynamoDB tables would have required an excessive amount of work.
Future Plans
Final season we gave out $1.5 million in prizes. That was actual cash that was on the road! Nonetheless, it was primarily a proof of idea for our Rockset-based analytics platform, which carried out flawlessly. We’ve reduce the variety of question errors and timed-out queries to zero. Each question runs quick out of the field. Our common question latency has shrunk from six seconds to 300 milliseconds. And that’s true for small datasets and bigger ones.
Furthermore, Rockset makes my builders tremendous productive, with the easy-to-use UI and Write API and SQL help. And options like Converged Index and question optimization remove the necessity to spend invaluable engineering time on question efficiency.
For the approaching NFL season, we’re speaking to numerous potential large title companions within the sports activities media and fantasy enterprise. They’re coming to us as a result of we’re the one platform I do know of at present that integrates the blockchain on prime of a utility-based NFT answer.
We’re additionally engaged on plenty of backend modifications akin to constructing new APIs in Rockset and new integrations. We’re additionally making ready for 10x progress on each dimension – consumer base, participant NFTs, knowledge information and extra. What gained’t change is Rockset. It’s confirmed to us that it may deal with all of our wants: ultra-fast, scalable and sophisticated analytics which can be straightforward to develop and cost-effective to handle.
