\\nThis is a combined proposal for the following RFPs: [NEAR Lake](https://dev.near.org/infrastructure-committee.near/widget/app?page=rfp&id=3), [BigQuery and Data Pipelines](https://dev.near.org/infrastructure-committee.near/widget/app?page=rfp&id=4) and [KitWallet](https://dev.near.org/infrastructure-committee.near/widget/app?page=rfp&id=1). \\n# Organization\\n\\n\\n### Primary person\\u{2019}s contact information\\n\\nPavel Kudinov \\u{2013} pavel@blocklake.io; Telegram @paulksg\\nWebsite: [https://www.blocklake.io/](https://www.blocklake.io/)\\n\\n\\n### Size of the whole team\\n\\nCurrently two people\\n\\n\\n### The size of the engineering team\\n\\nCurrently two people \\u{2013} Pavel Kudinov and Eduardo Ohe. We plan to engage DevOps/SRE on a contract when necessary.\\n\\n\\n### Team location\\n\\nVancouver, Canada\\n\\n\\n### Team background and bios\\n\\nFor the past ~2 years, we both have been the senior-most members of the Pagoda Data Team, working on NEAR Lake, Databricks Lakehouse powering NEAR Foundation analytics, BigQuery and Dune integration, as well as optimizing the performance of critical databases of EnhancedAPI, Explorer, and QueryAPI.\\n\\n**Eduardo Ohe** \\u{2013} Currently a staff engineer at the Pagoda team running data infra, including hands-on work on NEAR Lake, Databricks, Public Datasets/BigQuery/Dune integrations, and KitWallet. He shared and advocated for NEAR data solutions in the [Google tech blogs](https://cloud.google.com/blog/topics/partners/increasing-blockchain-accessibility-for-devops) and in the last [Databricks DATA+AI Summit](https://www.databricks.com/dataaisummit/session/sponsored-google-scale-your-databricks-workloads-15k-nodes-google-cloud) as a co-speaker. Over 28 years of experience with software engineering and big data, he has an MBA and a Bachelor of Computer Science that helped him to make bridges between business needs and tech solutions. He has experience as a CTO | Co-founder running his own BI/CRM company with +30 employees and successful exit. He also led data engineering projects for [petabyte-scale datasets at Twitter](https://blog.x.com/engineering/en_us/topics/infrastructure/2022/scaling-data-access-by-moving-an-exabyte-of-data-to-google-cloud) ensuring its [data quality/reliability.](https://blog.x.com/engineering/en_us/topics/infrastructure/2022/data-quality-automation-at-twitter) \\n\\n**Pavel Kudinov** \\u{2013} Currently a senior manager at the Pagoda team running data infra, including NEAR Lake. Over 18 years of experience in building Data Science products. More than 5 years serving in Director of Engineering or CTO-like roles, experience running private data consulting and creating startups from scratch.\\n\\n\\n# Professional Experience\\n\\n\\n### Examples of relevant experience with NEAR blockchain\\n\\n\\n\\n* NEAR Lake Primitives library, simplifying data extraction from NEAR Lake JSON files: [https://github.com/near/near-lake-framework-js/tree/main/packages/near-lake-primitives](https://github.com/near/near-lake-framework-js/tree/main/packages/near-lake-primitives) \\n* Databricks Lakehouse, powered by NEAR Lake: [https://github.com/near/near-public-lakehouse](https://github.com/near/near-public-lakehouse)\\n* BigQuery integration: [https://console.cloud.google.com/bigquery?p=bigquery-public-data&d=crypto_near_mainnet_us&page=dataset](https://console.cloud.google.com/bigquery?p=bigquery-public-data&d=crypto_near_mainnet_us&page=dataset)\\n* NEAR QueryAPI, powered by NEAR Lake: [https://github.com/near/queryapi](https://github.com/near/queryapi)\\n\\n\\n### Non-technical capabilities:\\n\\n\\n\\n* **Leadership.** Pavel has experience building and running product, data, and software engineering since 2008.\\n* **Data Consulting.** Both Eduardo and Pavel have experience running data consulting (data engineering and business intelligence) for small and big businesses in Brazil and Singapore.\\n* **Business Development**. we have experience working with B2B clients and establishing partnerships.\\n* **Project Management**. Pavel has experience running large-scale, multi-quarter projects across multiple teams involving data-intensive applications with high SLAs.\\n\\n\\n### In-House Resources;\\n\\n\\n\\n* Design, UX/UI\\n * For this RFP, we don\\u{2019}t need Design or UX work. However, we have preliminary agreements with existing Pagoda designers to help with design work if necessary.\\n* Project management\\n * Pavel Kudinov will be handling project management\\n* Customer support\\n * Eduardo and Pavel will share handling customer support. Based on our experience so far, there are not many daily requests.\\n* BD / Marketing\\n * Pavel and Eduardo have experience with BD and partnership management from our consulting experience. For Marketing, we will rely on external partners known in the NEAR ecosystem when required.\\n\\n\\n### Portfolio of relevant work\\n\\nFor this RFP, the most relevant is that we had been working with NEAR Lake for 2 years and significantly contributed to many tools running on top of it.\\n\\n\\n# Body\\n\\nBlockLake Inc will be focusing on the following:\\n\\n\\n\\n* raw blockchain data extraction, hosting, and serving;\\n* building data pipelines for analytics, powering Near Foundation, Dune and BigQuery;\\n* critical APIs for most popular and essential use cases, such as native and FT token balance history;\\n* consulting services for external clients;\\n* In the future, multichain indexing and APIs.\\n\\n\\n### The main approach and solution of the proposal\\n\\n\\n\\n1. Take over existing infrastructure for NEAR Lake: hosting NEAR Indexer nodes and the S3 bucket.\\n2. Utilize our expertise to build better and even cheaper access tools for people using NEAR Lake and offering it based on subscription:\\n 1. Block index to query only relevant blocks\\n 2. Block and transaction streaming through WebSockets and/or Webhooks\\n3. Take over critical data pipelines for integrating Dune (and maintaining ~$350k annual savings) and BigQuery.\\n\\n\\n### Why should the committee choose this project?\\n\\n\\n\\n1. The team behind this proposal has profound knowledge of the domain and technical architecture. Zero migration and onboarding costs.\\n2. Internal and External users rely on the high-quality datasets that we already built; other data sources are not as reliable. Data Analytics and AI use cases require high-quality data.\\n3. Dune\\u{2019}s proposal cost was reduced by 50% only because we did most of the work on the NEAR Lakehouse.\\n\\n\\n### Will this solution be open source? What licensing will be used?\\n\\nWe will keep the same open source license that NEAR GitHub repos have currently for all public goods code \\u{2013} the NEAR Lake, client libraries and data pipelines.\\n\\n\\n# Goals / Milestones\\n\\n\\n\\n* By February 2025\\n * Incorporate the company, establish operations\\n * Takeover existing Lake infrastructure as is with existing costs, ensuring non-disruptive transition of all existing clients\\n * Replace Databricks with the cheaper alternative to empower Dune and BigQuery use-cases\\n* By July 2025\\n * First revenue from consulting services and custom indexers\\n * First revenue from NEAR Streams \\u{2013} a cheaper way to stream transactions to clients, reducing network costs and latency.\\n * Reduction of infrastructure costs to run NEAR Lake (using dedicated servers)\\n* By December 2025\\n * Expand the Lake for another blockchain(s)\\n\\n\\n# Service Level Agreements \\n\\n\\n### Support\\n\\n\\n\\n* **Support Availability**: Community-based support only (e.g., through forums, GitHub issues, or a developer community Slack/Telegram/Discord channel)\\n* **Response Time**: No guaranteed response time, but best-effort responses within 24-48 hours\\n* **No SLA Credits**: No financial credits or compensation for outages or performance issues to the users.\\n\\n\\n### Legal\\n\\n\\n\\n* **Terms & Conditions:** We will provide a clear T&C and Privacy Policy facilitating fair usage of services, following best practices.\\n* **Fair Use Policy**: Explicit statements about the intended use for development and the prohibition of production workloads\\n* **Termination Rights**: The right to suspend or terminate the service for users who exceed fair use or misuse the service. IC must be informed about these.\\n\\n\\n### Technical\\n\\n\\n\\n* **Availability**\\n * **Uptime Guarantee**: 99.0% for Lake and 95% for Public Datasets monthly uptime \\n * **Downtime Allowance**: Up to 1% for Lake and 5% for Public Datasets of downtime per month\\n* **Data freshness**:\\n * NEAR Lake: 99% monthly SLA that the Lake is less than 5 blocks from the latest finalized block.\\n * Public Datasets: 99% monthly SLA that all datasets are within 1 hour from the tip of the network.\\n* **Rate Limiting**:\\n * As defined in the rate limit plan with gradual reductions for both Lake and public datasets\\n* **Rate Limit Enforcement**: Strict enforcement with immediate rate limit application once thresholds are reached for both Lake and public datasets\\n* **Updates and Changes**\\n* **Notice for Breaking Changes**: At least 7 days\\\' notice for any breaking changes in the Lake services or infrastructure\\n\\n\\n# Usage & Examples\\n\\n\\n\\n* NEAR Lake is critical for the Ecosystem\\n * NEAR Lake has 75 active users with ~103 blocks/second served on average\\n * 2 Billion HTTP requests served monthly\\n* Public Dataset is critical for the Ecosystem\\n * BigQuery is used by Nansen, SWEAT, NEAR NDC and others\\n * Allows to save ~$350K on Dune integration annually\\n\\n\\n# Budget\\n\\n\\n### Budget Total\\n\\nWe ask for a budget for 2 years, with possible extension further upon review. The budget includes infrastructure growth costs, as we expect to put engineering effort to reduce them.\\n\\nProposed: **$327,500** / quarter\\n\\n\\n### Quarterly Budget Breakdown\\n\\n\\n<table>\\n <tr>\\n <td>Estimated Costs\\n </td>\\n <td>Proposed (USD) / Quarter\\n </td>\\n </tr>\\n <tr>\\n <td>Labour (Data Engineering, DevOps, SRE)\\n </td>\\n <td>\\n$225,000\\n\\n </td>\\n </tr>\\n <tr>\\n <td>Lake Infra (Mainnet & Testnet)\\n </td>\\n <td>\\n$37,500\\n\\n </td>\\n </tr>\\n <tr>\\n <td>Public Datasets Infra (Databricks, Dune, BQ, etc)\\n </td>\\n <td>\\n$25,000\\n\\n </td>\\n </tr>\\n <tr>\\n <td>KitWallet Infra\\n </td>\\n <td>\\n$7,500\\n\\n </td>\\n </tr>\\n <tr>\\n <td>SaaS subscriptions\\n </td>\\n <td>\\n$1,500\\n\\n </td>\\n </tr>\\n <tr>\\n <td>Admin, Legal and Accounting\\n </td>\\n <td>\\n$6,000\\n\\n </td>\\n </tr>\\n <tr>\\n <td>Customer Success, Documentation, Marketing\\n </td>\\n <td>\\n$25,000\\n\\n </td>\\n </tr>\\n <tr>\\n <td>\\n </td>\\n <td>\\n$327,500 \\n\\n </td>\\n </tr>\\n</table>\\n\\n\\nWe are happy to discuss the Fiat/$NEAR ratio for labor costs, up to 50% of $NEAR. The payment for both Fiat and $NEAR should be prepaid for every 3 months.\\n

NEAR Lake, Public Datasets and KitWallet Proposal by former Pagoda Data Team

Two core developers on the Data Platform team propose to take over NEAR Lake, Public Datasets powering analytics, and KitWallet, on a 2-year contract. 

	Receipt	Action	Method	From		To	Value	Gas Limit

Детали транзакции

Overview

Execution Plan

Enhanced Plan

Tree Plan

Receipt Summary

Comments