My goal

Inspire and empower builders to design and implement their own checkers for retrievability of data stored on Filecoin.

Side effect: Get people excited about the Checker Network and gain new subnet builders.

Talk title

Designing new retrieval checkers for Filecoin

Other options:

• The future of retrievability checks

• Retrievability checks: the next chapter

• Builder's guide to retrieval checking

• Let's build the next generation of Filecoin retrieval checkers

• Build your own Filecoin retrieval checker

• Beyond Spark: Designing new retrieval checkers for Filecoin

• Designing new retrieval checkers for Filecoin

Abstract

In this talk, we explore the process of checking retrievability of data stored on Filecoin, drawing on examples like RetrievalBot and Spark. We will break down the key steps involved, e.g. deciding what to check and how to perform a retrieval. A case study of Spark will show one possible implementation of those steps.

Attendees will learn how to design their own retrieval checkers. We will discuss different options for selecting what to check, data retrieval protocols, and building SP reputation based on check results. We will also discuss the financial aspects of running a retrieval checking system, including costs and potential revenue streams.

Outline

• Prior art:
  • RetrievalBot
    • Nodes running in datacenters
    • Sampling FIL+ deals
    • Retrieving full Piece
  • Spark
    • Nodes running on home computers
    • Sampling FIL+ deals
    • Retrieving a single payload block (an IPFS/IPLD block)

• Anatomy of retrieval checking
  1. Decide what to check (define retrieval tasks)
  2. (platform magic: tasking)
  3. Perform a retrieval and report results
  4. (platform magic: fraud detection, sybil protection, consensus evaluation)
  5. Aggregate measurements and process check results

• Case study: Spark
  1. What to check: a randomly sampled FIL+ deal
  2. Retrieval: retrieval of one IPLD payload block. Use IPNI to find the provider address + protocol to use. Graphsync and IPFS Trustless HTTP GW.
  3. Aggregate measurements to daily RSR scores. Build network-wide, per-miner and per-allocator reputation.

• Build your own checker
  • What to check:
    • A randomly sampled FIL+ deal
    • A randomly sampled PDP root
    • A smart-contract picks a deal based on some criteria
      • PDP smart contract can emit an event when a proof was submited
      • FWS smart contract can schedule a retrieval check based on QoS contract
    • A randomly sampled L2 deal (Storacha, Acave)
    • What else can you think of?
  • Retrieval
    • Payload (IPLD block) retrieval
    • Piece byte-range retrieval
    • Retrieve from L2 CDN (Storacha, Akave)
    • What else can you think of?
  • Aggregate and process results
    • Update the reputation of nodes
    • Trigger payment for the verified epoch
    • Slash deals that are not retrievable
    • What else can you think of?

• Example: <name TBD>
  • Describe the idea we are exploring with FilOz & Luca

• One more thing: funding
  • Costs:
    • Rewards for checker node operators
    • Checker subnet builder fee (pays for your work)
    • Checker Network fee (pays for our work)
  • Revenue
    • A cut from paid deals (FWS, L2)
    • What else can you think of?