Alternative Architecture

TL;DR: Reuse as much of the existing PDP components as possible. Conceptually, Retrievability Checking should be just a boolean flag added to PDP deals (”yes, I also want retrievability”). The retrieval verifier can be very closely tied to the PDPVerifier contract and reuse its state to keep track of what should be checked for retrievability.

I am also arguing that this boolean should be actually set to true by default - what’s the point of storing data if you cannot retrieve it?

• Discussion with FilOz:

  https://space-meridian.slack.com/archives/C0838QS1CVC/p1747824258480759

High-level overview

In PDP, there is a concept of “proving period” (24h on mainnet, 30min on testnet). Every proving period, SP is required to post N “challenges per proof set” (hard-coded to 5 in Curio) every proving period. A challenge is an inclusion proof for a block inside the proofset that’s chosen in (deterministically) randomly way.

Let’s do the same for Retrievability Checking:

• Every proving period, the checkers should perform N retrieval requests for each proof set.

• Each retrieval request should pick a random root from the proofset.

• The service contract will check whether aggregated RSR meets the minimum required SLA.

• Suggested parameters:
  • Retrievals per period (hardcoded for June 6): 100 for mainnet, 2 for testnet.
  • SLA requirement: 90% for mainnet (10 out of 100 requests can fail), 50% for testnet (1 out of 2 requests can fail)

Components

• HotStorageService - combines PDP+Retrievability, will be based on PDPSimpleServiceWithPayments, needs to be built

• PDPVerifier - existing smart contract

• RetrievabilityVerifier - new smart contract we need to build
  • RetrievabilityVerifier is tied to PDPVerifier. (I.e. RetrievabilityVerifier’s constructor requires the addresss of PDPVerifier contract).
  • What does it mean:
    1. It’s not possible to schedule retrievability checks for content not stored via PDP.
    2. In the simplest form, VerifiabilityVerifier will have a hardcoded link to exactly on PDPVerifier contract.
  • The first point is an inherent part of this design.
  • The second point can be easily lifted, if HotStorageService.createProofSet() can forward to our verifier not only the proofset id, but also the address of the PDPVerifier contract owning this proofset. We can add this after June 6.

Workflows

A new deal is created

IIUC, this means creating a new PDP ProofSet with the attached payment rails.

• The client or SP calls HotStorageService to create a new deal and indicate whether retrievability is required.

• The SP calls PDPVerifier.createProofSet(potentially via the service contract), which:
  • calls HotStorageService.createProofSet
  • emits ProofSetCreated event

• HotStorageService.createProofSet will call RetrievabilityVerifier.createProofSet BUT only if the proofset is expected to be retrievable (the contract includes retrievability SLA).
  • This needs to be implemented (by FilOz).

• RetrievabilityVerifier.createProofSet will initialise the internal state tracking retrievability of this proofset and emint ProofSetCreated event.

• The Checker node listens for the event RetrievabilityVerifier.ProofSetCreated and updates its off-chain state to prepare for retrievability checking.

sequenceDiagram
  SP->>HotStorageService: Create a new deal (retrievable=y/n)
  SP->>PDPVerifier: createProofSet() (potentially via service contract):
  PDPVerifier->>HotStorageService: createProofSet()
  HotStorageService->>RetrievabilityVerifier: createProofSet() (ONLY if retrievable)
  RetrievabilityVerifier->>RetrievabilityVerifier: emit ProofSetCreated
  PDPVerifier->>PDPVerifier: emit ProofSetCreated
  loop on RetrievabilityVerifier.ProofSetCreated
    CheckerNode->>CheckerNode: Initialise off-chain state
  end
  

A new proving period starts

For each proofset, once a day on mainnet (every 30 min on testnet), the SP (Curio) sends a TX calling PDPVerifier.nextProvingPeriod(proofSetId,...).

• The smart contract function calls HotStorageService.nextProvingPeriod(proofSetId,...).

We need to modify the HotStorageService as follows:

• HotStorageService.nextProvingPeriod(...args) must check whether the proofset is expected to be retrievable.

• If the proofset should be retrievable, then the service must call RetrievabilityVerifier.nextProvingPeriod(...args)

The RetrievabilityVerifier contract updates its internal state that’s recording retrieval results and emits NextProvingPeriod event.

The Checker node listens for this event (RetrievabilityVerifier.NextProvingPeriod) to start a new round of retrieval checks for a proofset. (Note: this event is emitted only for proofsets that are retrievable according to HotStorageService internal state.)

1. Add this proofset to the off-chain list of proofsets to check (unless the proofset was already added before).

2. Next, the node schedules M retrieval checks for this proofset in the next T minutes.
   1. M can be initially set to N (number of retrieval requests per proving period). Later, we should use a random value so that SP’s don’t know how many retrieval checks to expect.
   2. T is the same as the length of the proving period.

3. When the time comes to run a retrieval check for a proofset:
   1. The node picks a random root CID in the proofset by reading the on-chain state of the linked PDPVerifier contract.
   2. The node attempts to retrieve that root CID from the provider
   3. The result is reported to the RetrievabilityVerifier.
      • I think we may need to batch these writes to save gas fee, but that’s beyond the minimal MVP scope.

sequenceDiagram
  loop On proofset period start
    SP->>PDPVerifier: nextProvingPeriod(proofSetId, ...)
    PDPVerifier->>HotStorageService: nextProvingPeriod(proofSetId, ...)
    HotStorageService->>RetrievabilityVerifier: nextProvingPeriod(...) (ONLY if retrievable)
    RetrievabilityVerifier->>RetrievabilityVerifier: emit NextProvingPeriod()
  end
  loop on RetrievabilityVerifier.NextProvingPeriod
    CheckerNode->>CheckerNode: Schedule retrieval checks
  end
  loop On schedule
    CheckerNode->>PDPVerifier: Pick random root CID
    CheckerNode->>SP: Attempt retrieval
    CheckerNode->>RetrievabilityVerifier: Report result
  end

A deal ends

• At some point, PDPVerifier.deleteProofSet() is called to clean up the on-chain state and reduce amount od data stored.

• PDPVerifier.deleteProofSet() calls HotStorageService.deleteProofSet()

• HotStorageService.deleteProofSet() calls RetrievabilityVerifier.deleteProofSet() but ONLY if the proofset was retrievable

• RetrievabilityVerifier.deleteProofSet() cleans up the on-chain state

SLA evaluation

Inside HotStorageService.nextProvingPeriod(...args), the service contract will ask RetrievabilityVerifier for the retrieval results reported for the proofset in the previous proving period and decide whether the SLA was met.

RetrievabilityVerifier interface (draft)

// This interface needs to be implemented by FilOz's service contract
interface RetrievabilityListener {
  function setIsRetrievable(uint256 setId, uint256 rootId, bool isRetrievable) external;
}

contract RetrievabilityVerifier is Initializable, Upgradeable {
  // Discussion point about pdpVerifier. Is this simplication worth it?
  // Would it be better to track pdpVerifier address on a per-proofset basis
  // instead? That will make this contract much more generic and future-proof.
  function initialize(address _pdpVerifier) {
    pdpVerifier = _pdpVerifier;
  }
  
  // Emitted when a new proofset with retrievability SLA was created
  event ProofSetCreated(uint256 indexed setId);
  
  // Emitted when a proofset was deleted (after a deal ended?)
  event ProofSetDeleted(uint256 indexed setId);
  
  
  // Emitted when a new proving period (a new round of retrieval checks)
  // starts.
  event NextProvingPeriod(uint256 indexed setId, uint256 challengeEpoch);
  
  // HotStorageService is required to call this method
  // but only for proofsets subject to retrievability SLA.
  //
  // Emits ProofSetCreated event.
  function createProofSet(uint256 setId);
  
  // HotStorageService is required to call this method
  // but only for proofsets subject to retrievability SLA
  // 
  // Emits ProofSetDeleted event.
  function deleteProofSet(uint256 setId);

  // HotStorageService is required to call this method
  // but only for proofsets subject to retrievability SLA
  //
  // Emits NextProvingPeriod event.  
  function nextProvingPeriod(uint256 setId, uint256 challengeEpoch)
  
  // Discussion points: maybe we don't need these wrappers and we should 
  // let the checker node to interact with the PDPVerifier contract directly
  // These wrappers would become more important in case we wanted to 
  // support more than one PDPVerifier contract on a per-proof-set basis
  
  // How many files are stored in this proof set.
  // Caveat: when files (roots) are deleted, their indices are not recycled.
  // Checker nodes must check if re-run the sampling if they hit a deleted file
  // (deleted = not live, see below)
  function getRootCount(uint256 setId) {
    return PDPVerifier(pdpVerifier).getNextRootId();
  }
  
  // Returns false if the proof set is not live or if the root id is 1) not yet created 2) deleted
  function isRootLive(uint256 setId, uint256 rootId) public view returns (bool) {
    return PDPVerifier(pdpVerifier).rootLive(setId, rootId)
  }
  
  function getRootCid(uint256 setId, uint256 rootId) 
    public view returns (Cids.Cid memory)
  {
      require(isRootLive(setId, rootId), "Root is not live");  
      return PDPVerifier(pdpVerifier).getRootCid(setId, rootId);
  }
  
  function setIsRetrievable(
    address retrievabilityListener,
    uint256 setId,
    uint256 rootId,
    bool isRetrievable
  )
    public onlyChecker 
  {
    return RetrievabilityListener(retrievabilityListener)
      .setIsRetrievable(setId, rootId, isRetrievable);
  }
}

Checker algorithm

Checker algorithm for a random retrieval check for proof-set setId using the API above:

1. count = RetrievabilityVerifier.getRootCount(setId)

2. ix = random(0, C)

3. isLive = RetrievabilityVerifier.isRootLive(setId, ix)

4. if not isLive → goto 2

5. cid = RetrievabilityVerifier.getRootCid(setId, ix)

6. check retrievability of cid

Integration with HotStorageService

We expect the FilOz team to build this integration.

The proposal is based on SimplePDPServiceWithPayments as proposed in https://github.com/FilOzone/pdp/pull/146/

contract SimplePDPServiceWithPayments is
    PDPListener,
    IArbiter,
    Initializable,
    UUPSUpgradeable,
    OwnableUpgradeable
{
  // ...

    function initialize(
        address _pdpVerifierAddress,
        address _retrievabilityVerifierAddress, // NEW
        address _paymentsContractAddress,
        address _usdFcTokenAddress,
        uint256 _initialOperatorCommissionBps
    ) public initializer {
       // ...(existing code)...
      
       // TODO: make this required
       retrievabilityVerifierAddress = _retrievabilityVerifierAddress;
    }
    
    function proofSetCreated(
        uint256 proofSetId,
        address creator,
        bytes calldata extraData
    ) external {  
      // ...(existing code)...
      
      RetrievabilityVerifier(retrievabilityVerifierAddress).proofSetCreated(proofSetId)
    }    
    
    // TODO: Payment rail termination; not needed in MVP
    // TODO: call RetrievabilityVerifier.proofSetDeleted
    function proofSetDeleted(
        uint256 proofSetId,
        uint256 deletedLeafCount,
        bytes calldata
    ) external {} 
    
    function nextProvingPeriod(
        uint256 proofSetId,
        uint256 challengeEpoch,
        uint256 leafCount,
        bytes calldata
    ) external {
      // ...(existing code)...

      RetrievabilityVerifier(retrievabilityVerifierAddress).nextProvingPeriod(proofSetId, challengeEpoch);
    }  
 }