SPARK Decentralization Analysis
Overview
In preparation for LabWeek2023, we are prioritizing SPARK features that allow us to 1) hit roadmap milestones, 2) allow for PMF testing of other interoperable systems (i.e. Station & Meridian), and 3) have minimal wasted work. While a fully decentralized SPARK Module remains our desired outcome, we’re examining paths forward within constraints (1), (2), & (3). In short, we need to decide how decentralized each of the functions within SPARK should be by LabWeek.
Given timelines and team bandwidth we will only be able to deliver a subset of the desired features for Lab Week. We should focus our efforts on critical features for testing PMF, namely the fraud detection functionality composed of:
- Retrieval with Attestation: Retrieve CID content from SP and obtain the attestation token
- Proof of data possession: Create proof that the entire CID content was retrieved
Unfortunately a focus on these critical features implies less time to deliver a fully decentralized system
Proposed pathways
1) [Recommended] Centralized membership service, tasking service and CID sampling, but robust Retrieval with Attestation and Proof of data possession
- Pros: Test SPARK’s most novel and critical features directly to assess whether there is user demand for the product
- Cons: We don’t fully explore the implementation of decentralized modules from the start, which will introduce some debt / re-writes down the road
- Rationale: If we don’t prove demand for the SPARK service at a minimum, it will be redundant if it is decentralized or not. This comes at the cost of incurring some iteration on both Meridian and Spark modules
2) Decentralized membership service, Tasking service, and CID sampling but minimal implementation of Retrieval with Attestation and Proof of data possession
- Pros: We build for the end state of decentralization, and worth through the difficult technical challenges that come with it up front
- Cons: Will take much longer to deliver (order of magnitude: months to quarters). Less robust user-facing features likely results in poorer user experience and gives us noisy signal on PMF / ability to control fraud
Spark Functions (detailed):
The following functions (at minimum) exist within the SPARK module and must eventually meet the requirement of decentralization:
| Membership Service | Maintain a list of individual checkers (and their IP addresses) that are eligible for inclusion in an epoch’s committees, ideally in a way that the community can trust and that is private | Centralized |
|---|---|---|
| Tasking Service | Cluster the online SPARK checkers into several committees | Centralized |
| CID Sampling | Choose a random (CID, SP) pair for each committee. | Centralized |
| Retrieval with Attestation | Retrieve CID content from SP and obtain the attestation token | Robust |
| Proof of Data Possession | Create proof that the entire CID content was retrieved | Robust |
| Measurement Service | Report job outcome to Meridian contracts | Decentralized |
| Evaluation & Verification | Evaluate the impact of checkers (and detect fraud) | Decentralized |
| Fraud Detection | The SPARK module has both fraud resistant properties (e.g. committees of checkers structured as an honest majority) and fraud detection services (e.g. discarding jobs that fail to pass retrieval attestation or inclusion proofs) | WIP |
Decentralization Notes:
Measurement Service
Target LabWeek Status: Decentralized
Notes:
- This is solved in Meridian already
- Multiple parties may host measurement services
- Peer can submit measurements without using a measurement service
Evaluation & Verification
Target LabWeek Status: Decentralized
Notes:
- This is solved in Meridian already
- Trusted parties can reproduce the evaluation outcome and thereby verify correctness of the evaluation service run by PL
—————