Overhide Broker API

Base URL: /broker, Version: 1.0.0, overhide documentation

This is the API contract expected to be exposed by an overhide "broker". This API is represented by the green "broker API" connector in the overhide component model:

overhide component model

Read this API with reference to the glossary.


A note on schemes used in the API.

Operations GET, PUT, POST, DELETE use the HTTPS scheme. These endpoints should be understood as standard REST requests. These configuration/discovery exchanges involve text parameters on the URL line and text/numeric payloads in the JSON representation. They use BASE64 encoding: the BASE64 encoding is explicitly stated.

Operations WIRE are WebSocket events that use a custom wire protocol. The SSL secured WebSockets connection is to the same address as the HTTPS connection. Authentication occurs using HTTPS and seamlessly propagates to the WSS connection.

Wire-protocol events require an additional step of a handshake between client and broker: beyond just being authenticated. Failure to handshake will result in "no handshake" 400-series error codes on wire-protocol events.

Please review the flow-control documentation to understand the WIRE protocol flow-control mechanisms.


A note on the wire-protocol payload notation.

WSS event exchanged payloads are comprised of compact byte streams. In the documentation payloads with byte streams are described as descriptors in the following format:

syntax bytes value
foo 2
bar 1 N
for i in N {
baz 4
}

The above indicates that the first 3 bytes are followed by a loop of 4 byte words. The number of iteration is specified by bits 16-23.

All numeric values greater than a byte are sent in network order, big-endian format. The foo above is 2-bytes wide, the first byte having the most significant bits.


Schemes: https, wss

Summary

Tag: capabilities

Discover capabilities of the overhide broker system.

Operation Description
GET /capabilities/remuneration-providers

List of remuneration providers supported.

GET /capabilities/{remuneration-key}/tiers

List of subscription tiers available.

GET /capabilities/address

Retrieve this broker's broker-address for the remuneration-provider used by caller's identity.

GET /capabilities/version

Retrieve this broker's API version.

Tag: flow-control

Flow controls include toggles for the client and the broker to agree on message maximums, message fragmentation parameters, as well as acknowledgment and continuation messages for fragments of large payloads.

The various flow-control events are modeled below:

overhide flow-control infographic

Before the client can exchange data with the overhide broker it needs to authenticate and handshake.

The first synchronous HTTPS call is a PUT to /auth/credentials; returning an identity. Subsequently, when the client establishes a Websocket connection with the broker (Websocket connect(identity)) it provides the identity as per WSS Auth.

From this point on the exchanges take place over the Websocket protocol, ursng WIRE events as documented here.

Before any data exchanges can take place, the client communicates its desired fragment and aggregate message parameters using the handshake event. The server agrees--"shakes on it" with a "response" WIRE event having a code of 200/OK.

The client proceeds to send a large datastore-value that is necessairly fragmented into multiple WIRE events. The exchange is started with a set data event. The flags attribute of this set data event does not have the 0x80 bit set: indicating that the payload byte-stream is the first fragment of multiple fragments for the large aggregate message. The broker replies with a "response" WIRE event having a code of 200/OK. The client does not send the subsequent fragment until the broker confirms receipt of the previous fragment with such a "response" WIRE event.

The client sends subsequent fragments using the continue events. Before sending each subsequent fragment the client expects a "response" WIRE event having a code of 200/OK.

The client sends the final fragment with a flags having the 0x80 bit toggled: indicating last fragment sent.

The final exchange modeled is the client retrieving a large payload from the broker. The exchange is initiated with a get data event. The broker responds with a WIRE event having a code of 206/fragment. The client accepts the payload byte-stream as the first fragment of a larger response. The client sends an acknowledge event to the broker; acknowledging receipt of the fragment and freeing the broker to send the subsequent fragment.

The broker does so, looping through the 206-acknowledge exchanges.

When the broker sends the last fragment of the aggregate message byte-stream, it sends it with a code of 200/OK. The client doesn't have to acknowledge this last response as it's not a 206/fragment.

Operation Description
WIRE handshake

Client initiates handshake with broker.

WIRE continue

Continue sending fragments of a message to a "setter" endpoint of a broker.

WIRE acknowledge

Acknowledge receipt of a response fragment from a "getter" endpoint of a broker.

WIRE halt

Halt fragmented sending or receiving.

WIRE watchdog

Check if connection is live.

Tag: data-stewardship

Stewardship of a users's data.

Stewardship of data is important regardless whether an identity opts-in into decentralized storage or not. All but one of these APIs deal with stwardship regardless of decentralization opt-in selected: see opt-in endpoint and segment-key specific opt-in.

These APIs allow a client to:

Envisioned use cases include backup and broker transfer including data migration.

The stewardship APIs enable data decentralization and broker trustlessness.

Each use case below is accompanied by a sequence diagram with the following markup:

Sample Use Case: Designation of Stewards

Consider the following sequence modelling overhide API calls required to associate user data with two overhide brokers; one designated as an active data steward and one designated to be a passive data steward.

overhide designation of stewards infographic

In this sequence the client subscribes to broker 1 and authenticates [1]. It's important to note that the first time a new user authenticates to a broker a new set of GUIDs is generated pointing to the user's data on the broker's distributed persistence network (e.g. IPFS "names" PKI pair for IPFS).

At this point broker 1 is read-only. Any attempts to write data or post messages would yield a 400 -- "cannot write, broker not an active steward of user's data". To make the broker writable, it's designated as an active data steward [3].

When participating in broker-lookup the PUT /active-stewardship call--to make broker 1 an active data steward--requires we know the broker's host name (or IP). The PUT /active-stewardship call is the mechanism that registers the broker as a user's active broker; hence the necessary signing of the particular metadata to push into the broker-lookup network. Step [2] retrieves the broker's hostname for signing and pushing into the method [3].

The client writes then reads data as desired [5]. The data is eventually made consistent with the distributed persistence network (e.g. IPFS) [5.1].

Presumably the distributed persistence network backing broker 1 is a public decentralized data-store that allows common tooling access [7]. To access the client's data we need its reference. The reference is retrieved as a GUID [6].

Another overhide broker is subscribed to by the client--to guarantee pinning of the user's data as a backup [8].

At this point broker 2 is subscribed to, but it is not associated with the user's data as referenced by broker 1 [9-10]. The second broker has a compeltelly different GUID for the user's data on the distributed persistence network.

Once the client furnished the GUID form broker 1 to broker 2 [11], broker 2 starts acting as a passive steward of the user's data [12-14].

Sample Use Case: Disassociation from Steward Over Persistence Network With Pour-Active Steward

The user may have a reason to diassociate from an overhide broker. In the following sequence we model the API calls required to disassociate user data from broker 1: including ensuring the disassociated broker cannot clobber any new data and that old user data is not consuming resources on the broker.

This use case leverages broker-lookup and a distributed persistence network (e.g. IPFS) for the migration.

overhide disassociation from steward using persistence network infographic

We start with broker 1 being the active data steward and broker 2 being the passive data steward. Use of broker-lookup with GET /broker-lookup [1] indicates broker 1 as the activeBrokerHost to the client. The client works with broker 1 [2] and broker 2 is eventually consistent via the distributed persistence network [2.1,2.1.1].

Before continuing we verify that broker 2 (passive) is synching to the data from broker 1 [3,4]. If both brokers return the same GUID we know they're working with the same data.

We prevent any further writes to broker 1 by making it passive [5].

We put broker 2 into a special "pour-active" mode [6]. A "pour-active" data-steward is an "active" data-steward with special caveats to allow for consistency during data migration. Enabling "pour-active" mode on a broker is a temporary measure before transitioning to "active" mode.

Putting broker 2 into "pour-active" mode prompts it to tell the broker-lookup network that broker 1 is now the drainBrokerHost and that broker 2 is the new activeBrokerHost [6.1].

The client discovers that state of the hosts [7] and continues to use the brokers in tandem, uninterrupted [8,9]. The reads on both brokers [8,9] respect the decision matrix as per result code 246. Reads against broker 2 could return result code 422 when a segment-key is not yet available. Writes on broker 1 are disallowed with result code 400 -- "cannot write, broker not an active steward of user's data". Writes to broker 2 [8] proceed as usual except are not synched with the distributed persistence network (while in "pour-active" mode).

While the user is accessing the brokers in tandem, broker 2 is busy finalizing it's eventual consistency with the distirbuted persistence network. Data access and migration are occuring at the same time: ovrehide broker 2 accepts distributed persistence network data changes for all segment-keys not already overwritten via an explicit API write.

[11] and [12] check to see if syching data over the distributed persistence network is finished. Once done, we can make broker 2 a standard active steward [13]. The change to the broker's stewardship mode is reflected in the broker-lookup [13.1].

At this point broker 1 is a passive steward for the user's data. But since we set out to migrate to broker 2 and disassociate the user's data from broker 1; now that the migration is done we tell broker 1 to delete the data and stop synching [14].

The user continues to use the client to access their data [15-16.1].

Sample Use Case: Client Driven Export+Fill With Pour-Active Steward

In the previous use case we model data migration over a distributed persistence network. In that case we require all data to be migrated to be opted-in into distributed persistence.

In this use case we accomplish the same migration over-the-top without use of a distributed persistence network. This works for all your data; opted-in and not.

overhide disassociation from steward using export+fill infographic

The use case starts with broker 1 as the active broker and broker 2 is not in the picture. The client works with broker 1 [1,2].

When the user decides to switch brokers of their data, they use the client to subscribe to broker 2 [3].

Switching stewardship modes is done in quick succession to avoid interruption [4-5.1]. We now have broker 2 in pour-active mode and broker 1 as a passive steward.

Putting broker 2 into "pour-active" mode prompts it to tell the broker-lookup network that broker 1 is now the drainBrokerHost and that broker 2 is the new activeBrokerHost [5.1].

The client discovers that state of the hosts [6] and continues to use the brokers in tandem, uninterrupted [7,8]. The reads on both brokers [7,8] respect the decision matrix as per result code 246. Reads against broker 2 could return result code 422 when a segment-key is not yet available. Writes on broker 1 are disallowed with result code 400 -- "cannot write, broker not an active steward of user's data". Writes to broker 2 [7] proceed as usual.

While the user is accessing the brokers in tandem, the client is used to export data out of broker 1 [9] and fill the data into broker 2 [10]. The fill operation is an over-the-top alternative to indiscriminate importing: the fill operation does not overwrite exitsting segment-keys on broker 2. This, in combination with the result codes describe in the previous paragraph, allows for over-the-top migration of user's data while the user continues to access their data via the client.

At the end the user deletes their data from [12] and continues to access data with broker 2 [13,14].

Sample Use Case: Client Driven Export+Import

The previous two use cases described data migration with minimal interruption to user's access to their data.

For completeness, this use case describes the most basic over-the-top data migration that does interrupt the user's access to their data. This use case is likely practical when a single user is migrating their data and they're fine waiting for their data to export and import before continuing to use it. Contrast this to the previous use cases--which are likely more practical when the data migrating user is a service provider--whose data is being accessed by many other users (e.g. delegtes) while the migration is in progress.