Decentralyze

I have an idea that I think is very important but I haven’t yet polished to the point where I’m comfortable sharing it. I’m going to share it anyway, unpolished, because I think it’s that useful.

So here I am, handing you a dull, gray stone, and I’m saying there’s a diamond inside. Maybe even a dilithium crystal. My hope is that a few experts will see what I see and help me safely extract it. Or maybe someone has already extracted it, and they can just show me.

The problem I’m trying to solve is at the core of decentralized (or loosely-coupled) systems. When you have an overall system (like the Web) composed of many subsystems which are managed on their own authority (websites), how can you add new features to the system without someone coordinating the changes?

RDF offers a solution to this, but it turns out to be pretty hard to put into practice. As I was thinking about how to make that easier, I realized my solution works independently of the rest of RDF. It can be applied to JSON, XML, or whatever. For now, I’m going to start with JSON.

Consider two on-the-web temperature sensors:

 > GET /temp HTTP/1.1
 > Host: paris.example.org
 > Accept: text/json
 >
 < HTTP/1.1 200 OK
 < Content-Type: text/json
 <
 {"temp":35.2}

 > GET /temp HTTP/1.1
 > Host: berkeley.example.org
 > Accept: text/json
 >
 < HTTP/1.1 200 OK
 < Content-Type: text/json
 <
 {"temp":35.2}

The careful human reader will immediately wonder whether these temperatures are in Celcius or Fahrenheit, or if maybe the first is in Celcius and the second Fahrenheit. This is a trivial example of a much deeper problem.

Here’s the first sketch of my solution:

 > GET /temp HTTP/1.1
 > Host: paris.example.org
 > Accept: text/json
 >
 < HTTP/1.1 200 OK
 < Content-Type: text/json
 <
 [
 {"GrowJSONVersion": 0.1,
 "defs": {
 "temp": "The temperature in degrees Fahrenheit as measured by a sensor and expressed as a JSON number"
 },
 {"temp":35.2}
 ]

 > GET /temp HTTP/1.1
 > Host: berkeley.example.org
 > Accept: text/json
 >
 < HTTP/1.1 200 OK
 < Content-Type: text/json
 <
 [
 {"GrowJSONVersion": 0.1,
 "defs": {
 "temp": "The temperature in degrees Fahrenheit as measured by a sensor and expressed as a JSON number"
 },
 {"temp":35.2}
 ]

I know it looks ugly, but now it’s clear that both readings are in Fahrenheit.

My proposal is that much like some data-consuming systems do schema validation now, GrowJSON data-consuming systems would actually look for that exact definition string.

This way, if a third sensor came on line:

 > GET /temp HTTP/1.1
 > Host: doha.example.org
 > Accept: text/json
 >
 < HTTP/1.1 200 OK
 < Content-Type: text/json
 <
 [
 {"GrowJSONVersion": 0.1,
 "defs": {
 "temp": "The temperature in degrees Celcius as measured by a sensor and expressed as a JSON number"
 },
 {"temp":35.2}
 ]

the software could automatically determine that it does not contain data in the format it was expecting. In this case, a human could easily read the definition and make the software handle both formats.

That’s the essence of the idea. Any place you might have ambiguity or a naming collision in your JSON, instead use natural language definitions that are detailed enough that (1) two people are very unlikely to chose the same text, and (2) if they did, they’re extremely likely to have meant the same thing, and while we’re at it (3) will help people implement code to handle it.

I see you shaking your head in disbelief, confusion, or possibly disgust. Let me try answering a few questions:

Question: Are you really suggesting every JSON document would include complete documentation of all the fields used in that JSON document?

Conceptually, yes, but in practice we’d want to have an “import” mechanism, allowing those definitions to be in another file or Web Resource. That might look something like:

 > GET /temp HTTP/1.1
 > Host: paris.example.org
 > Accept: text/json
 >
 < HTTP/1.1 200 OK
 < Content-Type: text/json
 <
 [
 {"GrowJSONVersion": 0.1}
 {"import": "http://example.org/schema",
 "requireSHA256": "7998bb7d2ff3cfa2666016ea0cd7a379b42eb5b0cebbb1142d8f086efaccfbc6",
 },
 {"temp":35.2}
 ]

 > GET /schema HTTP/1.1
 > Host: example.org
 > Accept: text/json
 >
 < HTTP/1.1 200 OK
 < Content-Type: text/json
 <
 [
 {"GrowJSONVersion": 0.1,
 "defs": {
 "temp": "The temperature in degrees Fahrenheit as measured by a sensor and expressed as a JSON number"
 }
 ]

Question: Would that break if you didn’t have a working Internet connection?

No, by including the SHA we make it clear the bytes aren’t allowed to change. So the data-consumer can actually hard-code the results of retrieval obtained at build time.

Question: Would the data-consumer have to copy the definition without changing one letter?

Yes, because the machines don’t know which letters might be important. In practice the person programming the data-consumer could do the same kind of import, referring to the same frozen schema on the Web, if they want to. Or they can just cut-and-paste the definitions they are using.

Question: Would the object keys still have to match?

No, only the definitions. If the Berkeley sensor used tmp instead of temp, the consumer would still be able to understand it just the same.

Question: Is that documentation string just plaintext?

I’m not sure yet. I wish markdown were properly standardized, but it’s not. The main kind of formatting I want in the definitions is links to other terms defined in the same document. Something like these [[term]] expressions:

{"GrowJSONVersion": 0.1,
 "defs": {
 "temp": "The temperature in degrees Fahrenheit as measured by a sensor at the current [[location]] and expressed as a JSON number"
 "location": "The place where the temperature reading [[temp]] was taken, expressed as a JSON array of two JSON numbers, being the longitude and latitude respectively, expressed as per GRS80 (as adopted by the IUGG in Canberra, December 1979)"
}

As I’ve been playing around with this, I keep finding good documentation strings include links to related object keys (properties), and I want to move the names of the keys outside the normal text, since they’re supposed to be able to change without changing the meaning.

Question: Can I fix the wording in some definition I wrote?

Yes, clearly that has to be supported. It would be done by keeping around the older text as an old version. As long as the meaning didn’t change, that’s okay.

Question: Does this have to be in English?

No. There can be multiple languages available, just like having old versions available. If any one of them matches, it counts as a match.