Home

Symbiotic-Data is an initiative intending to standardize primitive data types and their serialization formats, on different platforms, programming languages, and target encodings.

Motivation

Different platforms have different paradigms, capabilities, preconceptions, and opinions on how data should be formatted and stored internally, and this affects how the language or platform naturally tries to encode that data. This project aims to clarify some of those differences, and formalize data serialization standards. It is trying to act as a Rosetta Stone for various languages.

Most implementations of Symbiotic Data should be as painless and mostly cost-free — ideally being a no-op. But for some instances, it is necessary to restrict and refine a platform’s natural understanding of a data type, in order to meet the criteria that constitutes its definition in Symbiotic-Data.

Similarly, serialization and de-serialization for Symbiotic-Data should be as simple, intuitive, and fast as possible, and rigorously defined for every data type.


Supported Types

All supported types can be found in the documentation under Data-Types. The following classes of data are defined:

  • Primitives: Elementary data types supported by most programming languages — Integers, Floating-point numbers, Boolean values, and Strings.

  • Casual Data: Every-day useful data types for various tasks — Chronological data (like time, dates, etc.), and Network-relevant data (URIs, email addresses, etc.) are included under this classification.

  • Primitive Composites: Elementary container data types — Fixed-size Arrays, Varying-size Vectors, Tuples, and Tagged Unions.

    We understand not every language supports parametric polymorphism and cannot implement “container types”. However, the behavior is defined such that the user’s specialization is still qualified under the same umbrella.

  • Sophisticated Composites: Specialty container data types — Mappings, Recursive Mappings (Tries), and other complex containers may be placed in here.

    Sets (unique vectors) are not included due to this being a serialization definition library, not a data type library in generally. Similarly, qualified mappings are also not included (i.e. a HashMap), because we are only concerned with the different methods of serializing data, not storing it.

Supported Target Encodings

Currently, only two target encodings are defined for Symbiotic-Data:

  • JSON (JavaScript Object Notation), because of its wide utility and usage throughout the Internet — it is also a very likely suspect for incorrect encodings between languages.
  • Raw binary stream — binary serialization is the foundation for all data storage and communication, and is therefore the foundation for this project as well. All types have strict definitions for how they’re stored in a byte-wise (8-bit) string.

Test Suite

The backbone of the insurance policy that guarantees these types are indeed correctly serialized and interpreted is due to the Symbiote Test Suite: a protocol for randomly generating data, and communicating that data over-the-wire to another platform.

Every implementation of Symbiotic-Data should also implement this test suite and protocol, to verify its correct behavior in accordance with the standard. Every data type in the standard declares what operations it supports.

Properties

The types defined in Symbiotic-Data may satisfy the definitions of certain algebraic objects, and by extension should have certain properties of behavior associated with their operations. You can see what properties and objects we use in the test suites in the Algebraic Properties page. Every data type in the standard declares what instances it supports, and by extension, how they are used in the test suite as testable operations.

Todo

Define the security and client identifier enrichment mechanisms for each serialization target;

ZeroMQ is used for binary data, over CurveMQ, with the server public key available in a TXT DNS record.

WebSocket is used for JSON, and secured through a TLS connections. However, the testing protocol is enriched with a UUID tag for identifying the client connected, similar to Router-Dealer sockets in ZeroMQ.

Todo

List the reference implementation server links for the serialization targets: wss://ws.symbiotic-data.io and tcp://zmq.symbiotic-data.io

Supported Platforms

Currently the supported platforms are:

There are planned implementations for:

  • JavaScript (by use of the PureScript implementation)
  • Rust
  • Java / Clojure / Kotlin
  • Objective-C / Swift
  • C# / F#
  • C / C++
  • PHP
  • Ruby
  • Python
  • Go
  • D

If you have any other requested implementations or contributions, feel free to open an issue or pull request!