How we developed a scalable distributed communications platform
Design and develop a distributed communication system that works like a crypto currency network.
Providing services like ephemeral messaging, file storage, chat and video calling all in a distributed and secure manner without requiring a central server.
The Answer: Ghost.
Ghost is a distributed network and works much like torrents files on the internet. Whenever a user wants to send a file to someone on the internet. the file is first encrypted then broken down into many pieces and stored into multiple computers in the system. When you want to access your files. The system looks into the many computers your file is stored in gets the data then decrypt's it and you have your file. The same goes on if you want to do chat messages or video calling.
For this reason the system doesn't require a central server for data transfer and just like in the case torrent files pieces of single files are stored in multiple computers so that even if some of them switches off data is not lost.
Ghost is built on top of very advanced technologies like NodeJs AngularJS, realtime technologies and uses custom algorithms to work the way it does.
How does it work.
When a node comes online for the first time, it's private and public key pair is generated.
The user has to enter the paraphrase to generate the key pairs.
Based on his key-pair, some random sequence of bits and hashing, a unique id for the user is generated.
When all this done, the node then connects to a trusted node and sends the initialise request, when the trusted node receives the initialise requests it checks wether the node already exist in the user_index, if not then it adds the node's ID, Public Key and IP address to the user_index, and if the node already exists then it updates the IP address of the node and sends the user_index, to the node. user_index is a dictionary which consists of the ID, Public Key, and IPaddress of all the users connected to the platform.
Once an entry has been added in the user_index, then only the IP address can be updated, no other info can be updated, also the IP address update request must contain a signature, which is then verified by the public key of the user in the user_index.
If the signature is matched then the update request is processed, otherwise it gets rejected.
The discovery of nodes is done using the user_index, each connected node has a user_index, which consists of all the users and their address.
Each node connects to each other using their ID's, if you want to send data to any other node you will enter the ID of the node, the application will perform the lookup of that ID in the user_index, if it exists then it will send the data to the IP address and Port associated with the ID.
Messaging and Chat
To send a message, you will enter the ID of the user you want to send the message to.
The application will lookup that ID in the user_index, if that ID exist then it will try sending the message to the IP address and Port associated with that ID.
If a user wanted to send a message to another user, then first user will encrypt the message using the public_key of the user he/she wants to send, the public key data, IP Address and Port information is stored in the user_index, then it will try sending it to the using on the listed IP address and Port in the user_index.
If the transfer is successful then the message has been sent, if the receiving user is offline then the message will be sent to other nodes, this entry will be added in the message_log, which is basically a distributed log entries consisting of who sent the message to whom.
When the user comes online it will download the message_log, and check if it contains the any new messages, if there are some messages for the user in the message list then the user will decrypt the message using it's private key to view it.
This system also provides the ability for distributed file upload. The use will be able to use the file upload by allocating a quota which can be used by the rest of the system.
Use can store the data which is equal to 25% of the allocated quota.
For e.g: If you have allocated 10 GB of storage on your disk, you will get 2.5 GB of storage on the network.
You get only 25% of the allocated storage because each time you upload a file, it is replicated 4 times to increase the availability.
The file which is to be uploaded is divided into smaller chunks, and each chunk is encrypted and distributed in the network and replicated 4 times.
The information about each chunks location is stored in the upload_log.
When the user wants to download the file, all the chunks are downloaded, decrypted and assembled.
Video calling using WebRTC
We have added video calling capabilities to this using WebRTC.
In WebRTC you can do video calling in P2P manner, but a server is required for signalling, which like handshaking allows the two parties to communicate the connection details with each other, and in WebRTC you can use anything for signalling, so we have used Ghost for signalling, this allows exchanging the WebRTC connection data like the the SDP and ICE candidates in a distributed manner.
By using Ghost even this (SDP and ICE Candidates) data is transferred in a p2p manner, once the SDP and ICE candidates are exchanged the video call can initiate.
Notable Application: Sync
We already have file upload ability, we would use that to develop a Sync application which can work similar to dropbox like sync services, in which an application is installed on all your computers which watches certain folders and sync them with your devices/computers.
You could run sync in two modes, in the first mode it would directly sync the files between all your devices without uploading to the network, thus giving you unlimited data transfer.
In the second mode, it will upload all your files to the network and the other devices will download the files from the network, this will give you a limited storage (Depending upon your allocated quota), but it allow other devices to download the files when they will come online and all the devices are not required to be online at the same time to be in sync.
Other Distributed Applications
The node provides very simple API's to connect to different nodes and send raw data packets to different nodes in the network, hence a variety of applications are possible like multiplayer gaming, a distributed web interface, where websites can be hosted and served from a distributed network and many more applications.
Each node application is written in Node.JS, and the app provides REST API's. There are API's to do all the functions like get a user list, send a message, upload file, send raw data to a node etc.
Using these API's a web front-end has been created in AngularJS to provide a user interface. This AngularJS interface can be swapped with any other HTML or even Native UI.
Protocols for Communication
UDP protocol is used to for communication between the Nodes, any un-privilege random UDP port (1024 or larger) is opened on the system, and used for communication.
This IP and Port information is transmitted to other node for communication.
Encryption and Hashing
Each and every message is digitally signed, encrypted and it's hash is generated and then transmitted.
On the receiver side, the receiver validates the digital signatures, validates the hash and decrypts the message.
This IP and Port information is transmitted to other node for communication.
We believe that this is a very powerful technology and there are endless applications. It has a very open and unrestrictive API which provides powerful features and distributed communication in a secure manner.
We are very excited about this technology and also how people are going to use it, it's open API would allow developer to build very cool and amazing stuff which was never seen before
We are giving Ghost access for free. In the coming day's we are going to open source it. If you want to try your hand at it right now or if you want to say something to us, just drop us an email at [email protected]
we will be sure to get back to you in no time.
Ghost is being built and maintained by Mohammed Lakkadshaw & Co. A mobile and web design and development company.
Visit us at http://Mohammedlakkadshaw.com
See out Blog: http://blog.mohammedlakkadshaw.com