MaidSafe Developer Conference 2017

If someone would have told us that our first developer conference would be held 7,500 miles away from Troon we would most certainly not have believed them. However, on the 20th of February, in conjunction with our partners at MaidSafe Asia, we found ourselves Fairmont Hotel in Jakarta hosting MaidSafe Devcon 1. With a population of 300 million Indonesia is one of Asia’s most populated countries. With a thriving developer scene and a highly motivated populace, Jakarta, the country’s capital, proved to be an excellent location.

Short note about MaidSafe Asia

We first announced the intention to set up a joint venture in 2016 and we are now delighted to be able to confirm that the agreement has been signed off and the entity MaidSafe Asia has been incorporated in Singapore. As a quick refresher, the intention in setting up this partnership is to enable MaidSafe UK to focus on developing the network, scale the development team and support developers on the platform. MaidSafe Asia’s priorities are to raise awareness, reach out to developers and generally market the technology. In fact, the name MaidSafe Asia maybe somewhat misleading as we start to discuss extending the area in which the new entity operates. This is a fluid situation and we will keep everyone updated as things progress. Anyway, back to the conference.

Different attitudes

The conferences 250 attendees came from all across Indonesia with some groups even travelling from Singapore and beyond. Many of the developers were freelancers, but also included those working for companies, and some very eager students.

During the morning session, Nick introduced the high level concepts of the SAFE Network and issues with the existing Internet, and the problems caused by it’s current centralised architecture. David spoke later during the morning session, describing MaidSafe’s vision, and the benefits the platform will offer to developers.

David explains the vision

David explains the vision

Nick describes centralised threats

Nick describes centralised threats

 

What struck us speaking with local people both during and after the event was the different attitudes that both users and developers have toward data security. Both of these issues are becoming of ever increasing importance in the UK and Europe, but these concerns are not shared to the same extent in Indonesia. Developers in Indonesia are more excited by the prospect of being able to compete with large technology companies using SAFEs costless infrastructure, and the concept of Safecoin was something that also seemed to resonate, with many liking the built in revenue streams that it provides. Monetisation it seems is a more significant factor.

CoinPayments

After presenting at the main conference, David and Nick went to speak with members of the press and were joined by Coin Payments CEO Alex Alexandrov. Coin Payments, a partner of MaidSafe Asia, are the largest alt coin payment processor in the world, processing in excess of $50million of transactions per month and have 132,000 vendors across 182 different countries. Processing over 55 alt coins, including of course MaidSafeCoin, the company have been great supporters of the the SAFE Network and are great advocates of our technology. They have also created their own MaidSafeCoin wallet and offer secure coin storage via their vault. You can find out more on their website.

Getting down to business

By the time that Krishna’s developer workshop started in the afternoon session, the polite and shy audience had lost any inhibitions, and became animated and engaged as Krishna explained the networks data types, the core concepts behind the APIs, before going on to showcase the developer tutorials that have been created. It was evident from the questions that followed Krishna’s first afternoon session that the audience had taken much of the information on board.

Developers put Krishna through his paces

Developers put Krishna through his paces

Krishna explains the APIs

Krishna explains the APIs

 

Krishna’s second session focussed upon the current transition from our REST API paradigm, which while being language agnostic does not cater well for mobile devices. REST demands that the devices hold state which is problematic given the fact that mobile devices automatically disconnect from networks after very short periods. Explaining our current transition to an SDK, he gave an overview of the plans for the next few months, specifically the transitioning of the existing example applications and producing new developer documentation.

After the closing remarks the event finished with much hand shaking and more questions from the attendees whom it cannot be emphasised enough where some of the most friendly conference attendees we have ever had the pleasure to meet. It was also great to see so many female coders, who, while still outnumbered by their male counterparts, were as well represented as we have seen at any recent conference we have attended.

The following day, David and Nick gave filmed interviews with CNN Indonesia. The interviewer politely confirmed our suspicions that Indonesians are not as concerned as we are in Europe regards security and privacy of data, but are very much interested in the sharing economy and the desire to contribute to a crowd sourced Internet. Maybe in time attitudes will change, although maybe they won’t have to as the SAFE Network continues to roll out and starts to deliver the security and privacy many Britons and Europeans value so highly.

Developer Case Study – Dsensor

Decentralized Mapping Protocol Project – Dsensor

Continuing our series of case studies highlighting early stage application development on the SAFE (Secure Access For Everyone) Network, Dsensor is being developed by James Littlejohn. James explored various platforms to store and protect the data he would be collecting and decided to use the SAFE Network, because it reflected his belief that the network should not be driven by economics, but be focused first and foremost on the data.

MaidSafe’s fully secure, decentralised approach supported James’ view that knowledge or data should be in the complete control of user. While it is early days, Dsensor’s use of the SAFE Network APIs in its proof of concept form shows its potential as a viable platform for the management of data. James was also attracted to the SAFE Network, because of its strong encryption, and its ability to break data into chunks before scattering it around the decentralised network of nodes. This ensures the highest possible security and privacy for users when combined with the decentralised architecture, which avoids offering hackers central points of attack on a network, as we experience in today’s centralised, server-based model.

Being open source and supported by a strong community in the SAFE Network forum also means James has ready access to experts and potential partners, who can help to build out the application and trouble-shoot any technical questions. In the future James may also explore using safecoin to incentivise participation on Dsensor.

The Problem with Science

James Littlejohn has been involved in entrepreneurial projects since the dot com boom and while investigating opportunities around text mining identified an opportunity for lifestyle linking analytics, particularly in the area of wearable tech. In the course of his evaluation he recognised a broader application to data mining and analysis in the field of scientific and academic research. James assessed a number of drivers, including emerging technologies and changing economic conditions, which were beginning to have an effect on the way research was conducted.

Firstly, walled garden applications such as Facebook and wearable technologies were becoming more prevalent, and while they were a rich source of data on human activity, access to that information was restricted. At a time when the internet is supposed to be democratising many aspects of work and social life this is endangering an important source of information on lifestyle and health patterns, which could benefit societies around the world.

Secondly, the sustained economic impact of the financial crisis was creating significant pressure on public funding for research at a time when it was needed more than ever. Technology and the availability of large amounts of data is leading to opportunities for breakthroughs in a wide variety of academic and research fields. If the funding is not available via traditional public sources then there is an urgent to find new forms of investment. The rise of alternative cryptocurrencies could potentially address this point, offering a new, fairer way to incentivise and reward individuals for participating in research projects. For example, James envisages a scenario where the grant funder might ‘tokenise’ a percentage of their funding money and issue it via a science blockchain (like Dsensor). This would help to ensure the funding could be traced directly ensuring good governance of scientific research projects and fairer access to resources.

The final driver for a new model reflects an on-going debate about the model of peer-reviewed scientific research. For a number of years there has been a recognition of some fundamental weaknesses in the model in areas such as the replicability of research. In a poll conducted by Nature in May 2016 more than 70% of researchers admitted they had tried and failed to reproduce the experiments of other scientists and more than 50% failed to reproduce their own experiments. Of course this is in part due to the nature of frontier scientific research, which is reliant on trial and error, but there are clearly inefficiencies in the process.

Furthermore, there are questions about efficiency of current research models – in 2009 Chalmers and Glaziou identified some key sources of avoidable waste in biomedical research. They estimated that the cumulative effect was that about 85% of research investment – equating to about $200 billion of the investment in 2010 – is wasted. A blockchain provides a potential solution to this reproducibility crisis as Dr. Sönke Bartling and Benedikt Fecher outline in their paper, “Blockchain for science and knowledge creation.” Although scientific research should be delivered at arm’s length from the individual contributors it is ultimately reliant on individual scientists to gather and interpret data without bias. It is also often reliant on finite data sets, controlled samples or clinical trials, thus limiting the ability to cross reference the findings against other data sources.

Given the availability of data via the internet and the rise of automation technologies, such as machine learning, James believes that if individuals have control of their information they can decide to contribute their information to research projects without the interference of third parties such as academics or technology providers. Using automation scientists, academics – and more importantly citizen scientists – can draw data from anywhere in the world beyond the confines of a specific controlled sample and review independently to provide a data driven outcome.

Building A Blockchain for Science Research – A Truth Engine for Mankind

James’ investigation of text mining approaches led him to peer to peer models, which were enabling the owners of data to take control of how and with whom their information was shared.  

It led to the development of Dsensor.org (Decentralized Mapping Protocol), a peer to peer network for science knowledge to be investigated, discovered and shared. It has been based on the principle of science “SenseMaking” and it is designed to evolve peer review to a computational consensus model.  Using Dsensor if a scientist creates a thesis and wants to test it the scientist enters the hypothesis in computational form (called a Dmap in Dsensor speak) . The Mapping protocol then automates the testing of the science, starting by trawling the Dsensor network for relevant data from other peers. That data is then sampled and ‘scored’ based on its prediction power to verify or challenge the thesis until a computation consensus is established.  Science attaining this status then becomes ‘computationally active’ in the network meaning any peer has the ability to tap into the collective knowledge and feed in their own unique sensor data get the insights from the science working for them.

James has the ambitious goal to become a “truth engine for mankind” ensuring science is based on openness, transparency and reproducible results, essentially checking the accuracy of peer review.  Dsensor intends to deliver this outcome by building a network of trustless peers, which has inherit vast complexity making it economically and technically very costly and difficult to corrupt.  Dsensor, currently at proof of concept stage utilises  the Ethereum blockchain, using its cryptography and a variant of the proof of work process to create a technological and mathematical state where even with colluding computers it is impossible to game the system.   Rather than creating complexity using a proof of work Dsensor creates uncertainty using random sampling, in particular the selection of peers from the network and data sampling techniques.  The data sampling techniques are selected by each autonomous peer and the network sampling is a property of the Protocol for selecting peers from a Distributed Hash Table. In theory once the network gains a certain size the economic cost of gaming the network with false sensor data to support a fraudulent scientific computation will become extremely costly.

Additional safeguards include the role of reproducibility in science.  This creates an immutable audit trail or “mapping accounting” entries that support the most truthful science available to the network.  These networks are called GaiaBlocks and are open to be challenged by any peer on the network.  Scoring of the science also provides a rating model for scientific research and individual peers on the network.  Peers with poor outcomes will be demoted in favour of more highly rated scientific computations.

 

Glasgow University Students To Use The SAFE Network

Since its inception in 1451, Glasgow University has built a worldwide reputation as a centre for innovations which have had a profound effect on the world. Its famous alumni have included John Logie Baird, Lord Kelvin and Adam Smith, whose global impact has left a lasting impression on the world we live in today. Continuing the trend, Glasgow University Computing Science Students will be exposed to the latest in decentralised networking technology as MaidSafe’s SAFE (Secure Access For Everyone) Network puts them at the forefront of research and development into next generation Internet applications.

Students within the computer science department, under the guidance of Dr Inah Omoronyia, Lecturer in Software Engineering and Information Security, will work with the MaidSafe team, led by Scottish Engineer David Irvine. They will provide students guidance in building apps on top of the company’s platform, a secure and decentralised data and communications network that replaces the world’s data centres and servers with the spare computing capacity of the networks users. This comes at a time of great debate about the future of the Internet with leading academics, including the founder of the worldwide web Sir Tim Berners-Lee, seeking to improve the security and privacy offered to users.

The SAFE Network provides a zero cost infrastructure for students and the current APIs enable the creation of storage and email applications. This functionality is laid out to developers in tutorials created by the company, and this will be expanded over the next few months as MaidSafe release tutorials every 2 weeks, providing increasingly more complex functionality to application developers.

MaidSafe CEO, David Irvine, commented on the partnership; “We are delighted to be working with a university with such a rich heritage and we very much look forward to using the applications created by their students. Where better to push the envelope of evolutionary thinking than the country that Voltaire opined “We look to Scotland for all our ideas of civilisation”. Glasgow University has an excellent opportunity to be at the forefront of research and development in the field of Internet technologies, alongside the likes of MIT, which will further enhance its reputation – and that of Scotland – as a source of cutting edge innovation.”

Glasgow University has one of the leading computing science departments in the UK and is ranked amongst the top 100 in the world. Lecturer in Software Engineering and Information Security, Dr Inah Omoronyia confirmed “It’s a great opportunity for our students at Glasgow University to get hands-on experience with building apps for the SAFE network. Security and privacy functions is now core to modern day software systems; our students are really excited to learn new ways of building such systems using cutting-edge technology.”

MaidSafe Announces Equity Funding Round on BnktotheFuture

We wanted to share some next steps for MaidSafe and the future of the SAFE Network. On September 12th we will be launching an equity fund-raising round with BnkToTheFuture, a leading global online investment platform for qualifying investors. MaidSafe is looking to raise between £1.75m and £2m during the 30 day campaign.

Next steps

This is an important step for the company and the future of the platform coming on the back of the recent release of the SAFE alpha. We are delighted with the positive feedback we have had to date and the strong support of our community has been hugely important to us. We see this funding round as critical as we move to the next stage of development so we wanted to explain why we think this is the best way forward.

As you know launching the alpha is only one of many steps on our journey. There is a lot of work to do to build out functionality and create a robust platform that will attract users and application developers. That means we need to continue to recruit the best developer talent from around the world and build out a developer programme to support the growing interest in building on the SAFE Network. We will also be looking to develop our own applications, because we do see commercial potential for us as well as for the broader community.

Why BnkToTheFuture?

We have chosen BnktotheFuture for a number of reasons. They have a large online community of qualified investors, interested in the decentralised technology sector. There is a strong relationship with the crypto-communities, which is important to us, and investors will be able to invest in both traditional currency, bitcoin and even some of the more liquid altcoins like Ether. Above all it also means we will remain in control of our strategy and roadmap, which is really important to us as we want the SAFE Network to be available for everyone to use and develop on. This is why the core code-base is open-sourced, to encourage the establishment of a strong community around the technology. Furthermore, we have transferred the core intellectual property (IP) to a Scottish charity (The MaidSafe Foundation) to help ensure that MaidSafe is not seen to benefit unfairly from ownership of the network (although we hold a license in perpetuity to allow us to utilise and license the technology to partner companies).

We appreciate all your support so far and hope you will see this news as further evidence that the SAFE Network is on the fast track to becoming a viable alternative to today’s insecure, unreliable worldwide web. With this investment round we are seeking to accelerate progress further by adding functionality and working towards the beta version and beyond. I hope you will continue to contribute to this exciting journey, because we believe a secure, decentralised approach to the internet will present huge opportunities for users and developers alike.

SAFE Network Alpha Release

After 5 months of testing and 8 test nets it is our pleasure to announce the immediate availability of the alpha release of the SAFE Network. This represents another significant milestone on the way to creating a new, decentralised Internet.

Launcher and demo app

The release is available Windows, Mac and Linux, and comes with 2 components, the Launcher and the Demo App, each with their own installer. You can download both from the alpha page of our website.

The Launcher enables users to create their own account and access the network without providing their credentials (comprised of an account secret and account password) or giving third party applications access to them. This secure gateway enables users to stay in control of their details.

The Demo App must be run alongside Launcher and enables users to create their own SAFE webpage and public ID, store and retrieve private data and share public content. Users can also upload and host existing websites on the SAFE Network without charge.

This alpha release is focused on use of the client software. In a couple of weeks we will provide Vault installers to enable users to contribute their own computing resources. Running 2 parallel networks will enable us to provide a more stable network for end users and app developers, while also enabling ongoing development of the vault, which has a much more active code base at present. Once we are happy with performance, we will revert back to having one network.

As with any alpha software caveats apply to this release and users should be aware that data on the network will be wiped from time to time and there is a possibility that your data will be lost.

Not just for end users

The demo app is the first of several apps currently under development and with an updated release of SAFE Launcher APIs we anticipate that this list will increase in the coming weeks and months. We encourage any developers thinking about creating an app on the network to get in touch and we will help in anyway we can. In this vein a new developer focused SAFE Network forum will be launched next week, more details to follow…

Onward and upward

This is the first of several alpha releases that MaidSafe will make. Future versions will improve performance and increase stability, combine the Client and Vault networks, as well as adding features such as; contact management, messaging, test safecoin and safecoin wallets. More information on our rollout strategy is available here.

This is an incredibly exciting time for MaidSafe and the SAFE Network community. There is a growing movement to return to the original principles of the open web and decentralise the Internet. We believe our technology can contribute to this initiative and that is why we standby our vision to open source the technology so that all users and developers can benefit from it. We believe this is essential to fulfil our commitment to develop a decentralised network that gives users back control and offers far greater protections than today’s Worldwide Web.

Of course, we could not have got this far without the support of all our shareholders and community members. Thanks for all your help, we hope you enjoy the alpha and we look forward to hearing all your feedback! With your input we will iterate the network as quickly as possible to improve it and bring new innovations.

Community Engagement Program

We are happy to announce the Community Engagement Program (CEP). The purpose of this process is to match developers with immediate and fundamental SAFE Network requirements. This will take the form of a community funded ‘Kickstarter’ like process to encourage teams to bid for projects, accessing part of the 5 million available MAID fund for bounties, community projects…etc…put forward within the MaidSafeCoin announcement.

Each project will be identified up front as being in demand and then teams (or individuals) will propose a costed solution (in MaidSafeCoin) for each small proposal. It is anticipated that these will be mostly small (6 weeks or less) projects and will primarily cover end user applications, but may also include core components.

This proposed process is essentially an experiment to see if this way of working is both effective and scalable moving forward, only by running/cycling through this way of working will we truly understand its feasibility. The CEP will not replace the bounty program, which is more effective for smaller and specific pieces of work.

Applicants will be responding to a set of requirements established initially by MaidSafe and adapted to take into account community feedback. The community themselves will of course complement and expand on this process as it becomes more established, incorporating community based proposals. It is the intention that CEP projects will be 50% funded (in MaidSafeCoin) by MaidSafe and 50% by the community. If projects achieve less than 100% of the projected costs, it is up to them if they proceed. MaidSafe will assign a member of the company, whose skills and experience best suit the project, to oversee and act as the main point of contact. The following forum post explains the process in more detail.

The first project MaidSafe are seeking proposals for is a SAFE Network browser that is able to render standard web content and also enable web applications to invoke the APIs exposed by the Launcher. A mechanism to provide SAFE-only URLs as well as html…etc… is expected.  We would love to have your proposals, please submit them to the proposals category within the forum.

Good luck!

Introduction & Technical Overview of SAFE Consensus

The features included in decentralized networks can be quite varied based on the proposed goals of the technology. From the sharing capabilities offered by Bittorrent to user privacy enabled via Tor’s routing protocol, network designs directly reflect the mission set forth by their architects. Within autonomous networks that rely on data and system integrity where network critical actions may fail or produce faulty outputs, consensus mechanisms are an important feature which optimise reliability. Just like in greater society where important business or policy decisions are typically deferred to a board or committee rather than depending on a single individual, computer systems which manage data and user accounts in a diverse environment face quite a lot of potential for parts of that system to be inaccurate or unresponsive. In commonly owned and openly participatory networks, the risk of malicious behavior adds even greater importance of consensus around its current state and actions taken.

As a decentralized, peer-to-peer network with an initial focus on reliable storage and communication, the SAFE Network requires a high standard for data integrity. User files and account information are secured and stored in such a way that no major outage should affect access of data for the main network. While most P2P networks gain in security from a global network of distributed nodes (the SAFE Network further obfuscates traffic using global multiple hop routing), critical decisions for maintaining security of stored data are kept “localised” in SAFE for increased efficiency and privacy.

The Nature of Consensus: Global & Local

Before diving into the specifics of SAFE consensus, let’s do a bit of comparison with other recent developments in decentralized consensus design. One of the more interesting implementations was introduced several years ago with the launch of Bitcoin. The combination of proof of work and blockchain technologies has enabled an extremely reliable way to track a permanent and ordered list of cryptographically secure transactions. All nodes within the Bitcoin Network store and verify a copy of the blockchain which protects against tampering with historical transactions and faulty reporting of newly created ones. Unfortunately, the global and public nature of Bitcoin’s consensus process creates drawbacks with regard to efficiency and privacy. . The fact that all nodes in the network need to track and agree on a single, infinitely growing ledger has proven scaling problems and simplifies the ability for deep analysis of the ledger and user profiling. While various efforts are looking to solve theses issues, the years of research carried out by the MaidSafe team has resulted in a consensus mechanism designed specifically for privacy and efficiency – a different goal than the proof of concept architected by Satoshi Nakamoto for Bitcoin. This protocol is the basis of the SAFE Network and when compared to Bitcoin takes a very different approach, enabling actions and verifying network states based on local node consensus.

Those following MaidSafe may know of our preference for the emulation of natural systems which have hundreds of thousands of years of testing in diverse environments and harsh conditions. This philosophy can be extended to help understand a high level reasoning for our approach to consensus. Animal societies of all kinds localise decisions to reach agreements about immediate threat levels and other states of being while brains have evolved to localise neuron function for efficiency. Additionally, local consensus allows for the more sophisticated societies formed around humans to make intelligent decisions about sensitive actions such as an elected committee deciding on substantial policy changes for a community. Of course, these social situations come with their own vulnerabilities if the individuals involved in consensus decisions have similar self interested goals which do not reflect the interest of that which they govern. However thankfully, in computer networks, measures can be implemented which prevent local consensus abuse (or misuse) by nodes and it all starts with the foundation on which the network is built upon.

XOR Close Group Consensus

A recent post on this blog titled Structuring Networks with XOR outlines the basics of defining closeness within the SAFE Network’s distributed hash table (DHT) implementation. If you are not familiar with the foundation of Kademlia-based DHTs, that post will be a prerequisite to effectively understanding consensus process in SAFE that we will now dive deeper into. As we explore how such local consensus is approached using XOR closeness, it is important to keep in mind that “closeness” in this sense does not refer to geographical closeness, rather from a perspective of network address. So when nodes have IDs which are close in distance according to the XOR calculation, their physical locations could be on opposite sides of the planet.

By relating network elements in terms of XOR closeness, a unique group of the closest nodes to a given object can be determined and subsequently act in a managerial role for it. As long as these objects have unique IDs which can be translated to binary, everything from data to nodes can be related to each other in terms of distance and assigned a close group of network nodes (or as we call them, Vaults). This close group of Vaults can take on a variety of purposes depending on the object they surround, but center on management of data and node integrity consensus processes. The graph below shows how we can relate any object with ID n to its four closest online Vaults.

closest-group

Whether the data is public keys, encrypted personal files and client account information, or cryptocurrencies, close group authority is the basis for the SAFE Network’s ability to self-manage and self-heal. As long as nodes are not able to predetermine their location in the DHT address space, the inclusion within a close group is essentially random and drastically reduces any chance of group members colluding to disrupt the integrity of an action on particular piece of data. A future post detailing the security against various types of attacks will dive deeper into concepts like how IDs are assigned but for the purpose of understanding the consensus mechanism, we can view it as random. Further, each group consensus requires a quorum of vaults for authority approval which protects against a minority of unreliable nodes. The exact quorum to group size ratio will be investigated as part of ongoing test networks to balance efficiency with security. Additionally, as vaults in the network go on and offline (referred to as network churn), the members in close groups will be in a constant state of flux to accommodate for new or lost nodes.

Node and Group Authorities

The variety of close group authorities formed in SAFE are fundamentally determined based on the ID of the object which the vaults within that group surround. These distinct consensus responsibilities are referred to as personas. Client nodes act as a complementary authority for user authorised actions within the network and differ from Vault nodes as they do not route data but instead act as an interface for users to connect into the network and access or upload data. Each Vault node can also be considered an authority with the extremely limited capabilities of responding to requests for data they store. Using cryptographic key signing, the network verifies authority based on messages sent by Clients, personas and individual Vaults.

Some actions require just Client and Vault cryptographic authorisation (such as reading data already uploaded to the network) while others involve at least one persona consensus as well (such as storing new data to the network). Autonomous actions require no Client authority and solely rely on persona consensus and Vault cryptographic authorisation (such as network reconfiguration to reassign data storage when a Vault goes offline). These autonomous processes are what enables the SAFE Network’s ability to heal itself and manage nodes and stored data without the need for any centralised authority or Client action. This is a major difference from Bittorrent’s implementation of Kademlia which does not provide availability of data – if a few Bittorrent nodes hosting an niche piece of content all eventually go offline, there is no network procedure for reallocating that data and it therefore becomes inaccessible.

The Four Authorities

The network’s routing protocol defines four authorities. There are two node authorities: Client and ManagedNode (Vault storing data); and two fundamental persona types: ClientManager and NaeManager (Network Addressable Element) consensus groups. Fundamental persona types are defined based on placement in the DHT and XOR closeness to object IDs while ManagedNodes are determined based on their inclusion within a NaeManager group. The persona types are subcategorised into specialised responsibilities based on the type of data or account which they surround. It is expected that personas will overlap, meaning a single Vault might be included within several close groups simultaneously while also storing data assigned to it.

Authority Network
component
Persona Group
Sub-types
Responsibility
Client Client node N/A Private & public
data access
ClientManager
Persona
Vault node
close group
MaidManager Client authentication
& PUT allowance
MpidManager Client inbox/outbox
NaeManager
Persona
Vault node
close group
Data-
Manager
Immutable
data
GET rewards & data
relocation
Structured
data
GET rewards, data
updates & data
relocation
ComputeManager (TBA) TBA
ManagedNode Vault node N/A Store data & respond
to data requests

Client

While Clients have authority outside of group consensus, as previously mentioned, they have limited control and are never in a position which affects data reliability. Clients are the windows into the network for users and therefore will control data and messages from a client-side perspective such as encryption and decryption for uploaded data. For each client connection into the network, there is an anonymising proxy node which relays all data to and from destinations within the network, but the proxy does not have the ability to read any of it (for those familiar with Tor, this function is akin to a “guard node”).1

ClientManager

The ClientManager persona consists of Vaults closest to a Client ID and is subcategorised into MaidManager and MpidManager personas. MaidManager (Maid Anonymous ID) is adjacent to the personal ID associated with a Client and has the responsibility of managing that user’s personal account functions, such as upload allowance and authentication. MpidManger (Maid Public ID) surrounds the public ID associated with a Client and is responsible for maintaining the Client’s inbox and outbox for sending messages to other Clients.

NaeManager

The NaeManager persona consists of Vaults closest to network addressable elements such as data chunks. The initial release of SAFE will focus on implementing the persona type DataManager to take on the task of enforcing data storage reliability with future plans for ComputeManager persona type for reliably computing data. DataManager is further subcategorised into functions managing immutable data and structured data. ImmutableDataManager are a group of Vaults closest to the ID of an immutable chunk of data and manages GET rewards (safecoin farming) for successful ManagedNode responses and the relocation of data when one of these goes offline. Immutable data chunks are encrypted pieces of user uploaded files with IDs derived from the data chunk itself. A file is only able to be reassembled by users with access to the specific data map, more commonly known as a symmetric key file. StructuredDataManager is closest to the ID of structured data which are small, mutable data files owned by users on the network such as DNS entries, safecoin and data maps for immutable file reassembly. In addition to managing GET rewards for ManagedNodes storing the file and relocation responsibilities, StructuredDataManager will also acknowledge updates initiated by owners of the data (such reassigning ownership to another user).

ManagedNode

Like Clients, ManagedNodes have limited control in the authority functions they take on as individual Vaults. They only have control over data which they store and responses to requests for that data. Since all uploaded data is stored redundantly over at least a few ManagedNodes they are never in total control of any data piece. All Vaults in the network may store data and will take on this limited ManagedNode authority over a piece of data when assigned to the DataManager group surrounding that data ID. This means all DataManagers will also be ManagedNodes storing that data. The role a Vault takes on as a ManagedNode storing (or eventually computing) a piece of data is directly dependent on its role as a DataManager group member for that data, but the two authorities are nonetheless distinct.

The illustration below shows relationships of a Client (n) and the closest online Vaults which make up their ClientManager (n+2, n+5, n+7, n-8), a data chunk (m) and the closest online Vaults which make up its DataManager (m-1, m-2, m-3, m+7) and those DataManager Vaults acting also as ManagedNodes storing m.

circle-groups

Mapping Network Actions

With these various roles and authorities in mind, let’s explore some actions which can give a more complete view of how the network functions. For simplicity, we’ll use the same groups as the previous illustration for all examples. Each action within the network originates from one of the four authorities. Client initiates actions for storing new data or modifying and reading existing data they have access to while DataManager authority initiates restoring a piece of data when a ManagedNode becomes unresponsive. A ManagedNode will never initiate an action and only acts in authority to requests for data. Every action additionally involves cryptographic verification of authorities involved at each step.

PUT

When a logged in user uploads a new piece of data to the SAFE Network, a series of authorities come into play for securely and privately storing it. If a Client is putting a standard file type (such as a document or image) onto the network, it will first locally self-encrypt the data into chunks while creating a data map (symmetric key) and upload each data piece to be stored and managed within its own location on the DHT. As mentioned, these immutable data chunks have unique IDs derived from the contents of the encrypted chunk itself while the decryption key, or data map, is uploaded with its own unique ID within a structured data file. The self-encrypt video linked above illustrates how the data map is both a decryption key and list of data chunk IDs which double as pointers for locating them in the network. The authorities involved in uploading a single piece of data (whether immutable or structured) are as follows:

circle-groups-PUT

The Client sends a signed message via bootstrap relay node with the data chunk to its own ID which is picked up by the MaidManager close group in that part of the network. After checking the authority comes from the Client, a quorum of Vaults within this group must confirm the storage allowance for the Client before deducting an amount then sending the data and a message signed by the group to the location in the network where the ID for that data exists. In the case that the data already exists, no allowance is deducted and the Client is instead given access to existing data. If it does not exist, the message from the MaidManager is picked up by the closest group of Vaults to the data ID as a new DataManager, which checks the authority coming from the MaidManager persona. DataManager Vaults then initiate storage on each Vault in the group as individual ManagedNodes. Each ManagedNode sends a success response (or fail in the case of insufficient resources) back to data ID which is again picked up by the DataManager and forwarded back to the Client ID which is in turn picked up by the ClientManager and forwarded back to the Client.

GET

The action of reading a piece of data which a user has access to is a simpler process as there is no need for MaidManager consensus. Clients can send messages directly to ManagedNodes so long as they know the ID for the data which they store locally. In fact, there is no direct consensus needed to retrieve a data piece however in order to reward the data holder with proof of resource, DataManagers confirm successful response of data.

circle-groups-GET

The Client sends a message to the ID for the data they are looking for which is picked up by the closest ManagedNode among the DataManager group and responds with the data itself. If there is a problem obtaining the data from this Vault, a short timeout will trigger the second closest ManagedNode to instead respond with the data and so on. The DataManager group confirms response by the Vault and sends a reward message with a newly created structured data representing a unique safecoin to the public address of that particular node. A future post will go more in depth into safecoin creation, handling and cost metrics.

Relocate Stored Data

When the network churns and ManagedNodes holding data go offline, it is only natural that the network assigns data storage responsibilities to another node for preserving data retrievability. This is a network critical action and is one of the few instances of actions being initiated by a persona rather than a Client. The absence of a missing ManagedNode will be detected quickly as periodic heartbeat messages are sent between all connected nodes.

circle-groups-churn

A ManagedNode storing a particular piece of data that is unresponsive to a connected node will have its closest Vaults alerted. Once confirmed by the DataManager maintaining that data chunk, they choose the next closest Vault to the data ID to become a new ManagedNode and member of the DataManager group. The newly chosen ManagedNode sends a success response to the rest of the close group which is then confirmed.

Close Group Strategy

Decentralized data reliability is an important feature of systems which remove dependence on central parties. Furthermore, such systems which aim to preserve privacy for users must also consider their methods used for consensus and understand the trade-offs. Bitcoin’s global consensus around a single public ledger helps guarantee network and transaction status but lacks in scalability and privacy. By segregating vault responsibilities in SAFE based on XOR closeness, the network is able to achieve reliable data and network status maintenance without the need to reveal any information globally. The potential for attacks on consensus mechanisms also varies with their implementations. While no system can claim absolute security, sufficient measures can be put in place to reduce potential by increasing the difficulty and necessary resources for staging such an attack. In SAFE, for example, a series of close group consensus in a PUT forces attackers to control a quorum in multiple close groups. Additionally, network churn in such a large address space facilitates the constant distribution of new IDs makes Sybil attacks more difficult to attain as nodes are constantly showing up in random parts of the network. Node ranking can also be used in close groups to detect disagreements in consensus and downgrade or push out unagreeable nodes.

With the previous introduction of XOR properties in DHTs and overview of their use for close group consensus within SAFE, we hope we have provided a better general understanding of data reliability in the network. This authority process is used for every action on the network including managing safecoin ownership and transfer. Expect future posts which dive into details of attacks on close group consensus (and mitigations), data types in SAFE, and safecoin functionality including the built-in reward system for data holders, applications and public content. In the meantime, questions or discussion about the consensus approach are welcome in our community forum.

 

1This proxy also serves as the initial bootstrap node for introducing nodes back into the network whether a Client or Vault. All nodes start out as a client and negotiate connections to their future peers via their proxy node. The bootstrap node has no relationship to the Client or Vault (in terms of closeness) and is randomly chosen from a list of hardcoded nodes in a bootstrap config file, taken from a cache of previously used bootstrap nodes or through a service discovery mechanism such as a vault already connected to SAFE on the same local area network (LAN).

Further resources:

The Language of the Network (2014)

Autonomous Network whitepaper (2010)

‘Peer to Peer’ Public Key Infrastructure whitepaper (2010)

MaidSafe Distributed Hash Table whitepaper (2010)

MaidSafe Distributed File System whitepaper (2010)