Roadmap

SAFE Network Development Summary – May 2017

We’ve had quite few requests on social media and on email these past few days requesting updates on development progress. These messages serve to remind us that not everyone has the time or the inclination to read the weekly development updates which we post each Thursday onto the forum. So many projects, so little time! So the intention with this post is to provide a summary of the most recent events and our hopes and expectations moving forward.

Image: Richard Tilney Bassett

Roadmap
The best place to start is our development roadmap, which we updated and published late last week. This web page tries to encapsulate all the complexities of development over time on 1 page so it’s pretty high level, but it is this snapshot view that most people seem to appreciate. You will notice that the roadmap outlines the major aspects of development and a rough indication of the order in which we anticipate tackling them.

You will also notice that we haven’t included timescales. In the past we have provided timescales for ‘launch’ of the network. These have always been wrong despite our best efforts. We have found it difficult to estimate timescales since, we believe, so much of what we have been working on is brand new technology, sometimes completely bespoke, and other times building on the work of other projects. Testing is also interesting, it really helps us understand more about how the network fits together and how it is utilised by our community, but invariably leads to more tweaking and testing with previously unplanned and unknown rework and test durations.

We believe that publishing release dates that have a high degree of uncertainty attached is not helpful to anyone and can cause more frustration than not publishing them at all. Network related development is typically where the biggest black holes are and as we get into incremental development client-side, we anticipate time scales will become more predictable.

Stable decentralised network
In late March we released test 15, a network that incorporated both data centre resource as well as enabling user run vaults. Within this release, users were also able to run the SAFE Browser, Launcher and demo app, which continue to facilitate the storage of private and public data, as well as create public ID’s and publish SAFE websites.

After 3 days of running a stable network without any lost data we realised we had reached an important milestone. While we had done this extensively in private tests, it was fantastic to see it running publicly and see the community reaction to it. Of course, life has a sense of humour and shortly after it became apparent that a script had been written that created fake accounts and filled the relatively small network with data, stopping the creation of new accounts or the uploading of new data. This was really helpful to us as it enabled us to find out what happens to the network when it reaches capacity in a real world setting. The fact that it behaved as expected was reassuring, although we’d be lying if didn’t admit to finding the spam attack a little frustrating. This is of course something that the integration of safecoin would stop, as the requirement to ‘pay’ to store data will make the attack expensive, while the incentive of safecoin to farmers would lead to a significantly bigger network.

What now?
Looking forward we are currently focussed in 3 main areas:

  • Catering for mobile devices.
  • Enabling greater user participation.
  • Improving the resilience and robustness of the network.

Mobile
The patience app developers have shown to this point is soon to be rewarded. The process of converting our APIs away from a REST paradigm to SDKs was essential to cater for mobile devices, as the requirement for REST APIs to maintain state would not have worked with mobile devices that disconnect and reconnect regularly. Users of the SAFE Network will gain access through the Authenticator, a secure gateway that protects user credentials from the application itself. The Authenticator is currently being bundled with the SAFE browser and will enable users to securely authenticate themselves onto the network, or enable them to browse publicly available data without logging in.

To implement Authenticator the team required to add a new data type, mutable data. The new data type improves the network efficiency, saves bandwidth, and provides the granular access control required by mobile platforms.

With mobile devices being so ubiquitous throughout the world, enabling mobile client access to the network, mutable data has been receiving significant focus. From a resource provision perspective, both alpha and beta versions of the network will require laptop and desktop and in time single board computers to earn safecoin when it is released. In time, we will look at enabling mobile devices being able to farm for safecoins when plugged into a power outlet and when in range of WiFi, however, as we will detail below this is not a priority for now.

More alphas
Some of the example applications that have been created are currently being ported to suit the new data type and to be compatible with the new APIs. The team are updating the documentation and are testing the applications using a mock network, and they seem to be far more stable than previous iterations which looks positive. We anticipate alpha 2 will encompass the new Mutable Data type and Authenticator, SAFE Browser DOM APIs and Node.js SDK, along with example apps, tutorials and documentation.

Image: Clint Adair

Alpha 3 will see our focus shift onto enabling a greater number of users to run Vaults from home by integrating uTP. Presently users must TCP port forward, or enable UPnP on their routers which requires a little set up in some cases. Adding uTP support will make for a more seamless process for many while making the network accessible to more users. uTP is used in some BitTorrent protocols and when implemented effectively helps to mitigate poor latency and facilitate the reliable and ordered delivery of data packets.

During this phase we will also integrate node ageing, a feature that make the network more resilient to consensus group attacks. The team will also implement the first part of data chains, a feature that has been planned for a while which it is anticipated will ultimately enable the secure republish of data should the network ever lose power, and to provide validation that data has been stored on the network.

Looking ahead
Beyond alpha 3 we will focus on:

  • Data Chains, part 2.
  • Data republish and network restarts.
  • A security audit of the network
  • Test safecoin
  • Real-time network upgrades
  • Network validated upgrades

As has been the case to this point we will continue to release multiple test nets regularly between each alpha network to prove the technology in a public setting, and to mitigate against the code regressing.

We continue to be grateful to the huge support of the people that take the time to run these networks and report back, you all know who you are!

The Ants Are Coming!

It is fair to say that MaidSafe have been fairly quiet in our communication of late, not a complete radio blackout, but certainly low key. This has been intentional on our part. Communication can lose its meaning and impact if you shout about every little thing. We like to keep our powder dry, as they say.

But, now we have something meaningful to convey!

I want to bring everyone who is interested up to speed with what we have been doing during 2015, specifically the past three months, which have been some of the most transformational in MaidSafe’s nine-year history. We do make weekly dev updates on the community-run forum, but these can be pretty technical, and not everyone can get a full picture that way. So I will try and condense these points into as limited an amount of technical language as I can muster.

Rather than starting in early 2015, however, we will start this post at the present day. (I’m a sucker for Quentin Tarantino films and it will hopefully make for a less laborious post!)

Early(ish) on Friday (26th June) morning, the tireless MaidSafe dev team managed to get the SAFE Network functioning from end to end. This means that a User is able to self-authenticate (create their own username and password, and access the network without the permission of a third party) using the Authentication API via the SAFE Client. During the process, the network’s transport/connection layer (Crust) connects the peer nodes, allowing Routing (the layer that verifies the identity of each node cryptographically) to establish and maintain connections to other network nodes (for more information on how all the components fit together, please visit our wiki). The clients were also able to PUT (store) and GET (retrieve) data.

This is a phenomenal and long-sought achievement for everyone at MaidSafe, and is truly a major milestone ticked off. Yes, we have had an operational network before, but never one so stable, efficient or lacking in complexity. This lack of complexity is a very good thing and something of a personal mission for David. A significant reduction in the lines of code (from hundreds of thousands to a few tens of thousands!) not only enables new network features to be added much more quickly, it also makes it easier for bugs to be spotted and for other projects to utilise our code, much in the same way that we have employed the code of others. Open source collaboration is truly a wonderful thing to be a part of!

This new chapter in our development progress all started back in early February when David, again desperate to reduce complexity, spotted a pattern in the code, a repeatable and describable pattern that kickstarted the simplification of the decision-making logic within the Vaults (the series of processes that help look after all data on the network) and also improved network security. You can read David’s detailed blog post about it here.

Soon after, heavily frustrated by the speed at which we were developing in C++, we started looking into ways to speed up development without a reduction in code quality, surely the holy grail of software development. After much research, David became increasingly convinced that a new systems-level language, Rust, had something to offer. In his spare time (between about 2 a.m. and 5 a.m.) he started transposing one of our most complex libraries, Self-Encryption (the component that seamlessly splits data into smaller chunks and encrypts them), over to Rust, which at that time wasn’t even in Beta yet. This was very successful and fast! David followed a similar process with MaidSafe’s Routing library. With another successful test complete our development team was split for a few weeks while the core team remained in C++ and another team started transposing the rest of the code.

This was a risky and scary time. To split the dev team at a period when we were under significant pressure to produce a stable network seemed counterintuitive. But thankfully, going backwards to move forward paid off, and without this change there is no doubt in our minds that we would not be where we are today. It is not the intention to go into detail here about how and whether Rust is better than C++. For debates on that subject you can check out some of the threads on the forum and elsewhere. I think that is a debate as contentious as GPL vs MIT, or even Borg vs MacEnroe, fun to debate but don’t expect consensus any time soon. All I can say is that it’s working for us and allows us to iterate quickly, be more defined with our tasks and be more definitive with our timescales.

During this time we also realised that we needed to be more third-party-developer focussed. And that doesn’t mean being more open with our development process; we’re pretty open already. Rather, it means organising our libraries in such as way that they can be picked up and used by any third-party developer without being heavily coupled to other libraries. In addition to whatever assistance this gives to other projects in the open-source community, this helps Project SAFE in two specific ways: first, allowing external use of these discrete libraries helps validate their functionality in a variety of different projects; and, second, it allows us to demonstrate, at a library level, that each component of the network carries out its desired function.

So, beginning in late April we started to release each of the component libraries as console apps, essentially providing the libraries functionality with a text-only interface (such as terminal in mac and linux, or the command line on Windows) rather than a Graphical User Interface. The first console app was Self-Encryption, and since that time we have also released Crust and Routing. All MaidSafe libraries are published on crates.io, where a stable release of each library can be used by other projects, and dependencies known (dependencies are the other programs that the library relies on to run). This process is not only useful for third-party developers, it also gets the MaidSafe team into a habit of delivering working code regularly and in a routine way.

We have found that this more modular and more approachable way of working has led to many more developers committing code to the repositories, and it is valuable code. Such is the desire for this network, and also the desire of people to try Rust. We will accommodate these desires and pay anyone who contributes to our code sprints, starting from this upcoming sprint (we will update everyone on the bounty scheme next week on this blog). We anticipate this will give us better code faster and with a more engaged and willing community, the impact of this cannot be underestimated. This is an area we will work hard on, to ensure the community is well-rewarded and recognised for contributions, as we feel they should be.

So, now that we have a running and more stable network than we have ever had before, what next? What does the future hold? Well, we can look to our roadmap for the answers. In the very near future (in the next couple of weeks), we will be releasing installers to enable users to download and run a SAFE Network locally on their individual computers. Beyond that our next development sprint will start week commencing July 6th (probably lasting for around three weeks) and though we are still mapping out specific objectives, this sprint is likely to consist of some or all of the following:

  • Further implementation of safecoin (some work here has already started)
  • Implementation of messaging infrastructure
  • Remove Transaction Managers (reduce complexity, code and increase security on the network)
  • Implement App Launcher infrastructure (enables secure login to third-party apps without the App Developer getting visibility of a User’s credentials)
  • Implement cross-platform/cross-network/multi-protocol network connections (everyone can run farming Vaults at home or in any network)
  • Implement POSIX disk interface (any app can treat the network as a local hard disk)
  • Implement public names and shares ( SAFE version of www, email, DNS and more)

From there we should quickly (hopefully within 2 sprints) see the delivery of Dev Bundle 2. This iteration will provide a network where Farmers can start to contribute resource and have this measured, where developers will be able to access and utilise stable APIs and start to build applications that everyone can use. These apps might be simple at first, such as encrypted messaging and data storage apps, but in a short space of time will have the potential to develop into anything that can be done on the existing web and beyond. We will possibly split out Dev Bundle 2 on the roadmap so that some of what is currently listed will become Dev Bundle 3. This way the deliverable of each feature focussed sprint will be an updated bundle.

So, this recent development milestone is hugely significant and we are very close to delivering the SAFE Network, something that the world would seem to need even more than it did when David started this journey nine years ago. We will continue to keep you all updated via the forum and this blog.

Thanks all, for your continued participation and support.

SAFE Network Roadmap

Keeping stakeholders updated with development progress is vital, but can be a difficult to thing to get right. Not only is development of large projects complex, particularly when they span across multiple libraries, but trying to communicate the nuances of development to an audience of varying technical ability is not without its challenges.

When pulling together the latest version of the SAFE Network roadmap, which can be found here, we wanted to show how all the libraries fit together, but also try to do so it in such a way that the deliverables at each stage of release are clear. For example, as you will see we are nearing the release of Dev Bundle 1 which will provide each users with:

  • Cross Platform (Windows, OSX, Linux) desktop Installers
  • Network (Crust, Routing and Vaults), running on external nodes
  • Remote client access and account creation
  • Simple vaults – LAN only
  • Initial Farming Rate data storage mechanism
  • Communication through TCP
  • NFS REST API exposing persistent remote storage

This means that third party developers will have access to a stable API that they can use to store data on a Local Area Network, a network that users can access via remote clients using TCP.

Dev Bundle 2 will see a significant number of new features added, such as: test safecoin, enhancements to our transport layer with UTP and the commencement of UI based applications. As we continue to iterate through each new release, we will add these to the rolling development roadmap. Please keep referring back at regular intervals to check progress, and for those looking for detailed updates we would recommend continuing to read our weekly dev update posts on the forum, as well as monitoring our Jira dashboard during development sprints (we have currently just finished a sprint and are about to get back into planning, hence the finished status of the current dash board).

It is worth noting at this point that the object of each new release will not necessarily be to add new features to the network. Some will be to improve the efficiency of the code base while reducing technical debt as we move forward.

The roadmap will be made interactive in the coming days, providing overviews, explaining the purpose of each library, while also providing direct links to the example applications for each. The different approach (in comparison to the last version) used when compiling this roadmap marks MaidSafe’s move to a much more modular development approach, reducing the coupling of components and dependencies where possible. This is to enable non specific SAFE Network libraries to be picked up and easily used by other projects.

Those that read our weekly development updates will also know that we have appointed primary and secondary maintainers for each library so that third party developers know where to go should they have specific questions. Each maintainer is listed at the top of the appropriate GitHub readme. This change allows knowledge transfer to accelerate from David to the MaidSafe engineers. Each library maintainer (and secondary maintainer) are now responsible for that library (or module) and this accountability and responsibility allows the engineers to grow and develop each part of the network. We anticipate this will allow for much greater community participation while also enabling a pace of development that has, until relatively recently, eluded us.

So hopefully this roadmap achieves what it set out to do; provide a snapshot and high level summary of our progress, articulate our delivery and ongoing plans, while showing how each of the network components come together to make something much greater than the sum of their parts.