Development

SAFE Network Development Summary – May 2017

We’ve had quite few requests on social media and on email these past few days requesting updates on development progress. These messages serve to remind us that not everyone has the time or the inclination to read the weekly development updates which we post each Thursday onto the forum. So many projects, so little time! So the intention with this post is to provide a summary of the most recent events and our hopes and expectations moving forward.

Image: Richard Tilney Bassett

Roadmap
The best place to start is our development roadmap, which we updated and published late last week. This web page tries to encapsulate all the complexities of development over time on 1 page so it’s pretty high level, but it is this snapshot view that most people seem to appreciate. You will notice that the roadmap outlines the major aspects of development and a rough indication of the order in which we anticipate tackling them.

You will also notice that we haven’t included timescales. In the past we have provided timescales for ‘launch’ of the network. These have always been wrong despite our best efforts. We have found it difficult to estimate timescales since, we believe, so much of what we have been working on is brand new technology, sometimes completely bespoke, and other times building on the work of other projects. Testing is also interesting, it really helps us understand more about how the network fits together and how it is utilised by our community, but invariably leads to more tweaking and testing with previously unplanned and unknown rework and test durations.

We believe that publishing release dates that have a high degree of uncertainty attached is not helpful to anyone and can cause more frustration than not publishing them at all. Network related development is typically where the biggest black holes are and as we get into incremental development client-side, we anticipate time scales will become more predictable.

Stable decentralised network
In late March we released test 15, a network that incorporated both data centre resource as well as enabling user run vaults. Within this release, users were also able to run the SAFE Browser, Launcher and demo app, which continue to facilitate the storage of private and public data, as well as create public ID’s and publish SAFE websites.

After 3 days of running a stable network without any lost data we realised we had reached an important milestone. While we had done this extensively in private tests, it was fantastic to see it running publicly and see the community reaction to it. Of course, life has a sense of humour and shortly after it became apparent that a script had been written that created fake accounts and filled the relatively small network with data, stopping the creation of new accounts or the uploading of new data. This was really helpful to us as it enabled us to find out what happens to the network when it reaches capacity in a real world setting. The fact that it behaved as expected was reassuring, although we’d be lying if didn’t admit to finding the spam attack a little frustrating. This is of course something that the integration of safecoin would stop, as the requirement to ‘pay’ to store data will make the attack expensive, while the incentive of safecoin to farmers would lead to a significantly bigger network.

What now?
Looking forward we are currently focussed in 3 main areas:

  • Catering for mobile devices.
  • Enabling greater user participation.
  • Improving the resilience and robustness of the network.

Mobile
The patience app developers have shown to this point is soon to be rewarded. The process of converting our APIs away from a REST paradigm to SDKs was essential to cater for mobile devices, as the requirement for REST APIs to maintain state would not have worked with mobile devices that disconnect and reconnect regularly. Users of the SAFE Network will gain access through the Authenticator, a secure gateway that protects user credentials from the application itself. The Authenticator is currently being bundled with the SAFE browser and will enable users to securely authenticate themselves onto the network, or enable them to browse publicly available data without logging in.

To implement Authenticator the team required to add a new data type, mutable data. The new data type improves the network efficiency, saves bandwidth, and provides the granular access control required by mobile platforms.

With mobile devices being so ubiquitous throughout the world, enabling mobile client access to the network, mutable data has been receiving significant focus. From a resource provision perspective, both alpha and beta versions of the network will require laptop and desktop and in time single board computers to earn safecoin when it is released. In time, we will look at enabling mobile devices being able to farm for safecoins when plugged into a power outlet and when in range of WiFi, however, as we will detail below this is not a priority for now.

More alphas
Some of the example applications that have been created are currently being ported to suit the new data type and to be compatible with the new APIs. The team are updating the documentation and are testing the applications using a mock network, and they seem to be far more stable than previous iterations which looks positive. We anticipate alpha 2 will encompass the new Mutable Data type and Authenticator, SAFE Browser DOM APIs and Node.js SDK, along with example apps, tutorials and documentation.

Image: Clint Adair

Alpha 3 will see our focus shift onto enabling a greater number of users to run Vaults from home by integrating uTP. Presently users must TCP port forward, or enable UPnP on their routers which requires a little set up in some cases. Adding uTP support will make for a more seamless process for many while making the network accessible to more users. uTP is used in some BitTorrent protocols and when implemented effectively helps to mitigate poor latency and facilitate the reliable and ordered delivery of data packets.

During this phase we will also integrate node ageing, a feature that make the network more resilient to consensus group attacks. The team will also implement the first part of data chains, a feature that has been planned for a while which it is anticipated will ultimately enable the secure republish of data should the network ever lose power, and to provide validation that data has been stored on the network.

Looking ahead
Beyond alpha 3 we will focus on:

  • Data Chains, part 2.
  • Data republish and network restarts.
  • A security audit of the network
  • Test safecoin
  • Real-time network upgrades
  • Network validated upgrades

As has been the case to this point we will continue to release multiple test nets regularly between each alpha network to prove the technology in a public setting, and to mitigate against the code regressing.

We continue to be grateful to the huge support of the people that take the time to run these networks and report back, you all know who you are!

MaidSafe Development Update

It has been an exceedingly busy summer at MaidSafe and it seems like a good point to recap on what we have working on and how we see things rolling out as we move forward. Since our last major update, when we announced the network running end-to-end, we have been adding some pretty significant features and making several enhancements.

Amongst the highlights, we implemented Unified Structured Data, a change that enables the network to only recognise two primary data types; immutable and structured data. The repercussions for what this enhancement brings to the network is significant. Not only do they allow the reduction of network traffic (reducing load and making it more efficient), it also removes much complexity and enhances the security of the network. It is anticipated that Unified Structured Data will lay the groundwork for features such as smart contracts and global computation. For those looking for more technical detail you can visit the proposal as it was implemented here.

We also recently completed work on the Decentralised Naming System (DNS), essentially the SAFE Network’s version of Internet domains. Keen to avoid many of the issues that we experience with the existing system, such as being centrally controlled by an entity, the SAFE Network provides a way to look up data related to any name. So, no more ‘http’ and a lot more ‘safe:’. At the end of August we released an example application showing this functionality and the example itself can be downloaded here.

Screenshot from 2015-09-02 22-16-30

This was a really exciting development as it enabled us to get more (in addition to the Crust, Self Encryption and Routing examples released early summer) software into the hands of users. In essence, the DNS example enabled users to set up a local network, add files to it (even a website) and then view them using a Firefox extension across multiple platforms. MaidSafe focus a lot on usability and it is therefore great to see comments like this (from Justin Chellis):

“I am not great at things like this, but it worked very easy for me..”

Receiving positive feedback is a real motivator for all at MaidSafe, it’s proof that we are making very clear progress toward our goal of providing the average man, woman and child, access to technology that keeps their digital information safe.

In addition to adding features, we have also had to spend some time going back and tidying up sections of code that were inevitably less than perfect given the pace at which we are working. A technical debt sprint has just finished and we are pleased to see much increased stability in the network. This effort is all being helped by our recently launched bounty program which enables MaidSafe to benefit from the work of several community developers, who between them submitted 12 pull requests over a 2 week period. It’s great to be able to harness the passion of external developers in a way that is mutually beneficial and we are really looking forward to seeing the bounty program flourish.

In the immediate future we are planning the next round of development and there are some big ticket features being planned, these include:

uTP Hole Punching – This will enable the nodes (clients and vaults) to talk to each other beyond NAT boundaries and facilitate users to join and become part of the network by contributing their computing resources. Crowd sourced infrastructure has arrived!

Messaging Infrastructure – This will be a really fun deliverable, but is also a big one. The messaging APIs for inter-node communication would be put in place and potentially the addition of the MaidSafe Public Identity in this iteration. It would allow chat engines and clients to work on SAFE-Network.

Messaging Infrastructure and Launcher implementation – It is anticipated that Launcher would be the only application that users gives their credentials (PIN, Keyword and Password) to. Launcher would then authorise apps on behalf of the user, giving each one of them a sandbox environment to work with. This not only prevents Apps from knowing the user credentials, but also removes the ability for them to tinker with folders/data of other Apps or user’s files and folders.

So, we think that 2 more development sprints will see us reach Dev Bundle 2, a network that anyone can join remotely, store and retrieve data and farm for test safecoin. At this point, it will also be possible for third party developers to start building apps for the network. As we continue the roll out of the network, we will be replacing the existing MaidSafe website with two new sites, one that is focussed toward end users and farmers, and the other to provide developers with a clear channel into using the network.

All in all, very exciting times ahead and we’re very grateful to have you on this journey with us.

MaidSafe Code Bounty Program

We have mentioned the introduction of a code bounty programme a few times recently and, in the true spirit of Open Source software, we want the community to be able to take an active role in helping MaidSafe roll out the SAFE Network. This will not only decentralise and spread knowledge of how the network functions, it will also enable SAFE to be released more quickly.  The following post will provide a bit more detail on how we see MaidSafe’s code bounty programme working initially.

MaidSafe will publish tasks at the start (and potentially during) a development sprint. These tasks will be listed in Jira where anyone with a Jira ID will be able to claim and start a task. The tasks will be fully specced out by the maintainer of that library whose e-mail address is published in that library’s GitHub readme. If you feel the task description is not clear then you should contact the library maintainer prior to accepting the task in Jira. The fact that it is not clear could be a sign that it isn’t well suited to your skill set. Here is a rundown of the terms:

The terms

  • Participation will be on a first come first serve basis
  • The bounty only applies to open Jira tasks identified by the MaidSafe team
  • The developer that claims the task will be given a sufficient amount of time to complete it. For example, a task that is scheduled for 8 points (around 8 hours work) should be completed in one day. However, if a GitHub pull request is not received at the end of this period, or the task is not correct (doesn’t pass tests/ other task requirements as specified in the task description), it will be made available to the rest of the community. We will need to be strict about timescales and work quality to avoid delays
  • All submissions should take the form of a GitHub pull request
  • If the task is complete (meets the spec and passes all tests ((including coverage)), MaidSafe will pay the bounty (in bitcoin) to the successful developer
  • Each contributor agrees to the terms of the MaidSafe Contributor Agreement
  • MaidSafe’s core development team and employees are not eligible for rewards

The rewards

  • $20 per story point (paid in bitcoin only). Story points are used by MaidSafe to estimate task timescales
  • 1 story point is typically calculated to be around 1 hours work
  • The reward will be paid on the merge of a contributors pull request to a btc account of their choosing. We would also donate the bounty to the preferred charity (one that accepts bitcoin) of the developer should they prefer. Contributors should enter the bitcoin address they would like the bounty paid to into the pull request description field
  • An exclusive and limited SAFE Network core dev t-shirt (this is a one off reward and is limited to one shirt per developer)
  • The developers name added to the main SAFE Network repositories

This is a process that is likely to change over time as the bounty programme develops and grows. We plan on starting the programme slowly and in about 2 weeks time we will identify the tasks from the current sprint (RUST-3) that are eligible. Over time, we anticipate opening up all Jira tasks to the community, but we would like to trial the process first. In future, we may start to pick much bigger pieces of work, such as the addition of specific network features, and ask community members to compete for them. These types of projects would obviously carry many more story points and be much more lucrative, but we should learn to walk before we can run!

So, hopefully this provides some more detail about the bounty scheme as a whole and we will start to announce applicable tasks in the next couple of weeks. We can’t wait to start working with you all more closely!

SAFE Network Roadmap

Keeping stakeholders updated with development progress is vital, but can be a difficult to thing to get right. Not only is development of large projects complex, particularly when they span across multiple libraries, but trying to communicate the nuances of development to an audience of varying technical ability is not without its challenges.

When pulling together the latest version of the SAFE Network roadmap, which can be found here, we wanted to show how all the libraries fit together, but also try to do so it in such a way that the deliverables at each stage of release are clear. For example, as you will see we are nearing the release of Dev Bundle 1 which will provide each users with:

  • Cross Platform (Windows, OSX, Linux) desktop Installers
  • Network (Crust, Routing and Vaults), running on external nodes
  • Remote client access and account creation
  • Simple vaults – LAN only
  • Initial Farming Rate data storage mechanism
  • Communication through TCP
  • NFS REST API exposing persistent remote storage

This means that third party developers will have access to a stable API that they can use to store data on a Local Area Network, a network that users can access via remote clients using TCP.

Dev Bundle 2 will see a significant number of new features added, such as: test safecoin, enhancements to our transport layer with UTP and the commencement of UI based applications. As we continue to iterate through each new release, we will add these to the rolling development roadmap. Please keep referring back at regular intervals to check progress, and for those looking for detailed updates we would recommend continuing to read our weekly dev update posts on the forum, as well as monitoring our Jira dashboard during development sprints (we have currently just finished a sprint and are about to get back into planning, hence the finished status of the current dash board).

It is worth noting at this point that the object of each new release will not necessarily be to add new features to the network. Some will be to improve the efficiency of the code base while reducing technical debt as we move forward.

The roadmap will be made interactive in the coming days, providing overviews, explaining the purpose of each library, while also providing direct links to the example applications for each. The different approach (in comparison to the last version) used when compiling this roadmap marks MaidSafe’s move to a much more modular development approach, reducing the coupling of components and dependencies where possible. This is to enable non specific SAFE Network libraries to be picked up and easily used by other projects.

Those that read our weekly development updates will also know that we have appointed primary and secondary maintainers for each library so that third party developers know where to go should they have specific questions. Each maintainer is listed at the top of the appropriate GitHub readme. This change allows knowledge transfer to accelerate from David to the MaidSafe engineers. Each library maintainer (and secondary maintainer) are now responsible for that library (or module) and this accountability and responsibility allows the engineers to grow and develop each part of the network. We anticipate this will allow for much greater community participation while also enabling a pace of development that has, until relatively recently, eluded us.

So hopefully this roadmap achieves what it set out to do; provide a snapshot and high level summary of our progress, articulate our delivery and ongoing plans, while showing how each of the network components come together to make something much greater than the sum of their parts.