Implement peers / network

[yoshuawuyts]

hypercore now works pretty well, it's time to implement the networking part.

Todos

• implement feed.announce() (ref)
• implement feed.unannounce() (ref)
• implement pub feed.has() (ref)
• implement pub feed.has_all() (polymorphic variant of feed.has())
• implement pub feed.head() (ref)
• implement pub feed.replicate() (ref)
• implement pub feed.update() (ref)
• implement pub feed.downloaded() (ref)
• implement pub feed.finalize() (ref)
• implement Peer struct
• implement futures channel for events
• implement Event enum

Bonus

• implement pub feed.cancel() (ref)
• implement pub feed.clear() (ref)
• implement pub feed.seek() (ref)

Skipping

• implement pub feed.download() (ref)
• implement pub feed.undownload() (ref)

Resources

• source: replicate.js
• test: streams.js
• test: replicate.js
• bench: replicate-16kb-blocks

[yoshuawuyts]

Networking Vaporware API

let feed = Feed::default();

feed.append(b"data")?;

for event in feed {
  match event {
    Event::Data(data) => {},
    Event::PeerAdd => {},
    Event::PeerRemove => {},
    Event::Download => {},
    Event::Synchronize => {},
    Event::Append => {},
  }
}

[pierd]

What are you planning on using for networking?

1. std::net
2. tokio
3. Are we waiting for official async/await support?
4. ...

[yoshuawuyts]

@pierd great question! -- I've bee thinking about this a bunch, even to the point where I've joined the networking WG to make more sense of the landscape.

I think there's no evading that we need tokio + async/await support for this to work well. Luckily most of this should be in a usable state sometime soon, which is great! But there's no guarantees.

So I'm thinking for next steps it'd be good to start implementing the Dat networking layer. This would include extracting / re-building (if it isn't in libp2p already) of the Kademlia Peer Selection structure https://github.com/ethereum/wiki/wiki/Kademlia-Peer-Selection, and the work our way up to implement the Dat networking layer. WIP references are available here dat-ecosystem-archive/DEPs#8 and here dat-ecosystem-archive/DEPs#7. Providing feedback to those would be super helpful too!

So yeah, there's a few steps to take here. I'd happily provide write access to datrs if anyone wants to start building out parts of the networking layer (so we can keep the stack centralized, which is great for maintenance!).

Hope this all makes sense! -- Sorry for not communicating much of this earlier; only got a good grasp of the next steps to take here when I met up with mafintosh in Vancouver earlier this month.

References

• https://github.com/ethereum/wiki/wiki/Kademlia-Peer-Selection
• Draft: Wire protocol (DEP) dat-ecosystem-archive/DEPs#8
• WIP: Peer Discovery DEP dat-ecosystem-archive/DEPs#7

[soyuka]

I think that a nice addition would be to follow https://github.com/bluejekyll/trust-dns and their technical decisions when it comes to networking (https://github.com/bluejekyll/trust-dns/issues/569#issuecomment-433561439). IIRC dns multicast is also part of the swarm-discovery process (see https://www.npmjs.com/package/@hyperswarm/discovery)

Also worth mentioning: libp2p/libp2p#28 libp2p/rust-libp2p#590

[RangerMauve]

Regarding networking, it seems that Dat is going to be migrating to hyperswarm once that's up and running.

[bltavares]

I would like to help with this. I've started a small test to discover how mdns and libp2p works, and if it would interop with current dat on node, specially with the hyperswarm migration.

Any pointers to start/continue with this?

[yoshuawuyts]

@bltavares I haven't done any research on the network protocol yet, so it probably requires scouring through the JS reference impl to find out. Doing that and sharing the notes would probably be the first step in making progress on this.

I'm thinking that a first version of the network protocol should probably tackle TCP first, and a single discovery mechanism. It can then later be expanded to cover more protocols once we have an MVP.

[bltavares]

I've started a research project on the network interop on bltavares/colmeia, specially to understand how viable is to use existing rust crates for discovery.

My plan is to find a running dat repo on my local network, then find it use hyperswarm dht, and provide a list of ip:ports to connect for a given dat public key. (No url DNS lookup yet).

I will take this slowly, as much of my computer time is being used at work and I'm trying to take things slowly :)

I'll push my notes and discoveries on that repo :)

[Frando]

I played around with hypercore-protocol in Rust too and made it up to receiving a first data message from a regular nodejs hypercore replication stream, but with the NOISE handshaking and encryption disabled. See hypercore-protocol-rust-experiments and this PR to disable NOISE in hypercore.

[bltavares]

I've finished this weekend with a barely functional dat server. It is broadcasting MDNS requests on the network, and it can be reached by npx dat share on the same computer.

The handshake part is working, as well as the encryption. It needs now to implement the remaining protocol to exchange data.

I'll park this a bit, and try to get it cross-compiled to run on Android just to check if it is working :)

[bltavares]

On another news: colmeia runs in Android already! pics

[bltavares]

I've started playing around with the idea of colmeia-clone.

The design I'm coming from was the idea of having a Observer structure that implements the network stack, and contains two hypercore feeds. It did not work as hypercore is not Send.

Draft PR

The networks stack on hypercore API seems to live inside hypercore, given the .download method, .replicate, Peer struct and the vaporware API example on events,

I can think of a couple of ideas:

• Implement the network stack trait on hypercore directly in attempt to match the JS implementation
• Make hypercore send so it can be used on async-io fashion (might be needed to implement streams for the feed event api)

I'll try to write an async test case on hypercore to drive the changes to make it Send. Is this feasible and a good action plan?