Concerns about network communications

[gabrielecirulli]

Admittedly, I know almost nothing about networking. Reading enough documentation, I can get a basic file transfer thing going, but there might be some important things I'm glossing over. I'd like to list the ones I can think of here, and keep this issue open for discussion to figure out any missing points.

Security

The server is currently implemented with net.createServer. This is a simple TCP server, which I guess already covers most of the error-checking needs, but I think does not provide any security. Node also offers a tcp module, and I think Koral should take security at the core, but having had no experience with it, I'm not sure the requirements for setting this up are (certificates, key pairs, etc)

Error checking

When taking an initial stab at file transfer (in c082261), I realized that there's a need for transferring extra information, such as (rough list):

• File path/name (the only one currently implemented)
  • Currently implemented by adding a UInt16BE to the start of the file to hold the length in bytes of the "file path" relative to the user's Koral work directory, a string which is then read directly after this number, up to its length. This maybe hints at the need for a proper header format for these payloads, especially to encode the rest of the information.
  • How do you secure this against tampering (e.g. malicious clients embedding ".." in the file path, or similar attempts)
    • Should the Koral server run under a chroot?
• A CRC32 of the file
  • Is this actually necessary? Maybe TCP already renders it redundant.
• A precise timestamp of when the change was made, in order to allow multiple users to modify the same file
  • This doesn't prevent/help solve conflicts. How should that be done?
  • A token to recognise the user
• Storing the file size: I'm currently sending the file size as a UInt32 field. This means the file size is capped to 4GB.
  • Does this make sense?
  • Should it be higher?
  • Should it not be present at all (just extract the contents until the end of the buffer)?
• Buffer overflows: do we have to protect against them/prevent them? I don't know enough about this to even know what to look for.

Authentication

The current basic implementation accepts all inbound connections on its port (I chose 5555, is is a good idea?) and just chugs with it. The Koral server will have a database of users (each Koral installation should act as a full-fledged web app, with its own private database), and it should check against a token sent with every payload to recognise the user it is bound to and to authenticate it. The tokens should be issued through a standard method of authentication, like oauth.

Rsync algorithm

One idea would be to use the rdiff/rsync algorithm to perform the entire file transfer. Besides being quite performant, it also benefits from years of widespread usage, and it means we wouldn't have to face a lot of the hurdles of file syncing.