Secure Copy Protocol implemented in Go

Overview

Production-ready Secure Copy Protocol (aka: SCP) implemented in Go with well documentation and neat dependency.

Introduction

Secure Copy Protocol uses Secure Shell (SSH) to transfer files between host on a network.

There is no RFC that defines the specifics of the protocol. This package simply implements SCP against the OpenSSH's scp tool, thus you can directly transfer files to/from *uinx system within your Go code, as long as the remote host has OpenSSH installed.

Features

• Copy file from local to remote.
• Copy file from remote to local.
• Copy from buffer to remote file. (e.g: copy from bytes.Reader)
• Copy from remote file to buffer. (e.g: copy to os.Stdout)
• Recursively copy directory from local to remote.
• Recursively copy directory from remote to local.
• Set permission bits for transferred files.
• Set timeout/context for transfer.
• Preserve the permission bits and modification time at transfer.
• No resources leak. (e.g: goroutine, file descriptor)
• Low memory consuming for transferring huge files.
• TODO:
  • Transfer speed limit.
  • Performance benchmark/optimization for lots of small files.
• Won't support:
  • Copy file from remote to remote.

Install

go get github.com/povsister/scp

Example usage

This package leverages golang.org/x/crypto/ssh to establish a SSH connection to remote host.

Error handling are omitted in examples!

Copy a file to remote

// Build a SSH config from username/password
sshConf := scp.NewSSHConfigFromPassword("username", "password")

// Build a SSH config from private key
privPEM, err := ioutil.ReadFile("/path/to/privateKey")
// without passphrase
sshConf, err := scp.NewSSHConfigFromPrivateKey("username", privPEM)
// with passphrase
sshConf, err := scp.NewSSHConfigFromPrivateKey("username", privPEM, passphrase)

// Dial SSH to "my.server.com:22".
// If your SSH server does not listen on 22, simply suffix the address with port.
// e.g: "my.server.com:1234"
scpClient, err := scp.NewClient("my.server.com", sshConf, &scp.ClientOption{})

// Build a SCP client based on existing "golang.org/x/crypto/ssh.Client"
scpClient, err := scp.NewClientFromExistingSSH(existingSSHClient, &scp.ClientOption{})

defer scpClient.Close()


// Do the file transfer without timeout/context
err = scpClient.CopyFileToRemote("/path/to/local/file", "/path/at/remote", &scp.FileTransferOption{})

// Do the file copy with timeout, context and file properties preserved.
// Note that the context and timeout will both take effect.
fo := &scp.FileTransferOption{
    Context: yourCotext,
    Timeout: 30 * time.Second, 
    PreserveProp: true,
}
err = scpClient.CopyFileToRemote("/path/to/local/file", "/path/at/remote", fo)

Copy a file from remote

// Copy the file from remote and save it as "/path/to/local/file".
err = scpClient.CopyFileFromRemote("/path/to/remote/file", "/path/to/local/file", &scp.FileTransferOption{})

// Copy the remote file and print it in Stdout.
err = scpClient.CopyFromRemote("/path/to/remote/file", os.Stdout, &scp.FileTransferOption{})

Copy from buffer to remote as a file

// From buffer
buffer := []byte("something excited")
reader := bytes.NewReader(buffer)

// From fd
// Note that its YOUR responsibility to CLOSE the fd after transfer.
reader, err := os.Open("/path/to/local/file")
defer reader.Close()


// Note that the reader must implement "KnownSize" interface except os.File
// For the content length must be provided before transfer.
// The last part of remote location will be used as file name at remote.
err := scpClient.CopyToRemote(reader, "/path/to/remote/file", &scp.FileTransferOption{})

Recursively copy a directory to remote

// recursively copy to remote
err := scpClient.CopyDirToRemote("/path/to/local/dir", "/path/to/remote/dir", &scp.DirTransferOption{})

// recursively copy to remote with timeout, context and file properties.
// Note that the context and timeout will both take effect.
do := &scp.DirTransferOption{
    Context: yourContext,
    Timeout: 10 * time.Minute,
    PreserveProp: true,
}
err:= scpClient.CopyDirToRemote("/path/to/local/dir", "/path/to/remote/dir", do)

// recursively copy directory contents to remote. Does not create the
// source directory on the target side, like `scp -r /path/to/dir/* ...`
do := &scp.DirTransferOption{
    ContentOnly: true,
}
err := scpClient.CopyDirToRemote("/path/to/local/dir", "/path/to/remote/dir", do)

Recursively copy a directory from remote

// recursively copy from remote.
// The content of remote dir will be save under "/path/to/local".
err := scpClient.CopyDirFromRemote("/path/to/remote/dir", "/path/to/local", &scp.DirTransferOption{})

Progress

// Defines an anonymous function for ReceivePassthrough that conf the
// Writer attribute of PassthroughCopy to track and display the upload 
// progress. The new Writer writes data to the original writer, calculates
// the upload progress, and prints it as a percentage.

type Progress struct {
    TotalSize int64
    Writed int
}

func (p *Progress) Draw(n) () {
    p.Writed = n
    progress := float64(p.writed) / float64(p.TotalSize) * 100
    fmt.Printf("progress: %.2f%%\n", progress)
}

func (p *Progress) Close() error {
    return nil
}

var bar *Progress
do := &scp.DirTransferOption{
    ObserverCallback: func(oet scp.ObserveEventType, ti scp.TransferInfo) {
      if oet == scp.ObserveEventStart {
        bar = &Progress{TotalSize: ti.TotalSize()}
        return
      }
      bar.Draw(ti.TransferredSize())
    },
}
err = scpClient.CopyDirToRemote("/path/to/local/dir", "/path/to/remote/dir", do)

Something you need to know

SCP is a light-weighted protocol which implements file transfer only. It does not support advanced features like: directory listing, resume from break-point.

So, it's commonly used for transferring some small-size, temporary files. If you heavily depend on the file transfer, you may consider using SFTP instead.

Another thing you may notice is that I didn't put context.Context as the first argument in function signature. Instead, it's located in TransferOption. This is intentional because it makes the API also light-weighted.

License

MIT License