1.27 2016 Dec 2nd p1

Long time since the previous release (I hope you enjoy github master / trunk which we keep stable!) with lots of changes:

• Common "wdt> " prefix in all the logging (helps finding/filtering wdt specific logs when embedded in another service)
• Progress report during file discovery
• Throttler changes working with the resource controller and can be lowered in the middle of a transfer and improved fairness.
• Multiple applications support within one process, Wdt options not forced to be a singleton, apply to a tree of sub object below the application wdt object
• Allow receiver to specify its --hostname
• New --fsync option to fsync after the last block of each file is written.
• Incremental transfers (-enable_download_resumption) improved reporting
• Wdt URI encoding/decoding is now binary safe (transferid can be any binary data)
• Fixed disk mode (files transferred serially and not cut in blocks) >2Gb errors; integer overflow
• Fewer dependencies. Uses std::make_uniq (need cxx14)
• Receiver now drives Sender's download resumption through the URI
• Benchmark generation (bench/) code
• New more efficient Varint encoding and less likely to have memory corruption
• Bug fixes (see commits) like listening sockets staying open after receiver timeout
• Misc changes like: Release/build improvements, Code quality improvements like constructor cleanups (#16)

Enjoy!

1.26 2016 March 4th p0

• Option to use fadvise(POSIX_FADVISE_DONTNEED) to avoid taking too much cache/buffers on sender/read path
• other fixes/improvements
• Simpler/shorter tests

v1.26 Feb 9th 2016 p0

All of the goodies in v1.25 plus:

• Support for deletion during resumption through the delete_extra_files - if a subsequent snapshot has fewer files than were already sent before, delete those first. This is useful for use cases like rocksdb/myrocks where after compaction some files disappear
• Dynamic ports (available ports) is now the default, you don't need to specify -start_port 0 anymore for the receiver side of wdt, if you want to revert to fixed ports, use -static_ports or set a non 0 start_port explicitly. This should make it easier for wdt transfers using pipe to connect the receiver and sender to always succeed even if another transfer is already ongoing on the same destination

v1.25 Feb 5th 2016 p1

Long time since the previous release (but there is always github master!) - lot's of changes:

New features/major changes:

• Much better command line interface to match the need of the encryption key transmission securely, "-" argument for the sender reads the connection url from stdin, which the receiver emits on stdout. So the 2 can be plugged through a pipe (securely using ssh - which will only be used for secure key exchange, everything else blazing fast and encrypted through WDT's own socket)

ssh dsthost wdt -directory destdir | ssh srchost wdt -directory srcdir -
# Or more complex:
(ssh localhost wdt -fork -start_port 0 -directory /tmp/dst1; ls *.h ) | wdt -manifest - -

• Aes GCM crypto support (requires a recent openssl version) : is now the default - it does both encryption and validates the integrity in 1 pass. We also support this during error handling only resuming at the last block that was successfully validated.
• Socket buffer size can be set explicitly (using -send_buffer_size and -receive_buffer_size)

Bug fixes:

• improved TransferLogManager locking (fail fast when can't be acquired) and detection of deleted containing directory/log file
• ResourceController double create, wdtSend can cancel previous / preexisting transfer
• with opening files during discovery

Misc:

• Consistent use of "Mbytes" throughout WDT for 1024*1024 = 1048576 bytes (aka Mebibyte (MiB)
• Testing/path improvements
• Removing direct use of WdtOptions::get across the code base - now the options can be different for different sender/receivers in the same process.

v1.23 2015 Nov 30th 2015 p0

Big release !

New features:

• WDT now supports encryption (aes128)

The receiver is generating a one time 128bits secret, encoded in the URL. This secret needs to be transmitted securely (at facebook we use secure thrift for this but you can use ssh for instance). Note that we do not try to authenticate and only encrypt.

• Other improvements

Also estimate the time it may take for a slow receiver to consume the enqueued bytes to avoid timing out waiting for the ack to the end of transfer

There is a new https://github.com/facebook/wdt/blob/master/Wdt.h simpler API for C++ (which is work in progress but shows the direction we are headed)

• Bug fixes:

file descriptor (in FileInfo api) wasn't used when download resumption was also on, locking , error code summary, ...

• Known issues:

Issue #102

v1.22 2015 Oct 27th 2015 p0

Important bug fixes (notification change in big refactor was incorrect. issue with long running mode,...)

Changed code layout, moved to <wdt/X.h> from "X.h"

Clang-3.7 warnings

Summarize the error codes for the sender (more specific than ERROR/OK, for instance ABORT is differentiated from CONN_ERROR)

v1.22 2015 Oct 21st patch 0

New Features

1. Refactored the code to split Sender and Receiver into clearly identifiable single threaded code and multi-threaded part (SenderThread / ReceiverThread).
2. Wdt now has a thread controller, with thread primitives such as Barrier and Funnel Executors which made the writing of certain features of sender and receiver easier like doing process version mismatch, or sending file chunks in download resumption.
3. wdt_max_send_test.sh has been updated and is more robust. This test can be used to benchmark wdt and detect regressions.

Bugs fixed

1. A sender thread could fail on a connection and have a file in its thread transfer history that has not been acknowledged, however it wouldn't stop other threads to finish their part of the transfer even on getting the checkpoint, since the last source would not return to queue. This would result in the final status of transfers being error.
2. Transfer log manager had a writer thread that would write empty buffer to the transfer log file.
3. gcc4.9 SIOPF fix for one our static strings in GFlags.

v1.21 2015 Oct 5th patch 0

Changed the retry logic to be based on lack of progress instead of being arbitrarily attributed to files retry counts

Fixed serious int overflow in lseek result handling - files between 2 and 4g (and 6 and 8g and 10-12...) ie whose size has bit 31 set and being set using block mode would fail

Removed the "e"xit command on receiver (dated from old times where the receiver would run forever by default)

Download resumption (TransferLogManager), thread history, checkpoints cleanup and important bug fixes

v1.20 2015 Sept 24th p1

Really fixed URL backward compatibility (use -url_backward_compatibility=true if you have a wdt v20 talking to v18 or earlier)

new -exit_on_bad_flags=false if you want to continue despite bad command line (useful for forward compatibility)

shorter urls (wdt://host:port?num_port=4) and correct escaping of ipv6 [addr]

wdt receiver will now not overwrite any file by default unless you add -overwrite or use download resumption

misc bug fixes and cleanup

v1.19 2015 Sept 20th p0

Changes since previous tag (Sept 16) :

1. Added fd to FileInfo. If fd, is provided, we use that fd and do not open the file. I am currently using pread always. I have to more perf test for pread. Max send is really slow in my new dev server, so need to test for perf in some other machine.
2. If connection url was provided, sender ignored file list. Fixed that and
   added a test for file info list
3. Added an option in e2e simple test to test o_direct mode
4. Simplifying o_direct code. We should never define macros named O_DIRECT.