Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: gilbertchen/duplicacy
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v2.7.1
Choose a base ref
...
head repository: gilbertchen/duplicacy
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
6 contributors

Commits on Oct 8, 2020

  1. Skip chunks already verified in previous runs for check -chunks. 

    This is done by storing the list of verified chunks in a file
`.duplicacy/cache/<storage>/verified_chunks`.
    @gilbertchen
    gilbertchen committed Oct 8, 2020
    6
    Copy the full SHA
    d7c1903 View commit details

Commits on Oct 9, 2020

  1. Don't attemp to load verified_chunks when it doesn't exist.

    @gilbertchen
    gilbertchen committed Oct 9, 2020
    Copy the full SHA
    7f834e8 View commit details

Commits on Oct 11, 2020

  1. Change snapshot source path from / to /System/Volumes/Data 

    Also use a regex to extract the snapshot date from tmutil output.
    @gilbertchen
    gilbertchen committed Oct 11, 2020
    2
    Copy the full SHA
    7c36311 View commit details

Commits on Oct 17, 2020

  1. Use github.com/pkg/xattr for reading/writing extended attributes. 

    The one previously used, github.com/redsift/xattr, is old and can only process
user-defined extended attributes, not system ones.
    @gilbertchen
    gilbertchen committed Oct 17, 2020
    3
    Copy the full SHA
    b392302 View commit details

Commits on Nov 3, 2020

  1. Fixed a bug that caused a fresh restore to fail without the -overwrit… 

    …e option

When restoring a file that doesn't exit locally, if the file is large (>100M)
Duplicacy will create an empty sparse file.  But this newly created file will
be mistaken for a local copy and hence the restore will fail with a message
suggesting the -overwrite option.
    @gilbertchen
    gilbertchen committed Nov 3, 2020
    2
    Copy the full SHA
    bec3a0e View commit details

Commits on Nov 5, 2020

  1. Validate the repository id for the init and add command 

    Only letter, numbers, dashes, and underscores are allowed.
    @gilbertchen
    gilbertchen committed Nov 5, 2020
    2
    Copy the full SHA
    5eed6c6 View commit details

Commits on Nov 16, 2020

  1. Update dependency (for gilbertchen/go-dropbox and github.com/pkg/xattr)

    @gilbertchen
    gilbertchen committed Nov 16, 2020
    Copy the full SHA
    ae706e3 View commit details

  2. Bump version to 2.7.2

    @gilbertchen
    gilbertchen committed Nov 16, 2020
    Copy the full SHA
    175adb1 View commit details

Commits on Nov 23, 2020

  1. Support GCD impersonation via modified service account file

    @gilbertchen
    gilbertchen committed Nov 23, 2020
    Copy the full SHA
    474f07e View commit details

  2. Add test storage for StorageMadeEasy's File Fabric

    @gilbertchen
    gilbertchen committed Nov 23, 2020
    Copy the full SHA
    41a8f65 View commit details

Commits on Jan 4, 2021

  1. Exit with code 2 when an invalid command is provided

    @gilbertchen
    gilbertchen committed Jan 4, 2021
    Copy the full SHA
    4104c2f View commit details

  2. Show the path in the error when a subdirectory can't be listed

    @gilbertchen
    gilbertchen committed Jan 4, 2021
    Copy the full SHA
    4cfecf1 View commit details

  3. Check in the paper accepted to IEEE Transactions on Cloud Computing

    @gilbertchen
    gilbertchen committed Jan 4, 2021
    Copy the full SHA
    ee9355b View commit details

  4. Add a link to the paper

    @gilbertchen
    gilbertchen authored Jan 4, 2021
    Copy the full SHA
    ba091fb View commit details

  5. Update README.md

    @gilbertchen
    gilbertchen authored Jan 4, 2021
    Copy the full SHA
    ecf5191 View commit details

Commits on Jan 28, 2021

  1. Fix SNAPSHOT_INACTIVE log message

    @gilbertchen
    gilbertchen committed Jan 28, 2021
    1
    Copy the full SHA
    f83e4f3 View commit details

Commits on Mar 9, 2021

  1. Merge pull request #612 from gilbertchen/gcd_impersonate 

    Support GCD impersonation via modified service account file
    @gilbertchen
    gilbertchen authored Mar 9, 2021
    Copy the full SHA
    fd1b7e1 View commit details

Commits on Mar 10, 2021

  1. Find the storage path in shared folders first when connecting to Goog… 

    …le Drive

When connecting to Google Drive with a service account key, only files in the
service account's own hidden drive space are listable.  This change finds
the given storage path among shared folders first so that folders from the user
space can be made accessible via service account.
    @gilbertchen
    gilbertchen committed Mar 10, 2021
    1
    Copy the full SHA
    e43e848 View commit details

Commits on Oct 8, 2021

  1. Download a fossil directly instead of turning it back to a chunk first 

    This is to avoid the read-after-rename consistency issue where the effect
of renaming may not be observed by the subsequent attempt to download the
just renamed chunk.
    @gilbertchen
    gilbertchen committed Oct 8, 2021
    1
    Copy the full SHA
    cacf661 View commit details

Commits on Oct 16, 2021

  1. Add a global option to print memory usage 

    This option, -print-memory-usage, will print memory usage every second while
the program is running.
    @gilbertchen
    gilbertchen committed Oct 16, 2021
    1
    Copy the full SHA
    68b6049 View commit details

Commits on Oct 25, 2021

  1. Rewrite the backup procedure to reduce memory usage 

    Main changes:

* Change the listing order of files/directories so that the local and remote
  snapshots can be compared on-the-fly.

* Introduce a new struct called EntryList that maintains a list of
  files/directories, which are kept in memory when the number is lower, and
  serialized into a file when there are too many.

* EntryList can also be turned into an on-disk incomplete snapshot quickly,
  to support fast-resume on next run.

* ChunkOperator can now download and upload chunks, thus replacing original
  ChunkDownloader and ChunkUploader.  The new ChunkDownloader is only used
  to prefetch chunks during the restore operation.
    @gilbertchen
    gilbertchen committed Oct 25, 2021
    Copy the full SHA
    d9f6545 View commit details

Commits on Dec 4, 2021

  1. First cut of Swift v2

    @markfeit
    markfeit committed Dec 4, 2021
    Copy the full SHA
    c363d21 View commit details

  2. Import context

    @markfeit
    markfeit committed Dec 4, 2021
    Copy the full SHA
    934c251 View commit details

  3. Typo

    @markfeit
    markfeit committed Dec 4, 2021
    Copy the full SHA
    041ba94 View commit details

  4. More fixups

    @markfeit
    markfeit committed Dec 4, 2021
    Copy the full SHA
    1661cae View commit details

  5. Use empty context

    @markfeit
    markfeit committed Dec 4, 2021
    Copy the full SHA
    95b1227 View commit details

  6. More dev

    @markfeit
    markfeit committed Dec 4, 2021
    Copy the full SHA
    0590daf View commit details

  7. More development

    @markfeit
    markfeit committed Dec 4, 2021
    Copy the full SHA
    590d3b1 View commit details

Commits on Dec 17, 2021

  1. DOn't cancel the context and use a sane deadline.

    @markfeit
    markfeit committed Dec 17, 2021
    Copy the full SHA
    4743c7b View commit details

Commits on Mar 4, 2022

  1. Update totalChunkSize in the chunk operator to show stats during restore

    @gilbertchen
    gilbertchen committed Mar 4, 2022
    Copy the full SHA
    ade669d View commit details

Commits on Mar 31, 2022

  1. Merge branch 'master' into memory_optimization

    @gilbertchen
    gilbertchen authored Mar 31, 2022
    Copy the full SHA
    f52dcf7 View commit details

Commits on Apr 8, 2022

  1. Don't parse test parameters in init() 

    This is to make test parameter parsing work with newer versions
of Go
    @gilbertchen
    gilbertchen committed Apr 8, 2022
    Copy the full SHA
    a953c4e View commit details

  2. Merge branch 'master' into memory_optimization

    @gilbertchen
    gilbertchen committed Apr 8, 2022
    Copy the full SHA
    fede9c7 View commit details

  3. Merge pull request #625 from gilbertchen/memory_optimization 

    Rewrite the backup procedure to reduce memory usage
    @gilbertchen
    gilbertchen authored Apr 8, 2022
    Copy the full SHA
    f2d6de3 View commit details

  4. Fixed the type of a test parameter

    @gilbertchen
    gilbertchen committed Apr 8, 2022
    Copy the full SHA
    8df529d View commit details

Commits on Apr 9, 2022

  1. Merge pull request #626 from markfeit/swift-v2 

    Swift v2
    @gilbertchen
    gilbertchen authored Apr 9, 2022
    Copy the full SHA
    345fc5e View commit details

  2. Dependency change: update github.com/ncw/swift to v2.0.1

    @gilbertchen
    gilbertchen committed Apr 9, 2022
    Copy the full SHA
    0d8a37f View commit details

Commits on Jun 10, 2022

  1. Initialize startTime correctly in CreateChunkOperator

    @gilbertchen
    gilbertchen committed Jun 10, 2022
    1
    Copy the full SHA
    fc2386f View commit details

Commits on Jul 10, 2022

  1. Fixed a bug that referenced uninitialized operator.snapshotCache

    @gilbertchen
    gilbertchen committed Jul 10, 2022
    1
    Copy the full SHA
    54952ce View commit details

Commits on Jul 23, 2022

  1. CLI support for custom credentials for OneDrive (client_id/client_sec… 

    …ret)
    @sevimo123
    sevimo123 committed Jul 23, 2022
    Copy the full SHA
    238ef63 View commit details

  2. Added support for SharePoint document libraries via odb://DRIVEID@pat… 

    …h/to/storage.
    @sevimo123
    sevimo123 committed Jul 23, 2022
    Copy the full SHA
    d7593a8 View commit details

Commits on Aug 13, 2022

  1. Use long-lived refresh token for the Dropbox backend 

    The refresh token can be downloaded from https://duplicacy.com/dropbox_start
    @gilbertchen
    gilbertchen committed Aug 13, 2022
    2
    Copy the full SHA
    cde660e View commit details

Commits on Sep 30, 2022

  1. Add a storage backend for Storj 

    The url format is storj://satellite/bucket/path.  You can get the
satellite along with the api access key when requesting an Access
Grant of type API Access.
    @gilbertchen
    gilbertchen committed Sep 30, 2022
    2
    Copy the full SHA
    6009f64 View commit details

Commits on Oct 4, 2022

  1. Use swift V2

    @gilbertchen
    gilbertchen committed Oct 4, 2022
    Copy the full SHA
    2c63d32 View commit details

  2. Switch to go modules

    @gilbertchen
    gilbertchen committed Oct 4, 2022
    Copy the full SHA
    5c35ef7 View commit details

  3. Bump version to 2.8.0

    @gilbertchen
    gilbertchen committed Oct 4, 2022
    Copy the full SHA
    4b47ea5 View commit details

Commits on Oct 6, 2022

  1. Bump version to 3.0.0

    @gilbertchen
    gilbertchen committed Oct 6, 2022
    Copy the full SHA
    d6f5336 View commit details

Commits on Oct 7, 2022

  1. Update dependencies

    @gilbertchen
    gilbertchen committed Oct 7, 2022
    Copy the full SHA
    901044b View commit details

  2. Bump version to 3.0.1

    @gilbertchen
    gilbertchen committed Oct 7, 2022
    1
    Copy the full SHA
    72eb339 View commit details

Commits on Nov 9, 2022

  1. Handle zero-byte files correctly 

    This commit fixed 2 bugs.  The first bug occurs when an incomplete backup
contains a zero-byte file and no chunks.  The second bug occurs when the
repository contains only zero-byte files.
    @gilbertchen
    gilbertchen committed Nov 9, 2022
    3
    Copy the full SHA
    3472206 View commit details
Showing with 4,335 additions and 2,884 deletions.
  1. +0 −270 Gopkg.lock
  2. +0 −98 Gopkg.toml
  3. +3 −1 README.md
  4. +98 −8 duplicacy/duplicacy_main.go
  5. BIN duplicacy_paper.pdf
  6. +78 −0 go.mod
  7. +302 −0 go.sum
  8. +0 −153 src/duplicacy_acdclient_test.go
  9. +20 −2 src/duplicacy_b2client.go
  10. +0 −133 src/duplicacy_b2client_test.go
  11. +545 −606 src/duplicacy_backupmanager.go
  12. +39 −28 src/duplicacy_backupmanager_test.go
  13. +18 −18 src/duplicacy_benchmark.go
  14. +111 −33 src/duplicacy_chunk.go
  15. +5 −5 src/duplicacy_chunk_test.go
  16. +30 −290 src/duplicacy_chunkdownloader.go
  17. +99 −90 src/duplicacy_chunkmaker.go
  18. +29 −45 src/duplicacy_chunkmaker_test.go
  19. +408 −38 src/duplicacy_chunkoperator.go
  20. +11 −21 src/{duplicacy_chunkuploader_test.go → duplicacy_chunkoperator_test.go}
  21. +0 −151 src/duplicacy_chunkuploader.go
  22. +31 −7 src/duplicacy_config.go
  23. +2 −2 src/duplicacy_dropboxstorage.go
  24. +388 −42 src/duplicacy_entry.go
  25. +59 −13 src/duplicacy_entry_test.go
  26. +574 −0 src/duplicacy_entrylist.go
  27. +179 −0 src/duplicacy_entrylist_test.go
  28. +78 −23 src/duplicacy_gcdstorage.go
  29. +0 −149 src/duplicacy_hubicclient_test.go
  30. +51 −20 src/duplicacy_oneclient.go
  31. +0 −145 src/duplicacy_oneclient_test.go
  32. +2 −2 src/duplicacy_onestorage.go
  33. +250 −0 src/duplicacy_sambastorage.go
  34. +2 −2 src/duplicacy_sftpstorage.go
  35. +13 −13 src/duplicacy_shadowcopy_darwin.go
  36. +135 −133 src/duplicacy_snapshot.go
  37. +361 −248 src/duplicacy_snapshotmanager.go
  38. +15 −10 src/duplicacy_snapshotmanager_test.go
  39. +85 −3 src/duplicacy_storage.go
  40. +76 −53 src/duplicacy_storage_test.go
  41. +184 −0 src/duplicacy_storjstorage.go
  42. +26 −16 src/duplicacy_swiftstorage.go
  43. +14 −0 src/duplicacy_utils.go
  44. +14 −13 src/duplicacy_utils_others.go
270 changes: 0 additions & 270 deletions Gopkg.lock
View file

This file was deleted.

98 changes: 0 additions & 98 deletions Gopkg.toml
View file

This file was deleted.

4 changes: 3 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# Duplicacy: A lock-free deduplication cloud backup tool

Duplicacy is a new generation cross-platform cloud backup tool based on the idea of [Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication).
Duplicacy is a new generation cross-platform cloud backup tool based on the idea of [Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication).

Our paper explaining the inner workings of Duplicacy has been accepted by [IEEE Transactions on Cloud Computing](https://ieeexplore.ieee.org/document/9310668) and will appear in a future issue this year. The final draft version is available [here](https://github.com/gilbertchen/duplicacy/blob/master/duplicacy_paper.pdf) for those who don't have IEEE subscriptions.

This repository hosts source code, design documents, and binary releases of the command line version of Duplicacy. There is also a Web GUI frontend built for Windows, macOS, and Linux, available from https://duplicacy.com.

Loading