Code

From duply (simple duplicity) - a duplicity shell frontend
Jump to navigation Jump to search


duply sourcecode

duply's svn repository is not the most recent source for the code. Usually the svn only gets updated on releases. But not always. Development snapshots are taken daily. These are not guaranteed to work, but might contain unreleased fixes. Check the snapshot's changelog header accordingly.

SVN access

Browse the repository

https://sourceforge.net/p/ftplicity/code/HEAD/tree/duply/trunk/

SVN access

svn co https://svn.code.sf.net/p/ftplicity/code/duply/trunk duply

SVN Log

https://sourceforge.net/p/ftplicity/code/commit_browser

Latest Development Snapshot

mod time

ERROR in secure-include.php::ef_include_get_errors (line 976) - Executing this 'shell' code is currently prohibited because $wg_include_allowed_checksums[shell] does not contain a matching checksum!

plain/text -> duply.sh

   1 #!/usr/bin/env bash
   2 #
   3 ################################################################################
   4 #  duply (grown out of ftplicity), is a shell front end to duplicity that      #
   5 #  simplifies the usage by managing settings for backup jobs in profiles.      #
   6 #  It supports executing multiple commands in a batch mode to enable single    #
   7 #  line cron entries and executes pre/post backup scripts.                     #
   8 #  Since version 1.5.0 all duplicity backends are supported. Hence the name    #
   9 #  changed from ftplicity to duply.                                            #
  10 #  See http://duply.net or http://ftplicity.sourceforge.net/ for more info.    #
  11 #  (c) 2006 Christiane Ruetten, Heise Zeitschriften Verlag, Germany            #
  12 #  (c) 2008-2023 Edgar Soldin (changes since version 1.3)                      #
  13 ################################################################################
  14 #  LICENSE:                                                                    #
  15 #  This program is licensed under GPLv2.                                       #
  16 #  Please read the accompanying license information in gpl-2.0.txt.            #
  17 ################################################################################
  18 #  TODO/IDEAS/KNOWN PROBLEMS:
  19 #  - possibility to restore time frames (incl. deleted files)
  20 #    realizable by listing each backup and restore from 
  21 #    oldest to the newest, problem: not performant
  22 #  - search file in all backups function and show available
  23 #    versions with backups date (list old avail since 0.6.06)
  24 #  - edit profile opens conf file in vi 
  25 #  - implement log-fd interpretation
  26 #  - add a duplicity option check against the options pending 
  27 #    deprecation since 0.5.10 namely --time-separator
  28 #                                    --short-filenames
  29 #                                    --old-filenames
  30 #  - add 'exclude_<command>' list usage e.g. exclude_verify
  31 #  - featreq 25: a download/install duplicity option
  32 #  - import/export profile from/to .tgz function !!!
  33 #  - remove url_encode, test for invalid chars n throw error instead
  34 #
  35 #  CHANGELOG:
  36 #  2.5.2 (30.11.2023)
  37 #  - bug #139: "ampersand (&) in gpg passphrase breaks gpg tests"
  38 #
  39 #  2.5.1 (4.10.2023)
  40 #  - quotewrap only strings with quotes ('") or spaces from now on
  41 #  - add --verbosity only if set in profile conf
  42 #  - bugfix #138: fix quoting when filtering params, thx Eric
  43 #  - bugfix #137: relax version parsing regex
  44 #
  45 #  2.5.0 (25.09.2023)
  46 #  - bugfix #136: "not compatible with duplicity 2.x", thx tengel, lds, Rhomeo
  47 #    check for duplicity 2.1+ (2.0 broke implied commands), 
  48 #    command line ui changed incompatibly
  49 #  - filter in/excludes more strictly for more duplicity actions now
  50 #  - replace '--file-to-restore' with '--path-to-restore'
  51 #  - filter backup only params now
  52 #
  53 #  2.4.3 (05.05.2023)
  54 #  - bugfix #134: workaround bash 4.2 and earlier read bug (thx Tavio Wong)
  55 #
  56 #  2.4.2 (19.01.2023)
  57 #  - featreq #55: change to purgeAuto in systemd unit files (thx B.Foresman)
  58 #  - featreq #56: systemd files should go in /etc, not /lib (thx B.Foresman)
  59 #  - bugfix #133: read -N not available on macOS (thx Peter Torelli)
  60 #
  61 #  2.4.1 (09.09.2022)
  62 #  - fixup duplicity links, moved to http://duplicity.us
  63 #  - bugfix: duply hangs on awk version detection on OpenBSD (thx phthomas137)
  64 #
  65 #  2.4 (06.04.2022)
  66 #  - bugfix #127: date_from_nsecs ignores format string
  67 #  - bugfix #116: separators print date now too
  68 #  - featreq #48: add purgeAuto command (see man page)
  69 #  - replaced tab indents with 2spaces everywhere
  70 #  - bugfix #129,131,132: duply stumbles over 'python -s' shebang,
  71 #      python interpreter parse failed if duplicity is a snap app
  72 #  - bugfix #130: duplicity version check failed "gpg: WARNING: ..."
  73 #  - version output, always print PYTHONPATH, if interpreter was determined
  74 #  - update python references to python3
  75 #
  76 #  2.3.1 (11.02.2021)
  77 #  - bugfix 123: symmetric encryption errs out, asks for '' private key
  78 #
  79 #  2.3 (30.12.2020)
  80 #  - don't import whole key pair anymore if only pub/sec is requested
  81 #  - gpg import routine informs on missing key files in profile now
  82 #  - add check/import needed secret key for decryption
  83 #  - featreq 50: Disable GPG key backups, implemented/added settings
  84 #      GPG_IMPORT/GPG_EXPORT='disabled' to conf template
  85 #
  86 #  2.2.2 (24.02.2020)
  87 #  - bugfix 120: Failures in "Autoset trust of key" during restore 
  88 #    because of gpg2.2 fingerprint output change
  89 #
  90 #  2.2.1 (22.01.2020)
  91 #  - featreq 46: Example systemd units & Howto, courtesy of Jozef Riha
  92 #  - featreq 47: Clarify message about keeping the profile, also by Jozef Riha
  93 #  - fix abbreviation spelling of 'e.g.'
  94 #
  95 #  2.2 (30.12.2018)
  96 #  - featreq 44: implement grouping for batch commands
  97 #      new separators are [] (square brackets) or groupIn/groupOut
  98 #      command 'backup' translates now to [pre_bkp_post] to be skipped as
  99 #      one block in case a condition was set in the batch instruction
 100 #
 101 #  2.1 (23.07.2018)
 102 #  - be more verbose when duplicity version detection fails
 103 #  - using info shows python binary's path for easier identification now
 104 #  - reworked python interpreter handling, it's either
 105 #      configured per PYTHON var
 106 #      unconfigured, parsed from duplicity shebang
 107 #      or set to current duplicity default 'python2' (was 'python' until now)
 108 #  - do not quotewrap strings because of slashes (e.g. paths) anymore
 109 #  - bugfix: improved in/exclude stripping from conf DUPL_PARAMS
 110 #
 111 #  2.0.4 (20.02.2018)
 112 #  - bugfix 114: "duply usage is not current" wrt. purgeFull/Incr
 113 #  - bugfix 115: typo in error message - "Not GPG_KEY entries" should be "No"
 114 #  - bugfix 117: no duply_ prefix when ARCH_DIR is set in conf
 115 #  - bugfix debian 882159: duply: occasionally shows negative runtimes
 116 #
 117 #  2.0.3 (29.08.2017)
 118 #  - bugfix: "line 2231: CMDS: bad array subscript"
 119 #  - bugfix 112: "env: illegal option -- u" on MacOSX
 120 #
 121 #  2.0.2 (23.05.2017)
 122 #  - bugfix: never insert creds into file:// targets
 123 #  - bugfix: avail profiles hint sometimes shortend the names by one char
 124 #  - bugfix 108: CMD_NEXT variable should ignore conditional commands (and, or)
 125 #  - export condition before/after next/prev command as CND_PREV,CND_NEXT now
 126 #  - bugfix 97: Unknown command should be ERROR, not WARNING 
 127 #
 128 #  2.0.1 (16.11.2016)
 129 #  - bugfix 104: Duply 2.0 sets wrong archive dir, --name was always 'duply_'
 130 #
 131 #  2.0 (27.10.2016)
 132 #  made this a major version change, as we broke backward compatibility anyway
 133 #  (see last change in v1.10). got complaints that rightfully pointed out
 134 #  that should only come w/ a major version change. so, here we go ;)
 135 #  if your backend stops working w/ this version create a new profile and
 136 #  export the env vars needed as described in the comments of the conf file
 137 #  directly above the SOURCE setting.
 138 #  Changes:
 139 #  - making sure multi spaces in TARGET survive awk processing
 140 #  - new env var PROFILE exported to scripts 
 141 #  - fix 102: expose a unique timestamp variable for pre/post scripts
 142 #    actually a featreq. exporting RUN_START nanosec unix timestamp
 143 #  - fix 101: GPG_AGENT_INFO is 'bogus' (thx Thomas Harning Jr.)
 144 #  - fix 96: duply cannot handle two consecutive spaces in paths
 145 #
 146 #  1.11.3 (29.5.2016)
 147 #  - fix wrong "WARNING: No running gpg-agent ..." when sign key was not set
 148 #
 149 #  1.11.2 (11.2.2016)
 150 #  - fix "gpg: unsafe" version print out 
 151 #  - bugfix 91: v1.11 [r47] broke asymmetric encryption when using GPG_KEYS_ENC
 152 #  - bugfix 90: S3: TARGET_USER/PASS have no effect, added additional 
 153 #    documentation about needed env vars to template conf file
 154 #
 155 #  1.11.1 (18.12.2015)
 156 #  - bugfix 89: "Duply has trouble with PYTHON-interpreter" on OSX homebrew
 157 #  - reverted duply's default PYTHON to 'python'
 158 #
 159 #  1.11 (24.11.2015)
 160 #  - remove obsolete --ssh-askpass routine
 161 #  - add PYTHON conf var to allow global override of used python interpreter
 162 #  - enforced usage of "python2" in PATH as default interpreter for internal
 163 #    use _and_ to run duplicity (setup.py changed the shebang to the fixed
 164 #    path /usr/bin/python until 0.7.05, which we circumvent this way)
 165 #  - featreq 36: support gpg-connect-agent as a means to detect if an agent is 
 166 #    running (thx Thomas Harning Jr.), used gpg-agent for detection though
 167 #  - quotewrapped run_cmd parameters to protect it from spaces e.g. in TMP path
 168 #  - key export routine respects gpg-agent usage now
 169 #
 170 #  1.10.1 (19.8.2015)
 171 #  - bugfix 86: Duply+Swift outputs warning
 172 #  - bugfix 87: Swift fails without BACKEND_URL
 173 #
 174 #  1.10 (31.7.2015)
 175 #  - featreq 37: busybox issues - fix awk, grep version detection,
 176 #    fix grep failure because --color=never switch is unsupported
 177 #    (thx Thomas Harning Jr. for reporting and helping to debug/fix it)
 178 #  - bugfix 81: --exclude-globbing-filelist is deprecated since 0.7.03
 179 #    (thx Joachim Wiedorn, also for maintaining the debian package)
 180 #  - implemented base-/dirname as bash functions
 181 #  - featreq 31 " Support for duplicity Azure backend " - ignored a 
 182 #    contributed patch by Scott McKenzie and instead opted for removing almost
 183 #    all code that deals with special env vars required by backends.
 184 #    adding and modifying these results in too much overhead so i dropped this
 185 #    feature. the future alternative for users is to consult the duplicity 
 186 #    manpage and add the needed export definitions to the conf file.
 187 #    appended a commented example to the template conf below the auth section.
 188 #
 189 #  1.9.2 (21.6.2015)
 190 #  - bugfix: exporting keys with gpg2.1 works now (thx Philip Jocks)
 191 #  - documented GPG_OPTS needed for gpg2.1 to conf template (thx Troy Engel)
 192 #  - bugfix 82: GREP_OPTIONS=--color=always disrupted time calculation
 193 #  - added GPG conf var (see conf template for details)
 194 #  - added grep version output as it is an integral needed binary
 195 #  - added PYTHONPATH printout in version output
 196 #
 197 #  1.9.1 (13.10.2014)
 198 #  - export CMD_ERR now for scripts to detect if CMD_PREV failed/succeeded
 199 #  - bugfix: CMD_PREV contained command even if it was skipped
 200 #
 201 #  1.9.0 (24.8.2014)
 202 #  - bugfix: env vars were not exported when external script was executable
 203 #  - rework GPG_KEY handling, allow virtually anything now (uid, keyid etc.) 
 204 #    see gpg manpage, section "How to specify a user ID"
 205 #    let gpg complain when the delivered values are invalid for whatever reason
 206 #  - started to rework tmp space checking, exposed folder & writable check
 207 #    TODO: reimplement enough file space available checking
 208 #
 209 #  1.8.0 (13.7.2014)
 210 #  - add command verifyPath to expose 'verify --file-to-restore' action
 211 #  - add time parameter support to verify command
 212 #  - add section time formats to usage output 
 213 #
 214 #  1.7.4 (24.6.2014)
 215 #  - remove ubuntu one support, service is discontinued
 216 #  - featreq 31: add authenticated swift (contributed by Justus Seifert)
 217 #
 218 #  1.7.3 (3.4.2014)
 219 #  - bugfix: test routines, gpg2 asked for passphrase although GPG_PW was set
 220 #
 221 #  1.7.2 (1.4.2014 "April,April")
 222 #  - bugfix: debian Bug#743190 "duply no longer allows restoration without 
 223 #     gpg passphrase in conf file"
 224 #     GPG_AGENT_INFO env var is now needed to trigger --use-agent
 225 #  - bugfix: gpg keyenc test routines didn't work if GPG_PW was not set
 226 #
 227 #  1.7.1 (30.3.2014)
 228 #  - bugfix: purge-* commands renamed to purgeFull, purgeIncr due to 
 229 #     incompatibility with new minus batch separator 
 230 #
 231 #  1.7.0 (20.3.2014)
 232 #  - disabled gpg key id plausibility check, too many valid possibilities
 233 #  - featreq 7 "Halt if precondition fails":
 234 #     added and(+), or(-) batch command(separator) support
 235 #  - featreq 26 "pre/post script with shebang line": 
 236 #     if a script is flagged executable it's executed in a subshell 
 237 #     now as opposed to sourced to bash, which is the default
 238 #  - bugfix: do not check if dpbx, swift credentials are set anymore 
 239 #  - bugfix: properly escape profile name, archdir if used as arguments
 240 #  - add DUPL_PRECMD conf setting for use with e.g. trickle
 241 #
 242 #  1.6.0 (1.1.2014)
 243 #  - support gs backend
 244 #  - support dropbox backend
 245 #  - add gpg-agent support to gpg test routines
 246 #  - autoenable --use-agent if passwords were not defined in config
 247 #  - GPG_OPTS are now honored everywhere, keyrings or complete gpg
 248 #    homedir can thus be configured to be located anywhere
 249 #  - always import both secret and public key if avail from config profile
 250 #  - new explanatory comments in initial exclude file
 251 #  - bugfix 7: Duply only imports one key at a time 
 252 #
 253 #  1.5.11 (19.07.2013)
 254 #  - purge-incr command for remove-all-inc-of-but-n-full feature added
 255 #    patch provided by Moritz Augsburger, thanks!
 256 #  - documented version command in man page
 257 #
 258 #  1.5.10 (26.03.2013)
 259 #  - minor indent and documentation fixes
 260 #  - bugfix: exclude filter failed on ubuntu, mawk w/o posix char class support
 261 #  - bugfix: fix url_decoding generally and for python3
 262 #  - bugfix 3609075: wrong script results in status line (thx David Epping)
 263 #
 264 #  1.5.9 (22.11.2012)
 265 #  - bugfix 3588926: filter --exclude* params for restore/fetch ate too much
 266 #  - restore/fetch now also ignores --include* or --exclude='foobar' 
 267 #
 268 #  1.5.8 (26.10.2012)
 269 #  - bugfix 3575487: implement proper cloud files support
 270 #
 271 #  1.5.7 (10.06.2012)
 272 #  - bugfix 3531450: Cannot use space in target URL (file:///) anymore
 273 #
 274 #  1.5.6 (24.5.2012)
 275 #  - commands purge, purge-full have no default value anymore for security 
 276 #    reasons; instead max value can be given via cmd line or must be set
 277 #    in profile; else an error is shown.
 278 #  - minor man page modifications
 279 #
 280 #  versioning scheme will be simplified to [major].[minor].[patch] version
 281 #  with the next version raise
 282 #
 283 #  1.5.5.5 (4.2.2012)
 284 #  - bugfix 3479605: SEL context confused profile folder's permission check
 285 #  - colon ':' in url passphrase got ignored, added python driven url_decoding
 286 #    for user & pass to better process special chars
 287 #
 288 #  1.5.5.4 (16.10.2011)
 289 #  - bugfix 3421268: SFTP passwords from conf ignored and always prompted for
 290 #  - add support for separate sign passphrase (needs duplicity 0.6.14+)
 291 #
 292 #  1.5.5.3 (1.10.2011)
 293 #  - bugfix 3416690: preview threw echo1 error
 294 #  - fix unknown cmds error usage & friends if more than 2 params were given
 295 #
 296 #  1.5.5.2 (23.9.2011)
 297 #  - bugfix 3409643: ssh key auth did ask for passphrase (--ssh-askpass ?)
 298 #  - bugfix: mawk does not support \W and did not split multikey definitions
 299 #  - all parameters should survive  single (') and double (") quotes now
 300 #
 301 #  1.5.5.1 (7.6.2011)
 302 #  - featreq 3311881: add ftps as supported by duplicity 0.6.13 (thx mape2k)
 303 #  - bugfix 3312208: signing detection broke symmetric gpg test routine
 304 #
 305 #  1.5.5 (2.5.2011)
 306 #  - bugfix: fetch problem with space char in path, escape all params 
 307 #    containing non word chars
 308 #  - list available profiles, if given profile cannot be found
 309 #  - added --use-agent configuration hint
 310 #  - bugfix 3174133: --exclude* params in conf DUPL_PARAMS broke 
 311 #    fetch/restore
 312 #  - version command now prints out 'using installed' info
 313 #  - featreq 3166169: autotrust imported keys, based on code submitted by 
 314 #    Martin Ellis - imported keys are now automagically trusted ultimately 
 315 #  - new txt2man feature to create manpages for package maintainers
 316 #
 317 #  1.5.4.2 (6.1.2011)
 318 #  - new command changelog
 319 #  - bugfix 3109884: freebsd awk segfaulted on printf '%*', use print again
 320 #  - bugfix: freebsd awk hangs on 'awk -W version' 
 321 #  - bugfix 3150244: mawk does not know '--version'
 322 #  - minor help text improvements
 323 #  - new env vars CMD_PREV,CMD_NEXT replacing CMD env var for scripts
 324 #
 325 #  1.5.4.1 (4.12.2010)
 326 #  - output awk, python, bash version now in prolog
 327 #  - shebang uses /usr/bin/env now for freebsd compatibility, 
 328 #    bash not in /bin/bash 
 329 #  - new --disable-encryption parameter, 
 330 #    to override profile encr settings for one run
 331 #  - added exclude-if-present setting to conf template
 332 #  - bug 3126972: GPG_PW only needed for signing/symmetric encryption 
 333 #    (even though duplicity still needs it)
 334 #
 335 #  1.5.4 (15.11.2010)
 336 #  - as of 1.5.3 already, new ARCH_DIR config option
 337 #  - multiple key support
 338 #  - ftplicity-Feature Requests-2994929: separate encryption and signing key
 339 #  - key signing of symmetric encryption possible (duplicity patch committed)
 340 #  - gpg tests disable switch
 341 #  - gpg tests now previewable and more intelligent
 342 #
 343 #  1.5.3 (1.11.2010)
 344 #  - bugfix 3056628: improve busybox compatibility, grep did not have -m param
 345 #  - bugfix 2995408: allow empty password for PGP key
 346 #  - bugfix 2996459: Duply erroneously escapes '-' symbol in username
 347 #  - url_encode function is now pythonized
 348 #  - rsync uses FTP_PASSWORD now if duplicity 0.6.10+ , else issue warning
 349 #  - feature 3059262: Make pre and post aware of parameters, 
 350 #                     internal parameters + CMD of pre or post 
 351 #
 352 #  1.5.2.3 (16.4.2010)
 353 #  - bugfix: date again, should now work virtually anywhere
 354 #
 355 #  1.5.2.2 (3.4.2010)
 356 #  - minor bugfix: duplicity 0.6.8b version string now parsable
 357 #  - added INSTALL.txt
 358 #
 359 #  1.5.2.1 (23.3.2010)
 360 #  - bugfix: date formatting is awked now and should work on all platforms
 361 #
 362 #  1.5.2 (2.3.2010)
 363 #  - bugfix: errors print to STD_ERR now, failed tasks print an error message
 364 #  - added --name=duply_<profile> for duplicity 0.6.01+ to name cache folder
 365 #  - simplified & cleaned profileless commands, removed second instance
 366 #  - generalized separator time routines
 367 #  - added support for --no-encryption (GPG_KEY='disabled'), see conf examples
 368 #  - minor fixes
 369 #
 370 #  1.5.1.5 (5.2.2010)
 371 #  - bugfix: added special handling of credentials for rsync, imap(s)
 372 #
 373 #  1.5.1.4 (7.1.2010)
 374 #  - bugfix: nsecs defaults now to zeroes if date does not deliver [0-9]{9}
 375 #  - check if ncftp binary is available if url protocol is ftp
 376 #  - bugfix: duplicity output is now printed to screen directly to resolve
 377 #            'mem alloc problem' bug report
 378 #  - bugfix: passwords will not be in the url anymore to solve the 'duply shows
 379 #            sensitive data in process listing' bug report
 380 #
 381 #  1.5.1.3 (24.12.2009) 'merry xmas'
 382 #  - bugfix: gpg pass now apostrophed to allow space and friends
 383 #  - bugfix: credentials are now url encoded to allow special chars in them
 384 #            a note about url encoding has been added to the conf template
 385 #
 386 #  1.5.1.2 (1.11.2009)
 387 #  - bugfix: open parenthesis in password broke duplicity execution
 388 #  - bugfix: ssh/scp backend does not always need credentials e.g. key auth
 389 #
 390 #  1.5.1.1 (21.09.2009)
 391 #  - bugfix: fixed s3[+http] TARGET_PASS not needed routine
 392 #  - bugfix: TYPO in duply 1.5.1 prohibited the use of /etc/duply
 393 #    see https://sourceforge.net/tracker/index.php?func=detail&
 394 #                aid=2864410&group_id=217745&atid=1041147
 395 #
 396 #  1.5.1 (21.09.2009) - duply (fka. ftplicity)
 397 #  - first things first: ftplicity (being able to support all backends since 
 398 #    some time) will be called duply (fka. ftplicity) from now on. The addendum
 399 #    is for the time being to circumvent confusion.
 400 #  - bugfix: exit code is 1 (error) not 0 (success), if at least on duplicity 
 401 #            command failed
 402 #  - s3[+http] now supported natively by translating user/pass to access_key/
 403 #    secret_key environment variables needed by duplicity s3 boto backend 
 404 #  - bugfix: additional output lines do not confuse version check anymore
 405 #  - list command supports now age parameter (patch by stefan on feature 
 406 #    request tracker)
 407 #  - bugfix: option/param pairs are now correctly passed on to duplicity
 408 #  - bugfix: s3[+http] needs no TARGET_PASS if command is read only
 409 #
 410 #  1.5.0.2 (31.07.1009)
 411 #  - bugfix: insert password in target url didn't work with debian mawk
 412 #            related to previous bug report
 413 #
 414 #  1.5.0.1 (23.07.2009)
 415 #  - bugfix: gawk gensub dependency raised an error on debian's default mawk
 416 #            replaced with match/substr command combination (bug report)
 417 #            https://sf.net/tracker/?func=detail&atid=1041147&aid=2825388&
 418 #            group_id=217745
 419 #
 420 #  1.5.0 (01.07.2009)
 421 #  - removed ftp limitation, all duplicity backends should work now
 422 #  - bugfix: date for separator failed on openwrt busybox date, added a 
 423 #    detecting workaround, milliseconds are not available w/ busybox date
 424 #
 425 #  1.4.2.1 (14.05.2009)
 426 #  - bugfix: free temp space detection failed with lvm, fixed awk parse routine
 427 #
 428 #  1.4.2 (22.04.2009)
 429 #  - gpg keys are now exported as gpgkey.[id].asc , the suffix reflects the
 430 #    armored ascii nature, the id helps if the key is switched for some reason
 431 #    im/export routines are updated accordingly (import is backward compatible 
 432 #    to the old profile/gpgkey files)         
 433 #  - profile argument is treated as path if it contains slashes 
 434 #    (for details see usage)
 435 #  - non-ftplicity options (all but --preview currently) are now passed 
 436 #    on to duplicity 
 437 #  - removed need for stat in secure_conf, it is ls based now
 438 #  - added profile folder readable check
 439 #  - added gpg version & home info output
 440 #  - awk utility availability is now checked, because it was mandatory already
 441 #  - tmp space is now checked on writability and space requirement
 442 #    test fails on less than 25MB or configured $VOLSIZE, 
 443 #    test warns if there is less than two times $VOLSIZE because 
 444 #    that's required for --asynchronous-upload option  
 445 #  - gpg functionality is tested now before executing duplicity 
 446 #    test drive contains encryption, decryption, comparison, cleanup
 447 #    this is meant to detect non trusted or other gpg errors early
 448 #  - added possibility of doing symmetric encryption with duplicity
 449 #    set GPG_KEY="" or simply comment it out
 450 #  - added hints in config template on the depreciation of 
 451 #    --short-filenames, --time-separator duplicity options
 452 #
 453 #  new versioning scheme 1.4.2b => 1.4.2, 
 454 #  beta b's are replaced by a patch count number e.g. 1.4.2.1 will be assigned
 455 #  to the first bug fixing version and 1.4.2.2 to the second and so on
 456 #  also the releases will now have a release date formatted (Day.Month.Year)
 457 #
 458 #  1.4.1b1 - bugfix: ftplicity changed filesystem permission of a folder
 459 #            named exactly as the profile if existing in executing dir
 460 #          - improved plausibility checking of config and profile folder
 461 #          - secure_conf only acts if needed and prints a warning now
 462 #
 463 #  1.4.1b  - introduce status (duplicity collection-status) command
 464 #          - pre/post script output printed always now, not only on errors
 465 #          - new config parameter GPG_OPTS to pass gpg options
 466 #            added examples & comments to profile template conf
 467 #          - reworked separator times, added duration display
 468 #          - added --preview switch, to preview generated command lines
 469 #          - disabled MAX_AGE, MAX_FULL_BACKUPS, VERBOSITY in generated
 470 #            profiles because they have reasonable defaults now if not set
 471 #
 472 #  1.4.0b1 - bugfix: incr forces incremental backups on duplicity,
 473 #            therefore backup translates to pre_bkp_post now
 474 #          - bugfix: new command bkp, which represents duplicity's 
 475 #            default action (incr or full if full_if_older matches
 476 #            or no earlier backup chain is found)
 477 #
 478 #  new versioning scheme 1.4 => 1.4.0, added new minor revision number
 479 #  this is meant to slow down the rapid version growing but still keep 
 480 #  versions cleanly separated.
 481 #  only additional features will raise the new minor revision number. 
 482 #  all releases start as beta, each bugfix release will raise the beta 
 483 #  count, usually new features arrive before a version 'ripes' to stable
 484 #    
 485 #  1.4.0b
 486 #    1.4b  - added startup info on version, time, selected profile
 487 #          - added time output to separation lines
 488 #          - introduced: command purge-full implements duplicity's 
 489 #            remove-all-but-n-full functionality (patch by unknown),
 490 #            uses config variable $MAX_FULL_BACKUPS (default = 1)
 491 #          - purge config var $MAX_AGE defaults to 1M (month) now 
 492 #          - command full does not execute pre/post anymore
 493 #            use batch command pre_full_post if needed 
 494 #          - introduced batch mode cmd1_cmd2_etc
 495 #            (in turn removed the bvp command)
 496 #          - unknown/undefined command issues a warning/error now
 497 #          - bugfix: version check works with 0.4.2 and older now
 498 #    1.3b3 - introduced pre/post commands to execute/debug scripts
 499 #          - introduced bvp (backup, verify, purge)
 500 #          - bugfix: removed need for awk gensub, now mawk compatible
 501 #    1.3b2 - removed pre/post need executable bit set 
 502 #          - profiles now under ~/.ftplicity as folders
 503 #          - root can keep profiles in /etc/ftplicity, folder must be
 504 #            created by hand, existing profiles must be moved there
 505 #          - removed ftplicity in path requirement
 506 #          - bugfix: bash < v.3 did not know '=~'
 507 #          - bugfix: purge works again 
 508 #    1.3   - introduces multiple profiles support
 509 #          - modified some script errors/docs
 510 #          - reordered gpg key check import routine
 511 #          - added 'gpg key id not set' check
 512 #          - added error_gpg (adds how to setup gpg key howto)
 513 #          - bugfix: duplicity 0.4.4RC4+ parameter syntax changed
 514 #          - duplicity_version_check routine introduced
 515 #          - added time separator, shortnames, volsize, full_if_older 
 516 #            duplicity options to config file (inspired by stevie 
 517 #            from http://weareroot.de) 
 518 #    1.1.1 - bugfix: encryption reactivated
 519 #    1.1   - introduced config directory
 520 #    1.0   - first release
 521 ################################################################################
 522 
 523 # utility functions overriding binaries
 524 
 525 # wrap grep to override possible env set GREP_OPTIONS=--color=always
 526 function grep {
 527   command env "GREP_OPTIONS=" grep "$@"
 528 }
 529 
 530 # implement basename in plain bash
 531 function basename {
 532   local stripped="${1%/}"
 533   echo "${stripped##*/}"
 534 }
 535 
 536 # implement dirname in plain bash
 537 function dirname {
 538   echo ${1%/*}
 539 }
 540 
 541 # implement basic which in plain bash
 542 function which {
 543   type -p "$@"
 544 }
 545 
 546 # check availability of executables via file name or file paths
 547 function lookup {
 548   local bin="$1"
 549   # look for file names in path via bash hash OR 
 550   # look for executables at given relative/absolute location
 551   ( [ "${bin##*/}" == "$bin" ] && hash "$bin" 2>/dev/null ) || [ -x "$bin" ]
 552 }
 553 
 554 # important definitions #######################################################
 555 
 556 ME_LONG="$0"
 557 ME="$(basename $0)"
 558 ME_NAME="${ME%%.*}"
 559 ME_VERSION="2.5.2"
 560 ME_WEBSITE="https://duply.net"
 561 
 562 # default config values
 563 DEFAULT_SOURCE='/path/of/source'
 564 DEFAULT_TARGET='scheme://user[:password]@host[:port]/[/]path'
 565 DEFAULT_TARGET_USER='_backend_username_'
 566 DEFAULT_TARGET_PASS='_backend_password_'
 567 DEFAULT_GPG='gpg'
 568 DEFAULT_GPG_KEY='_KEY_ID_'
 569 DEFAULT_GPG_PW='_GPG_PASSWORD_'
 570 
 571 # function definitions ##########################
 572 
 573 function set_config { # sets global config vars
 574   local CONFHOME_COMPAT="$HOME/.ftplicity"
 575   local CONFHOME="$HOME/.duply"
 576   local CONFHOME_ETC_COMPAT="/etc/ftplicity"
 577   local CONFHOME_ETC="/etc/duply"
 578 
 579   # confdir can be delivered as path (must contain /)
 580   if [ `echo $FTPLCFG | grep /` ] ; then 
 581     CONFDIR=$(readlink -f $FTPLCFG 2>/dev/null || \
 582               ( echo $FTPLCFG|grep -v '^/' 1>/dev/null 2>&1 \
 583                && echo $(pwd)/${FTPLCFG} ) || \
 584               echo ${FTPLCFG})          
 585   # or DEFAULT in home/.duply folder (NEW)
 586   elif [ -d "${CONFHOME}" ]; then
 587     CONFDIR="${CONFHOME}/${FTPLCFG}"
 588   # or in home/.ftplicity folder (OLD)
 589   elif [ -d "${CONFHOME_COMPAT}" ]; then
 590     CONFDIR="${CONFHOME_COMPAT}/${FTPLCFG}"
 591     warning_oldhome "${CONFHOME_COMPAT}" "${CONFHOME}"
 592   # root can put profiles under /etc/duply (NEW) if path exists
 593   elif [ -d "${CONFHOME_ETC}" ] && [ "$EUID" -eq 0 ]; then
 594     CONFDIR="${CONFHOME_ETC}/${FTPLCFG}"
 595   # root can keep profiles under /etc/ftplicity (OLD) if path exists
 596   elif [ -d "${CONFHOME_ETC_COMPAT}" ] && [ "$EUID" -eq 0 ]; then
 597     CONFDIR="${CONFHOME_ETC_COMPAT}/${FTPLCFG}"
 598     warning_oldhome "${CONFHOME_ETC_COMPAT}" "${CONFHOME_ETC}"
 599   # hmm no profile folder there, then use default for error later
 600   else
 601     CONFDIR="${CONFHOME}/${FTPLCFG}" # continue, will fail later in main
 602   fi
 603 
 604   # remove trailing slash, get profile name etc.
 605   CONFDIR="${CONFDIR%/}"
 606   PROFILE="${CONFDIR##*/}"
 607   CONF="$CONFDIR/conf"
 608   PRE="$CONFDIR/pre"
 609   POST="$CONFDIR/post"
 610   EXCLUDE="$CONFDIR/exclude"
 611   KEYFILE="$CONFDIR/gpgkey.asc"
 612 }
 613 
 614 function version_info { # print version information
 615   cat <<END
 616   $ME_NAME version $ME_VERSION
 617   ($ME_WEBSITE)
 618 END
 619 }
 620 
 621 function version_info_using { 
 622   cat <<END
 623 $(version_info)
 624 
 625   $(using_info)
 626 END
 627 }
 628 
 629 function using_info {
 630   # init needed vars into global name space
 631   lookup duplicity && { duplicity_version_get; }
 632   local NOTFOUND="INVALID"
 633   local AWK_VERSION GREP_VERSION PYTHON_RUNNER \
 634     PYTHON_RUNNER_RESOLVED PYTHON_VERSION PYTHON_PATH
 635   # freebsd awk / GNU awk (--version), 
 636   # debian mawk (-W version), 
 637   # openbsd awk (-V, exitcode 0 when any program string is given regardless .e.g. "-W version", so place it last) 
 638   # some awks wait for input if they misinterpret/don't know the options, pipe '' as a precaution
 639   AWK_VERSION=$( lookup awk && ( 
 640 	echo | awk --version ||\
 641 	echo | awk -V ||\
 642 	echo | awk -W version ) 2>/dev/null | awk 'NR<=2&&tolower($0)~/(busybox|awk)/{success=1;print;exit} END{if(success<1) print "unknown"}' || echo "$NOTFOUND" )
 643   GREP_VERSION=$( lookup grep && grep --version 2>&1 | awk 'NR<=2&&tolower($0)~/(busybox|grep.*[0-9]+\.[0-9]+)/{success=1;print;exit} END{if(success<1) print "unknown"}' || echo "$NOTFOUND" )
 644 
 645   if [ -n "$PYTHON" ]; then
 646     PYTHON_RUNNER=$PYTHON
 647   else
 648     PYTHON_RUNNER="$(duplicity_python_binary_parse)"
 649   fi
 650   # fetch version and resolve python
 651   [ -n "$PYTHON_RUNNER" ] && {
 652     PYTHON_VERSION=$($PYTHON_RUNNER -V 2>&1| awk '{print tolower($0);exit}' || echo "'$PYTHON_RUNNER' $NOTFOUND" )
 653     local PYTHON_RUNNER_ARRAY=( $PYTHON_RUNNER )
 654     PYTHON_RUNNER_RESOLVED="$(which ${PYTHON_RUNNER_ARRAY[0]})"
 655     # readd params if there were
 656     [ ${#PYTHON_RUNNER_ARRAY[@]} -gt 1 ] && \
 657       PYTHON_RUNNER_RESOLVED="${PYTHON_RUNNER_RESOLVED} ${PYTHON_RUNNER_ARRAY[@]:1}"
 658       PYTHON_PATH="$($PYTHON_RUNNER -c "import sys;print(':'.join(sys.path));")"
 659   }
 660 
 661   local GPG_INFO=$(gpg_avail && gpg --version 2>&1| awk '/^gpg.*[0-9\.]+$/&&length(v)<1{v=$1" "$3}/^Home:/{h=" ("$0")"}END{print v""h}' || echo "gpg $NOTFOUND")
 662   local BASH_VERSION=$(bash --version | awk 'NR==1{IGNORECASE=1;sub(/GNU bash, version[ ]+/,"",$0);print $0}')
 663   # print out
 664   echo -e "Using installed duplicity version ${DUPL_VERSION:-$NOTFOUND}\
 665 ${PYTHON_VERSION+, $PYTHON_VERSION ${PYTHON_RUNNER:+($PYTHON_RUNNER_RESOLVED)}${PYTHON_PATH:+ 'PYTHONPATH=$PYTHON_PATH'}}\
 666 ${GPG_INFO:+, $GPG_INFO}${AWK_VERSION:+, awk '${AWK_VERSION}'}${GREP_VERSION:+, grep '${GREP_VERSION}'}\
 667 ${BASH_VERSION:+, bash '${BASH_VERSION}'}."
 668 }
 669 
 670 function usage_info { # print usage information
 671 
 672   cat <<USAGE_EOF
 673 VERSION:
 674 $(version_info)
 675   
 676 DESCRIPTION: 
 677   Duply deals as a wrapper for the mighty duplicity magic.
 678   It simplifies running duplicity with cron or on command line by:
 679 
 680     - keeping recurring settings in profiles per backup job
 681     - enabling batch operations e.g. backup_verify+purge
 682     - executing pre/post scripts (different actions possible 
 683       depending on previous or next command or it's exit status)
 684     - precondition checking for flawless duplicity operation
 685 
 686   For each backup job one configuration profile must be created.
 687   The profile folder will be stored under '~/.${ME_NAME}/<profile>'
 688   (where ~ is the current users home directory).
 689   Hint:  
 690    If the folder '/etc/${ME_NAME}' exists, the profiles for the super
 691    user root will be searched & created there.
 692 
 693 USAGE:
 694   first time usage (profile creation):  
 695     $ME <profile> create
 696 
 697   general usage in single or batch mode (see EXAMPLES):  
 698     $ME <profile> <command>[[_|+|-]<command>[_|+|-]...] [<options> ...]
 699 
 700   For batches the conditional separators can also be written as pseudo commands
 701   and(+), or(-). See SEPARATORS for details.
 702 
 703   Non $ME options are passed on to duplicity (see OPTIONS).
 704   All conf parameters can also be defined in the environment instead.
 705 
 706 PROFILE:
 707   Indicated by a path or a profile name (<profile>), which is resolved 
 708   to '~/.${ME_NAME}/<profile>' (~ expands to environment variable \$HOME).
 709 
 710   Superuser root can place profiles under '/etc/${ME_NAME}'. Simply create
 711   the folder manually before running $ME_NAME as superuser.
 712   Note:  
 713     Already existing profiles in root's home folder will cease to work
 714     unless they are moved to the new location manually.
 715 
 716   example 1:   $ME humbug backup
 717 
 718   Alternatively a _path_ might be used e.g. useful for quick testing, 
 719   restoring or exotic locations. Shell expansion should work as usual.
 720   Hint:  
 721     The path must contain at least one path separator '/', 
 722     e.g. './test' instead of only 'test'.
 723 
 724   example 2:   $ME ~/.${ME_NAME}/humbug backup
 725 
 726 SEPARATORS:
 727   _ (underscore)  
 728              neutral separator
 729   + (plus sign), _and_  
 730              conditional AND
 731              the next command will only be executed if the previous succeeded
 732   - (minus sign), _or_  
 733              conditional OR
 734              the next command will only be executed if the previous failed
 735   [] (square brackets), _groupIn_/_groupOut_  
 736              enables grouping of commands
 737 
 738    example:  
 739     'pre+[bkp-verify]_post' translates to
 740     'pre_and_groupIn_bkp_or_verify_groupOut_post'
 741 
 742 COMMANDS:
 743   usage      get usage help text
 744 
 745   and/or/groupIn/groupOut  
 746              pseudo commands used in batches (see SEPARATORS above)
 747 
 748   create     creates a configuration profile
 749   backup     backup with pre/post script execution (batch: [pre_bkp_post]),
 750               full (if full_if_older matches or no earlier backup is found)
 751               incremental (in all other cases)
 752   pre/post   execute '<profile>/$(basename "$PRE")', '<profile>/$(basename "$POST")' scripts
 753   bkp        as above but without executing pre/post scripts
 754   full       force full backup
 755   incr       force incremental backup
 756   list [<age>]  
 757              list all files in backup (as it was at <age>, default: now)
 758   status     prints backup sets and chains currently in repository
 759   verify [<age>] [--compare-data]  
 760              list files changed, since age if given
 761   verifyPath <rel_path_in_bkp> <local_path> [<age>] [--compare-data]  
 762              list changes of a file or folder path in backup compared to a
 763              local path, since age if given
 764   restore <target_path> [<age>]  
 765              restore the complete backup to <target_path> [as it was at <age>]
 766   fetch <src_path> <target_path> [<age>]  
 767              fetch single file/folder from backup [as it was at <age>]
 768   purge [<max_age>] [--force]  
 769              list outdated backup files (older than \$MAX_AGE)
 770               [use --force to actually delete these files]
 771   purgeFull [<max_full_backups>] [--force]  
 772              list outdated backup files (\$MAX_FULL_BACKUPS being the number of
 773              full backups and associated incrementals to keep, counting in 
 774              reverse chronological order)
 775               [use --force to actually delete these files]
 776   purgeIncr [<max_fulls_with_incrs>] [--force]  
 777              list outdated incremental backups (\$MAX_FULLS_WITH_INCRS being 
 778              the number of full backups which associated incrementals will be
 779              kept, counting in reverse chronological order) 
 780               [use --force to actually delete these files]
 781   purgeAuto [--force]  
 782              convenience batch wrapper for all purge commands above.
 783              purge, purgeFull, purgeIncr are added if their conf vars were set. e.g.
 784               MAX_AGE=1Y
 785               MAX_FULL_BACKUPS=6
 786               MAX_FULLS_WITH_INCR=3
 787              in profile conf file would result in
 788               [purge_purgeFull_purgeIncr]
 789   cleanup [--force]  
 790              list broken backup chain files archives (e.g. after unfinished run)
 791               [use --force to actually delete these files]
 792 
 793   changelog  print changelog / todo list
 794   txt2man    feature for package maintainers - create a manpage based on the 
 795              usage output. download txt2man from http://mvertes.free.fr/, put 
 796              it in the PATH and run '$ME txt2man' to create a man page.
 797   version    show version information of $ME_NAME and needed programs
 798 
 799 OPTIONS:
 800   --force    passed to duplicity (see commands: 
 801              purge, purgeFull, purgeIncr, cleanup)
 802   --preview  do nothing but print out generated duplicity command lines
 803   --disable-encryption  
 804              disable encryption, overrides profile settings
 805 
 806 TIME FORMATS:
 807   For all time related parameters like age, max_age etc.
 808   Refer to the duplicity manpage for all available formats. Here some examples:
 809     2002-01-25T07:00:00+02:00 (full date time format string)
 810     2002/3/5 (date string YYYY/MM/DD)
 811     12D (interval, 12 days ago)
 812     1h78m (interval, 1 hour 78 minutes ago)
 813 
 814 PRE/POST SCRIPTS:
 815   Some useful internal duply variables are exported to the scripts.
 816 
 817     PROFILE, CONFDIR, SOURCE, TARGET_URL_<PROT|HOSTPATH|USER|PASS>, 
 818     GPG_<KEYS_ENC|KEY_SIGN|PW>, CMD_ERR, RUN_START,
 819     CMD_<PREV|NEXT> (previous/next command), 
 820     CND_<PREV|NEXT> (condition before/after)
 821 
 822   The CMD_* variables were introduced to allow different actions according to 
 823   the command the scripts were attached to e.g. 'pre_bkp_post_pre_verify_post' 
 824   will call the pre script two times, with CMD_NEXT variable set to 'bkp' 
 825   on the first and to 'verify' on the second run.
 826   CMD_ERR holds the exit code of the CMD_PREV .
 827 
 828 EXAMPLES:
 829   create profile 'humbug':  
 830     $ME humbug create (don't forget to edit this new conf file)
 831   backup 'humbug' now:  
 832     $ME humbug backup
 833   list available backup sets of profile 'humbug':  
 834     $ME humbug status
 835   list and delete outdated backups of 'humbug':  
 836     $ME humbug purge --force
 837   restore latest backup of 'humbug' to /mnt/restore:  
 838     $ME humbug restore /mnt/restore
 839   restore /etc/passwd of 'humbug' from 4 days ago to /root/pw:  
 840     $ME humbug fetch etc/passwd /root/pw 4D
 841     (see "duplicity manpage", section TIME FORMATS)
 842   a one line batch job on 'humbug' for cron execution:  
 843     $ME humbug backup_verify_purge --force
 844   batch job to run a full backup with pre/post scripts:  
 845     $ME humbug pre_full_post
 846 
 847 FILES:
 848   in profile folder '~/.${ME_NAME}/<profile>' or '/etc/${ME_NAME}'
 849   conf             profile configuration file
 850   pre,post         pre/post scripts (see above for details)
 851   gpgkey.*.asc     exported GPG key files
 852   exclude          a globbing list of included or excluded files/folders
 853                    (see "duplicity manpage", section FILE SELECTION)
 854 
 855 $(hint_profile)
 856 
 857 SEE ALSO:
 858   duplicity man page duplicity(1) or http://duplicity.us/docs.html
 859 USAGE_EOF
 860 }
 861 
 862 # to check call 'duply txt2man | man -l -'
 863 function usage_txt2man {
 864   usage_info | \
 865   awk '/^^[^[:lower:][:space:]][^[:lower:]]+$/{gsub(/[^[:upper:]]/," ",$0)}{print}' |\
 866   txt2man -t"$(toupper "${ME_NAME}")" -s1 -r"${ME_NAME}-${ME_VERSION}" -v'User Manuals'
 867 }
 868 
 869 function changelog {
 870   cat $ME_LONG | awk '/^#####/{on=on+1}(on==3){sub(/^#(  )?/,"",$0);print}'
 871 }
 872 
 873 function create_config {
 874   if [ ! -d "$CONFDIR" ] ; then
 875     mkdir -p "$CONFDIR" || error "Couldn't create config '$CONFDIR'."
 876   # create initial config file
 877     cat <<EOF >"$CONF"
 878 # gpg encryption settings, simple settings:
 879 #  GPG_KEY='disabled' - disables encryption alltogether
 880 #  GPG_KEY='<key1>[,<key2>]'; GPG_PW='pass' - encrypt with keys,
 881 #   sign if secret key of key1 is available use GPG_PW for sign & decrypt
 882 #  Note: you can specify keys via all methods described in gpg manpage,
 883 #        section "How to specify a user ID", escape commas (,) via backslash (\)
 884 #        e.g. 'Mueller, Horst', 'Bernd' -> 'Mueller\, Horst, Bernd'
 885 #        as they are used to separate the entries
 886 #  GPG_PW='passphrase' - symmetric encryption using passphrase only
 887 GPG_KEY='${DEFAULT_GPG_KEY}'
 888 GPG_PW='${DEFAULT_GPG_PW}'
 889 # gpg encryption settings in detail (extended settings)
 890 #  the above settings translate to the following more specific settings
 891 #  GPG_KEYS_ENC='<keyid1>[,<keyid2>,...]' - list of pubkeys to encrypt to
 892 #  GPG_KEY_SIGN='<keyid1>|disabled' - a secret key for signing
 893 #  GPG_PW='<passphrase>' - needed for signing, decryption and symmetric
 894 #   encryption. If you want to deliver different passphrases for e.g. 
 895 #   several keys or symmetric encryption plus key signing you can use
 896 #   gpg-agent. Simply make sure that GPG_AGENT_INFO is set in environment.
 897 #   also see "A NOTE ON SYMMETRIC ENCRYPTION AND SIGNING" in duplicity manpage 
 898 # notes on en/decryption
 899 #  private key and passphrase will only be needed for decryption or signing.
 900 #  decryption happens on restore and incrementals (compare archdir contents).
 901 #  for security reasons it makes sense to separate the signing key from the
 902 #  encryption keys. https://answers.launchpad.net/duplicity/+question/107216
 903 #GPG_KEYS_ENC='<pubkey1>,<pubkey2>,...'
 904 #GPG_KEY_SIGN='<prvkey>'
 905 # set if signing key passphrase differs from encryption (key) passphrase
 906 # NOTE: available since duplicity 0.6.14, translates to SIGN_PASSPHRASE
 907 #GPG_PW_SIGN='<signpass>'
 908 
 909 # uncomment and set a file path or name force duply to use this gpg executable
 910 # available in duplicity 0.7.04 and above (currently unreleased 06/2015)
 911 #GPG='/usr/local/gpg-2.1/bin/gpg'
 912 
 913 # gpg options passed from duplicity to gpg process (default='')
 914 # e.g. "--trust-model pgp|classic|direct|always" 
 915 #   or "--compress-algo=bzip2 --bzip2-compress-level=9"
 916 #   or "--personal-cipher-preferences AES256,AES192,AES..."
 917 #   or "--homedir ~/.duply" - keep keyring and gpg settings duply specific
 918 #   or "--pinentry-mode loopback" - needed for GPG 2.1+ _and_
 919 #      also enable allow-loopback-pinentry in your .gnupg/gpg-agent.conf
 920 #GPG_OPTS=''
 921 
 922 # disable preliminary tests with the following setting
 923 #GPG_TEST='disabled'
 924 # disable automatic gpg key importing altogether
 925 #GPG_IMPORT='disabled'
 926 # disable automatic gpg key exporting to profile folder
 927 #GPG_EXPORT='disabled'
 928 
 929 # backend, credentials & location of the backup target (URL-Format)
 930 # generic syntax is
 931 #   scheme://[user[:password]@]host[:port]/[/]path
 932 # e.g.
 933 #   sftp://bob:secret@backupserver.com//home/bob/dupbkp
 934 # for details and available backends see duplicity manpage, section URL Format
 935 #   http://duplicity.us/vers8/duplicity.1.html#url-format
 936 # BE AWARE:
 937 #   some backends (cloudfiles, S3 etc.) need additional env vars to be set to
 938 #   work properly, read after the TARGET definition for more details.
 939 # ATTENTION:
 940 #   characters other than A-Za-z0-9.-_.~ in the URL have to be
 941 #   replaced by their url encoded pendants, see
 942 #     http://en.wikipedia.org/wiki/Url_encoding
 943 #   if you define the credentials as TARGET_USER, TARGET_PASS below $ME
 944 #   will try to url_encode them for you if the need arises.
 945 TARGET='${DEFAULT_TARGET}'
 946 
 947 # optionally the username/password can be defined as extra variables
 948 # setting them here _and_ in TARGET results in an error
 949 # ATTENTION:
 950 #   there are backends that do not support the user/pass auth scheme.
 951 #   prominent examples are S3, Azure, Cloudfiles. when in doubt consult the
 952 #   duplicity manpage. usually there is a NOTE section explaining if and which
 953 #   env vars should be set.
 954 #TARGET_USER='${DEFAULT_TARGET_USER}'
 955 #TARGET_PASS='${DEFAULT_TARGET_PASS}'
 956 # e.g. for cloud files backend it might look like this (uncomment for use!)
 957 #export CLOUDFILES_USERNAME='someuser'
 958 #export CLOUDFILES_APIKEY='somekey'
 959 #export CLOUDFILES_AUTHURL ='someurl'
 960 # the following is an incomplete list (<backend>: comma separated env vars list)
 961 # Azure: AZURE_ACCOUNT_NAME, AZURE_ACCOUNT_KEY
 962 # Cloudfiles: CLOUDFILES_USERNAME, CLOUDFILES_APIKEY, CLOUDFILES_AUTHURL
 963 # Google Cloud Storage: GS_ACCESS_KEY_ID, GS_SECRET_ACCESS_KEY
 964 # Pydrive: GOOGLE_DRIVE_ACCOUNT_KEY, GOOGLE_DRIVE_SETTINGS
 965 # S3: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
 966 # Swift: SWIFT_USERNAME, SWIFT_PASSWORD, SWIFT_AUTHURL,
 967 #        SWIFT_TENANTNAME OR SWIFT_PREAUTHURL, SWIFT_PREAUTHTOKEN
 968 
 969 # base directory to backup
 970 SOURCE='${DEFAULT_SOURCE}'
 971 
 972 # a command that runs duplicity e.g. 
 973 #  shape bandwidth use via trickle
 974 #  "trickle -s -u 640 -d 5120" # 5Mb up, 40Mb down"
 975 #DUPL_PRECMD=""
 976 
 977 # override the python interpreter to execute duplicity, unset by default
 978 #  e.g. "python3" or "/usr/bin/python3.8"
 979 #PYTHON="python"
 980 
 981 # exclude folders containing exclusion file (since duplicity 0.5.14)
 982 # Uncomment the following two lines to enable this setting.
 983 #FILENAME='.duplicity-ignore'
 984 #DUPL_PARAMS="\$DUPL_PARAMS --exclude-if-present '\$FILENAME'"
 985 
 986 # Time frame for old backups to keep, Used for the "purge" command.  
 987 # see duplicity man page, chapter TIME_FORMATS)
 988 #MAX_AGE=1M
 989 
 990 # Number of full backups to keep. Used for the "purgeFull" command. 
 991 # See duplicity man page, action "remove-all-but-n-full".
 992 #MAX_FULL_BACKUPS=1
 993 
 994 # Number of full backups for which incrementals will be kept for.
 995 # Used for the "purgeIncr" command.
 996 # See duplicity man page, action "remove-all-inc-of-but-n-full".
 997 #MAX_FULLS_WITH_INCRS=1
 998 
 999 # activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3) 
1000 # forces a full backup if last full backup reaches a specified age, for the 
1001 # format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
1002 # Uncomment the following two lines to enable this setting.
1003 #MAX_FULLBKP_AGE=1M
1004 #DUPL_PARAMS="\$DUPL_PARAMS --full-if-older-than \$MAX_FULLBKP_AGE " 
1005 
1006 # sets duplicity --volsize option (available since v0.4.3.RC7)
1007 # set the size of backup chunks to VOLSIZE MB instead of the default 25MB.
1008 # VOLSIZE must be number of MB's to set the volume size to.
1009 # Uncomment the following two lines to enable this setting. 
1010 #VOLSIZE=50
1011 #DUPL_PARAMS="\$DUPL_PARAMS --volsize \$VOLSIZE "
1012 
1013 # verbosity of output (error 0, warning 1-2, notice 3-4, info 5-8, debug 9)
1014 # default is 4, if not set
1015 #VERBOSITY=5
1016 
1017 # temporary file space. at least the size of the biggest file in backup
1018 # for a successful restoration process. (default is '/tmp', if not set)
1019 #TEMP_DIR=/tmp
1020 
1021 # Modifies archive-dir option (since 0.6.0) Defines a folder that holds 
1022 # unencrypted meta data of the backup, enabling new incrementals without the 
1023 # need to decrypt backend metadata first. If empty or deleted somehow, the 
1024 # private key and it's password are needed.
1025 # NOTE: This is confidential data. Put it somewhere safe. It can grow quite 
1026 #       big over time so you might want to put it not in the home dir.
1027 # default '~/.cache/duplicity/duply_<profile>/'
1028 # if set  '\${ARCH_DIR}/<profile>'
1029 #ARCH_DIR=/some/space/safe/.duply-cache
1030 
1031 # DEPRECATED setting
1032 # sets duplicity --time-separator option (since v0.4.4.RC2) to allow users 
1033 # to change the time separator from ':' to another character that will work 
1034 # on their system.  HINT: For Windows SMB shares, use --time-separator='_'.
1035 # NOTE: '-' is not valid as it conflicts with date separator.
1036 # ATTENTION: only use this with duplicity < 0.5.10, since then default file 
1037 #            naming is compatible and this option is pending depreciation 
1038 #DUPL_PARAMS="\$DUPL_PARAMS --time-separator _ "
1039 
1040 # DEPRECATED setting
1041 # activates duplicity --short-filenames option, when uploading to a file
1042 # system that can't have filenames longer than 30 characters (e.g. Mac OS 8)
1043 # or have problems with ':' as part of the filename (e.g. Microsoft Windows)
1044 # ATTENTION: only use this with duplicity < 0.5.10, later versions default file 
1045 #            naming is compatible and this option is pending depreciation
1046 #DUPL_PARAMS="\$DUPL_PARAMS --short-filenames "
1047 
1048 # more duplicity command line options can be added in the following way
1049 # don't forget to leave a separating space char at the end
1050 #DUPL_PARAMS="\$DUPL_PARAMS --put_your_options_here " 
1051 
1052 EOF
1053 
1054 # create initial exclude file
1055     cat <<EOF >"$EXCLUDE"
1056 # although called exclude, this file is actually a globbing file list
1057 # duplicity accepts some globbing patterns, even including ones here
1058 # here is an example, this incl. only 'dir/bar' except it's subfolder 'foo'
1059 # - dir/bar/foo
1060 # + dir/bar
1061 # - **
1062 # for more details see duplicity manpage, section File Selection
1063 # http://duplicity.us/vers8/duplicity.1.html#file-selection
1064 
1065 EOF
1066 
1067   # Hints on first usage
1068   cat <<EOF
1069 
1070 Congratulations. You just created the profile '$FTPLCFG'.
1071 The initial config file has been created as 
1072 '$CONF'.
1073 You should now adjust this config file to your needs.
1074 
1075 $(hint_profile)
1076 
1077 EOF
1078 fi
1079 
1080 }
1081 
1082 # used in usage AND create_config
1083 function hint_profile {
1084   cat <<EOF
1085 IMPORTANT:
1086   Copy the _whole_ profile folder after the first backup to a safe place.
1087   It contains everything (duply related) needed to restore your backups. 
1088 
1089   Pay attention to (possibly later added) external files such as credentials
1090   or auth files (e.g. netrc, .megarc, ssh keys) or environment variables
1091   (e.g. DPBX_ACCESS_TOKEN). 
1092   It is good policy to place those in the profile folder if possible at all.
1093     e.g. in case of 'multi://' target the config .json file
1094   Env vars should be added to duply profiles' conf file.
1095 
1096   Keep access to these files restricted as they contain information (gpg key,
1097   passphrases etc.) to access and modify your backups.
1098   
1099   Finally:
1100   You should attempt a restore from an unrelated host to be sure you really
1101   have everything needed for restoration.
1102 
1103   Repeat these steps after _all_ configuration changes. Some configuration 
1104   options are crucial for restoration.
1105 
1106 EOF
1107 }
1108 
1109 function separator {
1110   echo "--- $@ ---"
1111 }
1112 
1113 function inform {
1114   echo -e "\nINFO:\n\n$@\n"
1115 }
1116 
1117 function warning {
1118   echo -e "\nWARNING:\n\n$@\n"
1119 }
1120 
1121 function warning_oldhome {
1122   local old=$1 new=$2
1123   warning " ftplicity changed name to duply since you created your profiles.
1124   Please rename the old folder
1125   '$old'
1126   to
1127   '$new'
1128   and this warning will disappear.
1129   If you decide not to do so profiles will _only_ work from the old location."
1130 }
1131 
1132 function error_print {
1133   echo -e "$@" >&2
1134 }
1135 
1136 function error {
1137   error_print "\nSorry. A fatal ERROR occured:\n\n$@\n"
1138   exit -1
1139 }
1140 
1141 function error_gpg {
1142   [ -n "$2" ] && local hint="\n  $2\n\n  "
1143   
1144   error "$1
1145 
1146 Hint${hint:+s}:
1147   ${hint}Maybe you have not created a gpg key yet (e.g. gpg --gen-key)?
1148   Don't forget the used _password_ as you will need it.
1149   When done enter the 8 digit id & the password in the profile conf file.
1150 
1151   The key id can be found doing a 'gpg --list-keys'. In the example output 
1152   below the key id for the public key would be FFFFFFFF.
1153 
1154   pub   1024D/FFFFFFFF 2007-12-17
1155   uid                  duplicity
1156   sub   2048g/899FE27F 2007-12-17
1157 "
1158 }
1159 
1160 function error_gpg_test {
1161   [ -n "$2" ] && local hint="\n  $2\n\n  "
1162 
1163   error "$1
1164 
1165 Hint${hint:+s}:
1166   ${hint}This error means that gpg is probably misconfigured or not working 
1167   correctly. The error message above should help to solve the problem.
1168   However, if for some reason $ME_NAME should misinterpret the situation you 
1169   can define GPG_TEST='disabled' in the conf file to bypass the test.
1170   Please do not forget to report the bug in order to resolve the problem
1171   in future versions of $ME_NAME.
1172 "
1173 }
1174 
1175 function error_path {
1176   error "$@
1177 PATH='$PATH'
1178 "
1179 }
1180 
1181 function error_to_string {
1182   [ -n "$1" ] && [ "$1" -eq 0 ] && echo "OK" || echo "FAILED 'code $1'"
1183 }
1184 
1185 function duplicity_version_get {
1186   # use cached value, just print
1187   var_isset DUPL_VERSION && return
1188 
1189   local DUPL_VERSION_OUT DUPL_VERSION_AWK PYTHON_BIN CMD='duplicity'
1190   # only run with a user specific python if configured (running by default
1191   # breaks homebrew as they place a shell wrapper for duplicity in path)
1192   [ -n "$PYTHON" ] &&\
1193     CMD="$PYTHON $(qw "$(which duplicity)")"
1194 
1195   DUPL_VERSION_OUT=$($CMD --version)
1196   DUPL_VERSION=$(echo $DUPL_VERSION_OUT | awk '/^duplicity/{print $2; exit;}')
1197   #DUPL_VERSION='1.2.3' #'0.7.03' #'0.6.08b' #,0.4.4.RC4,0.6.08b
1198   DUPL_VERSION_VALUE=0
1199   DUPL_VERSION_AWK=$(awk -v v="$DUPL_VERSION" 'BEGIN{
1200   if (match(v,/[^\.0-9]+[0-9]*$/)){
1201     rest=substr(v,RSTART,RLENGTH);v=substr(v,0,RSTART-1);}
1202   if (pos=match(rest,/RC([0-9]+)$/)) rc=substr(rest,pos+2)
1203   split(v,f,"[. ]"); if(f[1]f[2]f[3]~/^[0-9]+$/) vvalue=f[1]*10000+f[2]*100+f[3]; else vvalue=0
1204   print "#"v"_"rest"("rc"):"f[1]"-"f[2]"-"f[3]
1205   print "DUPL_VERSION_VALUE=\047"vvalue"\047"
1206   print "DUPL_VERSION_RC=\047"rc"\047"
1207   print "DUPL_VERSION_SUFFIX=\047"rest"\047"
1208   }')
1209   eval "$DUPL_VERSION_AWK"
1210   #echo -e ",$DUPL_VERSION,$DUPL_VERSION_VALUE,$DUPL_VERSION_RC,$DUPL_VERSION_SUFFIX,"
1211 
1212   # doublecheck findings and report error
1213   if [ $DUPL_VERSION_VALUE -eq 0 ]; then
1214     inform "duplicity version check failed (please report, this is a bug)
1215 the command
1216   $CMD
1217 resulted in
1218   $DUPL_VERSION_OUT
1219 "
1220   elif [ $DUPL_VERSION_VALUE -le 404 ] && [ ${DUPL_VERSION_RC:-4} -lt 4 ]; then
1221     error "The installed version $DUPL_VERSION is incompatible with $ME_NAME v$ME_VERSION.
1222 You should upgrade your version of $ME_NAME to at least v0.4.4RC4 or
1223 use the older ftplicity version 1.1.1 from $ME_WEBSITE."
1224   elif [ $DUPL_VERSION_VALUE -le 20100 ] ; then
1225     error "The installed version $DUPL_VERSION is incompatible with $ME_NAME v$ME_VERSION.
1226 You should upgrade your version of duplicity to at least v2.1.0 or
1227 use the older $ME_NAME version 2.4.3 from $ME_WEBSITE."
1228   fi
1229 }
1230 
1231 function duplicity_version_ge {
1232   [ "$DUPL_VERSION_VALUE" -ge "$1" ]
1233 }
1234 
1235 function duplicity_version_lt {
1236   ! duplicity_version_ge "$1"
1237 }
1238 
1239 # parse interpreter from duplicity shebang
1240 function duplicity_python_binary_parse {
1241   # reuse cached result
1242   var_isset 'DUPL_PYTHON_BIN' && {
1243     [ -n "$DUPL_PYTHON_BIN" ] && {
1244       echo $DUPL_PYTHON_BIN
1245       return
1246     } || return 1
1247   }
1248 
1249   # no duplicity, nothing to parse
1250   lookup duplicity || return 1
1251 
1252   local DUPL_BIN=$(which duplicity)
1253   # test for shebang
1254   IFS= LC_ALL=C read -rn2 shebang < "$DUPL_BIN" && [ "$shebang" != '#!' ] && {
1255     DUPL_PYTHON_BIN=""
1256     return 1
1257   }
1258 
1259   # parse it or warn
1260   DUPL_PYTHON_BIN=$(awk 'NR==1&&/^#!/{sub(/^#!( *\/usr\/bin\/env *)?/,""); print}' < "$DUPL_BIN")
1261   if ! echo "$DUPL_PYTHON_BIN" | grep -q -i 'python'; then
1262     warning "Could not parse the python interpreter used from duplicity ($DUPL_BIN). Result was 
1263 '$DUPL_PYTHON_BIN'.
1264 "
1265     DUPL_PYTHON_BIN=""
1266     return 1
1267   fi
1268   
1269   # success
1270   echo $DUPL_PYTHON_BIN
1271   return
1272 }
1273 
1274 function run_script { # run pre/post scripts
1275   local ERR=0
1276   local SCRIPT="$1"
1277   if [ ! -z "$PREVIEW" ] ; then
1278     echo "$([ ! -x "$SCRIPT" ] && echo ". ")$SCRIPT"
1279   elif [ -r "$SCRIPT" ] ; then 
1280     echo -n "Running '$SCRIPT' "
1281     if [ -x "$SCRIPT" ]; then
1282       OUT=$("$SCRIPT" 2>&1)
1283       ERR=$?
1284     else
1285       OUT=$(. "$SCRIPT" 2>&1)
1286       ERR=$?
1287     fi
1288     [ $ERR -eq "0" ] && echo "- OK" || echo "- FAILED (code $ERR)"
1289     echo -en ${OUT:+"Output: $OUT\n"} ;
1290   else
1291     echo "Skipping n/a script '$SCRIPT'."
1292   fi
1293   return $ERR
1294 }
1295 
1296 function run_cmd {
1297   # run or print escaped cmd string
1298   local CMD_ERR=0
1299   if [ -n "$PREVIEW" ]; then
1300     CMD_OUT=$( echo "$@ 2>&1" )
1301     CMD_MSG="-- Run cmd -- $CMD_MSG --\n$CMD_OUT"
1302   elif [ -n "$CMD_DISABLED" ]; then
1303     CMD_MSG="$CMD_MSG (DISABLED) - $CMD_DISABLED"
1304   else
1305     echo -n -e "$CMD_MSG"
1306     CMD_OUT=` eval "$@" 2>&1 `
1307     CMD_ERR=$?
1308     if [ "$CMD_ERR" = "0" ]; then
1309       CMD_MSG=" (OK)"
1310     else
1311       CMD_MSG=" (FAILED)"
1312     fi
1313   fi
1314   echo -e "$CMD_MSG"
1315   # reset
1316   unset CMD_DISABLED CMD_MSG
1317   return $CMD_ERR
1318 }
1319 
1320 function qw { quotewrap $@; }
1321 
1322 function quotewrap {
1323   local param="$@"
1324   # quote strings having non word chars (e.g. spaces)
1325   if echo "$param" | awk '/[^a-zA-Z0-9,\._\+\-@%\/]/{exit 0}{exit 1}'; then
1326     echo "$param" | awk '{\
1327       gsub(/[\047]/,"\047\\\047\047",$0);\
1328       gsub(/[\042]/,"\047\\\042\047",$0);\
1329       print "\047"$0"\047"}'
1330     return
1331   fi
1332   echo $param
1333 }
1334 
1335 function duplicity_params_global {
1336   # already done? return
1337   var_isset 'DUPL_PARAMS_GLOBAL' && return
1338   local DUPL_ARG_ENC
1339 
1340   # use key only if set in config, else leave it to symmetric encryption
1341   if gpg_disabled; then
1342     local DUPL_PARAM_ENC='--no-encryption'
1343   else
1344     local DUPL_PARAM_ENC=$(gpg_prefix_keyset '--encrypt-key' "${GPG_KEYS_ENC_ARRAY[@]}")
1345     gpg_signing && local DUPL_PARAM_SIGN=$(gpg_prefix_keyset '--sign-key' "$GPG_KEY_SIGN")
1346     # interpret password settings
1347     var_isset 'GPG_PW' && DUPL_ARG_ENC="PASSPHRASE=$(qw "${GPG_PW}")"
1348     var_isset 'GPG_PW_SIGN' && DUPL_ARG_ENC="${DUPL_ARG_ENC} SIGN_PASSPHRASE=$(qw "${GPG_PW_SIGN}")"
1349   fi
1350 
1351   local GPG_OPTS=${GPG_OPTS:+"--gpg-options $(qw "${GPG_OPTS}")"}
1352 
1353   # set name for dupl archive folder, since 0.6.0
1354   if duplicity_version_ge 601; then
1355     local DUPL_ARCHDIR=''
1356     if var_isset 'ARCH_DIR'; then
1357       DUPL_ARCHDIR="--archive-dir $(qw "${ARCH_DIR}")"
1358       # reuse erronously duply_ prefixed folders from bug #117
1359       if [ -d "$ARCH_DIR/duply_${PROFILE}" ]; then
1360         DUPL_ARCHDIR="${DUPL_ARCHDIR} --name $(qw "duply_${PROFILE}")"
1361       else
1362         DUPL_ARCHDIR="${DUPL_ARCHDIR} --name $(qw "${PROFILE}")"
1363       fi
1364     else
1365       DUPL_ARCHDIR="--name $(qw "duply_${PROFILE}")"
1366     fi
1367   fi
1368 
1369 DUPL_PARAMS_GLOBAL="${DUPL_ARCHDIR} ${DUPL_PARAM_ENC}\
1370  ${DUPL_PARAM_SIGN} ${VERBOSITY:+--verbosity $VERBOSITY}\
1371  ${GPG_OPTS}"
1372 
1373 DUPL_VARS_GLOBAL="TMPDIR='$TEMP_DIR'\
1374  ${DUPL_ARG_ENC}"
1375 }
1376 
1377 # function to filter the DUPL_PARAMS var from user conf
1378 function duplicity_params_conf {
1379   local OUT="$DUPL_PARAMS"
1380   # reuse cmd var from main loop
1381   ## in/exclude parameters are currently not supported on 
1382   ## cleanup, status (collection_status), list (list_current_files), purge* (remove_*), fetch/restore
1383   case $cmd in
1384     cleanup | status | list | purge* | restore | fetch )
1385       # filter exclude params from fetch/restore/status
1386       OUT=$(eval stripXcludes $OUT)
1387     ;;
1388   esac
1389 
1390   case $cmd in
1391     bkp | incr | full )
1392       # nothing to strip, we're backing up'
1393     ;;
1394     *)
1395       OUT=$(eval stripBkpOnlyParams $OUT)
1396     ;;
1397   esac
1398 
1399   # print result
1400   echo "$OUT"
1401 }
1402 
1403 # strip in/exclude parameters from param string
1404 function stripXcludes {
1405   local STRIPNEXT OUT;
1406   for p in "$@"; do
1407     #echo "calc =$p="
1408     if [ -n "$STRIPNEXT" ]; then
1409       unset STRIPNEXT
1410       # strip the value of previous parameter
1411       continue
1412     elif echo "$p" | awk '/^\-\-(in|ex)clude(\-[a-zA-Z\-]+)?$/{exit 0;}{exit 1;}'; then
1413       # strips e.g. --include /foo/bar
1414       STRIPNEXT="yes"
1415       continue
1416     elif echo "$p" | awk '/^\-\-(in|ex)clude(\-[a-zA-Z\-]+)?=/{exit 0;}{exit 1;}'; then
1417       # strips e.g. --include=/foo/bar
1418       continue
1419     fi
1420 
1421     OUT="$OUT $(qw "$p")"
1422   done
1423   echo "$OUT"
1424 }
1425 
1426 # strip backup only parameters from param string
1427 function stripBkpOnlyParams {
1428   local STRIPNEXT OUT;
1429 
1430   for p in "$@"; do
1431     if [ -n "$STRIPNEXT" ]; then
1432       unset STRIPNEXT
1433       # strip the value of previous parameter
1434       continue
1435     elif echo "$p" | awk '/^\-\-(allow-source-mismatch|asynchronous-upload|dry-run)$/{exit 0;}{exit 1;}'; then
1436       continue
1437     elif echo "$p" | awk '/^\-\-(volsize)$/{exit 0;}{exit 1;}'; then
1438       # strips e.g. --volsize 100
1439       STRIPNEXT="yes"
1440       continue
1441     elif echo "$p" | awk '/^\-\-volsize=/{exit 0;}{exit 1;}'; then
1442       # strips e.g. --volsize=100
1443       continue
1444     fi
1445 
1446     OUT="$OUT $(qw "$p")"
1447   done
1448   echo "$OUT"
1449 }
1450 
1451 function duplify { # the actual wrapper function
1452   local PARAMSNOW DUPL_CMD DUPL_CMD_PARAMS
1453 
1454   # put command (with params) first in duplicity parameters
1455   for param in "$@" ; do
1456   # split cmd from params (everything before splitchar --)
1457     if [ "$param" == "--" ] ; then
1458       PARAMSNOW=1
1459     else
1460       # wrap in quotes to protect from spaces
1461       [ ! $PARAMSNOW ] && \
1462         DUPL_CMD="$DUPL_CMD $(qw "$param")" \
1463       || \
1464         DUPL_CMD_PARAMS="$DUPL_CMD_PARAMS $(qw "$param")"
1465     fi
1466   done
1467 
1468   # init global duplicity parameters same for all tasks
1469   duplicity_params_global
1470 
1471   local RUN=eval CMD=duplicity
1472   # run in cmd line preview mode if requested
1473   var_isset 'PREVIEW' && RUN=echo
1474   # only run with a user specific python if configured (running by default
1475   # breaks homebrew as they place a shell wrapper for duplicity in path)
1476   # resolve duplicity path for usage with python interpreter
1477   [ -n "$PYTHON" ] &&\
1478     CMD="$PYTHON $(qw "$(which duplicity)")"
1479 
1480 $RUN "${DUPL_VARS_GLOBAL} ${BACKEND_PARAMS}\
1481  ${DUPL_PRECMD} $CMD $DUPL_CMD $DUPL_PARAMS_GLOBAL $(duplicity_params_conf)\
1482  $GPG_USEAGENT $(gpg_custom_binary) $DUPL_CMD_PARAMS"
1483 
1484   local ERR=$?
1485   return $ERR
1486 }
1487 
1488 function secureconf { # secure the configuration dir
1489   #PERMS=$(ls -la $(dirname $CONFDIR) | grep -e " $(basename $CONFDIR)\$" | awk '{print $1}')
1490   local PERMS="$(ls -la "$CONFDIR/." | awk 'NR==2{print $1}')"
1491   if [ "${PERMS/#drwx------*/OK}" != 'OK' ] ; then
1492     chmod u+rwX,go= "$CONFDIR"; local ERR=$?
1493     warning "The profile's folder 
1494 '$CONFDIR'
1495 permissions are not safe ($PERMS). Secure them now. - ($(error_to_string $ERR))"
1496   fi
1497 }
1498 
1499 # params are $1=timeformatstring (default like date output), $2=epoch seconds since 1.1.1970 (default now)
1500 function date_fix {
1501   local DEFAULTFORMAT='%a %b %d %H:%M:%S %Z %Y'
1502   local date
1503   #[ "$1" == "%N" ] && return #test the no nsec test below
1504   # gnu date with -d @epoch
1505   date=$(date ${2:+-d @$2} ${1:++"$1"} 2> /dev/null) && \
1506     echo $date && return
1507   # date bsd,osx with -r epoch
1508   date=$(date ${2:+-r $2} ${1:++"$1"} 2> /dev/null) && \
1509     echo $date && return
1510   # date busybox with -d epoch -D %s
1511   date=$(date ${2:+-d $2 -D %s} ${1:++"$1"} 2> /dev/null) && \
1512     echo $date && return
1513   ## some date commands do not support giving a time w/o setting it systemwide (irix,solaris,others?)
1514   # python fallback
1515   #date=$("$(python_binary)" -c "import time;print time.strftime('${1:-$DEFAULTFORMAT}',time.localtime(${2}))" 2> /dev/null) && \
1516   #  echo $date && return
1517   # awk fallback
1518   date=$(awk "BEGIN{print strftime(\"${1:-$DEFAULTFORMAT}\"${2:+,$2})}" 2> /dev/null) && \
1519     echo $date && return
1520   # perl fallback
1521   date=$(perl  -e "use POSIX qw(strftime);\$date = strftime(\"${1:-$DEFAULTFORMAT}\",localtime(${2}));print \"\$date\n\";" 2> /dev/null) && \
1522     echo $date && return
1523   # error
1524   echo "ERROR"
1525   return 1
1526 }
1527 
1528 function nsecs {
1529   local NSECS
1530   # test if date supports nanosecond output
1531   if ! var_isset NSECS_DISABLED; then
1532     NSECS=$(date_fix %N 2> /dev/null | head -1 |grep -e "^[[:digit:]]\{9\}$")
1533     [ -n "$NSECS" ] && NSECS_DISABLED=0 || NSECS_DISABLED=1
1534   fi
1535 
1536   # add 9 digits, not all date(s) deliver nsecs e.g. busybox date
1537   if [ "$NSECS_DISABLED" == "1" ]; then
1538     date_fix %s000000000
1539   else
1540     date_fix %s%N
1541   fi
1542 }
1543 
1544 function nsecs_to_sec {
1545   echo $(($1/1000000000)).$(printf "%03d" $(($1/1000000%1000)) )
1546 }
1547 
1548 function datefull_from_nsecs {
1549   date_from_nsecs $1 '%F %T'
1550 }
1551 
1552 function date_from_nsecs {
1553   local FORMAT=${2:-%T}
1554   local TIME=$(nsecs_to_sec $1)
1555   local SECS=${TIME%.*}
1556   local DATE=$(date_fix "${FORMAT}" ${SECS:-0})
1557   echo $DATE.${TIME#*.}
1558 }
1559 
1560 function var_isset {
1561   if [ -z "$1" ]; then
1562     echo "ERROR: function var_isset needs a string as parameter"
1563   elif eval "[ \"\${$1}\" == 'not_set' ]" || eval "[ \"\${$1-not_set}\" != 'not_set' ]"; then
1564     return 0
1565   fi
1566   return 1
1567 }
1568 
1569 function is_condition {
1570   local CMD=$(tolower "$1")
1571   [ "$CMD" == 'and' ] || [ "$CMD" == 'or' ]
1572 }
1573 
1574 function is_groupMarker {
1575   local CMD=$(tolower "$1")
1576   [ "$CMD" == 'groupin' ] || [ "$CMD" == 'groupout' ]
1577 }
1578 
1579 function is_command {
1580   local CMD=$(tolower "$1")
1581   ! is_condition "$CMD" && ! is_groupMarker "$CMD"
1582 }
1583 
1584 function url_encode {
1585   # utilize python, silently do nothing on error - because no python no duplicity
1586   local PYTHON_RUNNER
1587   PYTHON_RUNNER="$(duplicity_python_binary_parse)" ||\
1588   PYTHON_RUNNER="python" &&\
1589     OUT=$($PYTHON_RUNNER -c "
1590 try: import urllib.request as urllib
1591 except ImportError: import urllib
1592 print(urllib.${2}quote('$1'));
1593 " 2>/dev/null ); ERR=$?
1594   [ "$ERR" -eq 0 ] && echo $OUT || echo $1
1595 }
1596 
1597 function url_decode {
1598   # reuse function above with a simple string param hack
1599   url_encode "$1" "un"
1600 }
1601 
1602 function toupper {
1603   echo "$@"|awk '$0=toupper($0)'
1604 }
1605 
1606 function tolower {
1607   echo "$@"|awk '$0=tolower($0)'
1608 }
1609 
1610 function isnumber {
1611   case "$*" in
1612     ''|*[!0-9]*) return 1;;
1613     *) return 0;;
1614   esac
1615 }
1616 
1617 #function tmp_space {
1618 #  
1619 #  if ! isnumber $VOLSIZE; then
1620 #    inform "failed to determine free space (please report, this is a bug)"
1621 #    return
1622 #  fi
1623 #  
1624 # get free temp space
1625 #  TEMP_FREE="$(df -P -k "$TEMP_DIR" 2>/dev/null | awk 'END{pos=(NF-2);if(pos>0) print $pos;}')"
1626 #  # check for free space or FAIL
1627 #  if [ $((${TEMP_FREE:-0}-${VOLSIZE:-0}*1024)) -lt 0-lt 0 ]; then
1628 #    error "Temporary file space '$TEMP_DIR' free space is smaller ($((TEMP_FREE/1024))MB)
1629 #than one duplicity volume (${VOLSIZE}MB).
1630 #    
1631 #  Hint: Free space or change TEMP_DIR setting."
1632 #fi
1633 #
1634 #}
1635 
1636 function gpg_disabled {
1637   echo "${GPG_KEY}" | grep -iq -e '^disabled$'
1638 }
1639 
1640 # usage: join SEPARATOR "entry1" "entry2"
1641 function join {
1642   local SEP="$1" ENTRY OUT; shift;
1643   for ENTRY in "$@"; do
1644     ENTRY=${ENTRY//$SEP/\\$SEP}
1645     [ -z "$OUT" ] && OUT=$ENTRY || OUT="$OUT$SEP$ENTRY"
1646   done
1647   echo $OUT
1648 }
1649 
1650 function gpg_testing {
1651   [ "$GPG_TEST" != "disabled" ]
1652 }
1653 
1654 function gpg_signing {
1655   echo ${GPG_KEY_SIGN} | grep -v -q -e '^disabled$'
1656 }
1657 
1658 function gpg_keytype {
1659   echo "$1" | awk '/^PUB$/{print "public"}/^SEC$/{print "secret"}'
1660 }
1661 
1662 # parameter key id, key_type
1663 function gpg_keyfile {
1664   local GPG_KEY=$(gpg_key_legalize $1) TYPE="$2"
1665   local KEYFILE="${KEYFILE//.asc/${GPG_KEY:+.$GPG_KEY}.asc}"
1666   echo "${KEYFILE//.asc/${TYPE:+.$(tolower $TYPE)}.asc}"
1667 }
1668 
1669 # parameter key id
1670 function gpg_import {
1671   local i FILE FOUND=0 KEY_ID="$1" KEY_TYPE="$2" KEY_FP="" ERR=0
1672   [ "$GPG_IMPORT" = "disabled" ] && {
1673     echo "Skipping import of needed $(gpg_keytype "$KEY_TYPE") key '$KEY_ID'. (GPG_IMPORT='disabled')"
1674     return
1675   }
1676 
1677   # create a list of legacy key file names and current naming scheme
1678   # we always import pub and sec if they are avail in conf folder
1679   local KEYFILES=( "$CONFDIR/gpgkey" $(gpg_keyfile "$KEY_ID") \
1680                    $(gpg_keyfile "$KEY_ID" "$KEY_TYPE") )
1681 
1682   # Try autoimport from existing old gpgkey files 
1683   # and new gpgkey.XXX.asc files (since v1.4.2)
1684   # and even newer gpgkey.XXX.[pub|sec].asc
1685   for (( i = 0 ; i < ${#KEYFILES[@]} ; i++ )); do
1686     FILE=${KEYFILES[$i]}
1687     if [ -f "$FILE" ]; then
1688       FOUND=1
1689       
1690       CMD_MSG="Import keyfile '$FILE' to keyring"
1691       run_cmd gpg $GPG_OPTS --batch --import $(qw "$FILE")
1692       if [ "$?" != "0" ]; then 
1693         warning "Import failed.${CMD_OUT:+\n$CMD_OUT}"
1694         ERR=1
1695         # continue with next
1696         continue
1697       fi
1698     fi
1699   done
1700 
1701   if [ "$FOUND" -eq 0 ]; then
1702     echo "Notice: No keyfile for '$KEY_ID' found in profile folder."
1703     return 1
1704   fi
1705 
1706   # try to set trust automagically
1707   CMD_MSG="Autoset trust of key '$KEY_ID' to ultimate"
1708   run_cmd echo $(gpg_fingerprint "$KEY_ID"):6: \| gpg $GPG_OPTS --import-ownertrust --batch --logger-fd 1
1709   if [ "$?" = "0" ] && [ -z "$PREVIEW" ]; then 
1710    # success on all levels, we're done
1711    return $ERR
1712   fi
1713 
1714   # failover: user has to set trust manually
1715   echo -e "For $ME_NAME to work you have to set the trust level 
1716 with the command \"trust\" to \"ultimate\" (5) now.
1717 Exit the edit mode of gpg with \"quit\"."
1718   CMD_MSG="Running gpg to manually edit key '$KEY_ID'"
1719   run_cmd sleep 5\; gpg $GPG_OPTS --edit-key $(qw "$KEY_ID")
1720 
1721   return $ERR
1722 }
1723 
1724 # see 'How to specify a user ID' on gpg manpage
1725 function gpg_fingerprint {
1726   gpg $GPG_OPTS --fingerprint "$1" 2>&1 | \
1727   awk 'NR==2{sub(/^.*=/,"");gsub(/[ \t]/,""); if ( $0 !~ /^[A-F0-9]+$/ || length($0) != 40 ) exit 1; print}'
1728 }
1729 
1730 function gpg_export_if_needed {
1731   [ "$GPG_EXPORT" = 'disabled' ] && { \
1732     echo "Skipping export of gpg keys. (GPG_EXPORT='disabled')"
1733     return
1734   }
1735 
1736   local SUCCESS FILE KEY_TYPE
1737   local TMPFILE="$TEMP_DIR/${ME_NAME}.$$.$(date_fix %s).gpgexp"
1738   for KEY_ID in "$@"; do
1739     # check if already exported, do it if not
1740     for KEY_TYPE in PUB SEC; do
1741       FILE="$(gpg_keyfile "$KEY_ID" $KEY_TYPE)"
1742       if [ ! -f "$FILE" ] && eval gpg_$(tolower $KEY_TYPE)_avail \"$KEY_ID\"; then
1743 
1744         # exporting
1745         CMD_MSG="Backup $(gpg_keytype "$KEY_TYPE") key '$KEY_ID' to profile."
1746         # gpg2.1 insists on passphrase here, gpg2.0- happily exports w/o it
1747         # we pipe an empty string when GPG_PW is not set to avoid gpg silently waiting for input
1748         run_cmd $(gpg_pass_pipein GPG_PW_SIGN GPG_PW) gpg $GPG_OPTS $GPG_USEAGENT $(gpg_param_passwd GPG_PW_SIGN GPG_PW) --armor --export"$(test "SEC" = "$KEY_TYPE" && echo -secret-keys)" $(qw "$KEY_ID") '>>' $(qw "$TMPFILE")
1749         CMD_ERR=$?
1750 
1751         if [ "$CMD_ERR" = "0" ]; then
1752           CMD_MSG="Write file '"$(basename "$FILE")"'"
1753           run_cmd mv $(qw "$TMPFILE") $(qw "$FILE")
1754         fi
1755 
1756         if [ "$CMD_ERR" != "0" ]; then
1757           warning "Backup failed.${CMD_OUT:+\n$CMD_OUT}"
1758         else
1759           SUCCESS=1
1760         fi
1761 
1762         # cleanup
1763         rm $(qw "$TMPFILE") 1>/dev/null 2>&1
1764       fi
1765     done
1766   done
1767   
1768   [ -n "$SUCCESS" ] && inform "$ME_NAME exported new keys to your profile.
1769 You should backup your changed profile folder now and store it in a safe place."
1770 }
1771 
1772 # replace all non-alnum chars with underscore (for file operations)
1773 function gpg_key_legalize {
1774   echo $* | awk '{gsub(/[^a-zA-Z0-9]/,"_",$0); print}'
1775 }
1776 
1777 function gpg_key_cache {
1778   local RES
1779   local MODE=$1
1780   shift
1781   local PREFIX="GPG_KEY"
1782   local SUFFIX=$(gpg_key_legalize "$@")
1783   local KEYID="$*"
1784   local CACHE="${PREFIX}_${MODE}_${SUFFIX}"
1785   if [ "$MODE" = "RESET" ]; then
1786     eval unset ${PREFIX}_PUB_$SUFFIX ${PREFIX}_SEC_$SUFFIX
1787     return 255
1788   elif ! var_isset "$CACHE"; then
1789     if [ "$MODE" = "PUB" ]; then
1790       RES=$(gpg $GPG_OPTS --list-key "$KEYID" > /dev/null 2>&1; echo -n $?)
1791     elif [ "$MODE" = "SEC" ]; then
1792       RES=$(gpg $GPG_OPTS --list-secret-key "$KEYID" > /dev/null 2>&1; echo -n $?)
1793     else
1794       return 255
1795     fi
1796     eval $CACHE=$RES
1797   fi
1798   eval return \$$CACHE
1799 }
1800 
1801 function gpg_pub_avail {
1802   gpg_key_cache PUB "$@"
1803 }
1804 
1805 function gpg_sec_avail {
1806   gpg_key_cache SEC "$@"
1807 }
1808 
1809 function gpg_key_format {
1810   echo $1 | grep -q '^[0-9a-fA-F]\{8\}$'
1811 }
1812 
1813 # splits a comma separated line into lines, respects escaped commas
1814 function gpg_split_keyset {
1815   local LIST
1816   LIST=$(echo "$@" | awk '{ gsub(/,/,"\n",$0); gsub(/\\\n/,",",$0); print $0 }')
1817   echo -e "$LIST"
1818 }
1819 
1820 function gpg_prefix_keyset {
1821   local PREFIX="$1" OUT=""
1822   shift
1823   for KEY_ID in "$@"; do
1824     OUT="${OUT} $PREFIX $(qw ${KEY_ID})"
1825   done
1826   echo $OUT
1827 }
1828 
1829 # grep a variable from conf text file (currently not used)
1830 function gpg_passwd {
1831   [ -r "$CONF" ] && \
1832   awk '/^[ \t]*GPG_PW[ \t=]/{\
1833         sub(/^[ \t]*GPG_PW[ \t]*=*/,"",$0);\
1834         gsub(/^[ \t]*[\047"]|[\047"][ \t]*$/,"",$0);\
1835         print $0; exit}' "$CONF"
1836 }
1837 
1838 # return success if at least one secret key is available
1839 function gpg_key_decryptable {
1840   # no keys, no problem
1841   gpg_symmetric && return 0
1842 
1843   local KEY_ID
1844   for KEY_ID in "${GPG_KEYS_ENC_ARRAY[@]}"; do
1845     gpg_sec_avail "$KEY_ID" && return 0
1846   done
1847   return 1
1848 }
1849 
1850 function gpg_symmetric {
1851   [ -z "${GPG_KEY}${GPG_KEYS_ENC_ARRAY}" ]
1852 }
1853 
1854 # checks for max two params if they are set, typically GPG_PW & GPG_PW_SIGN
1855 function gpg_param_passwd {
1856   var_isset GPG_USEAGENT && exit 1
1857   
1858   if ( [ -n "$1" ] && var_isset "$1" ) || ( [ -n "$2" ] && var_isset "$2" ); then
1859     echo "--passphrase-fd 0 --batch"
1860   fi
1861 }
1862 
1863 # select the earliest defined and create an "echo <value> |" string
1864 function gpg_pass_pipein {
1865   var_isset GPG_USEAGENT && exit 1
1866   
1867   for var in "$@"
1868   do
1869     if var_isset "$var"; then
1870       echo "echo $(qw $(eval echo \$$var)) |"
1871       return 0
1872     fi
1873   done
1874   
1875   return 1
1876 }
1877 
1878 # checks if gpg-agent is available, returns error code
1879 # 0 on success
1880 # 1 if GPG_AGENT_INFO is not set (unused, should probably be merged w/ 3)
1881 # 2 if GPG_AGENT_INFO is stale
1882 # 3 cannot connect to gpg-agent
1883 function gpg_agent_avail {
1884   # GPG_AGENT_INFO is deprecated in gpg2.1, 
1885   # first try to connect to a possibly running agent here
1886   local ERR=3
1887   gpg-agent > /dev/null 2>&1 && return 0
1888 
1889   # detect stale pid in legacy GPG_AGENT_INFO env var
1890   if var_isset GPG_AGENT_INFO; then
1891     # check if a pid matching process is running at all
1892     local GPG_AGENT_PID=$(echo $GPG_AGENT_INFO|awk -F: '{print $2}')
1893     if isnumber "$GPG_AGENT_PID"; then
1894       ps -p "$GPG_AGENT_PID" > /dev/null 2>&1 || ERR=2
1895     fi
1896   fi
1897 
1898   return $ERR
1899 }
1900 
1901 function gpg_custom_binary {
1902   var_isset GPG && [ "$GPG" != "$DEFAULT_GPG" ] &&\
1903     echo "--gpg-binary $(qw "$GPG")"
1904 }
1905 
1906 function gpg_binary {
1907   local BIN
1908   var_isset GPG && BIN="$GPG" || BIN="$DEFAULT_GPG"
1909   echo "$BIN"
1910 }
1911 
1912 function gpg_avail {
1913   lookup "$(gpg_binary)"
1914 }
1915 
1916 # enforce the use of our selected gpg binary
1917 function gpg {
1918   command "$(gpg_binary)" "$@"
1919 }
1920 export -f gpg
1921 
1922 # start of script #######################################################################
1923 
1924 # confidentiality first, all we create is only readable by us
1925 umask 077
1926 
1927 # check if ftplicity is there & executable
1928 [ -n "$ME_LONG" ] && [ -x "$ME_LONG" ] || error "$ME missing. Executable & available in path? ($ME_LONG)"
1929 
1930 if [ ${#@} -eq 1 ]; then
1931   cmd="${1}"
1932 else
1933   FTPLCFG="${1}" ; cmd="${2}"
1934 fi
1935 
1936 # deal with command before profile validation calls
1937 # show requested version
1938 # OR requested usage info
1939 # OR create a profile
1940 # OR fall through
1941 ##if [ ${#@} -le 2 ]; then
1942 case "$cmd" in
1943   changelog)
1944     changelog
1945     exit 0
1946     ;;
1947   create)
1948     set_config
1949     if [ -d "$CONFDIR" ]; then
1950       error "The profile '$FTPLCFG' already exists in
1951 '$CONFDIR'.
1952 
1953 Hint:
1954  If you _really_ want to create a new profile by this name you will 
1955  have to manually delete the existing profile folder first."
1956       exit 1
1957     else
1958       create_config
1959       exit 0
1960     fi
1961     ;;
1962   txt2man)
1963     set_config
1964     usage_txt2man
1965     exit 0
1966     ;;
1967   usage|-help|--help|-h|-H)
1968     set_config
1969     usage_info
1970     exit 0
1971     ;;
1972   version|-version|--version|-v|-V)
1973     # profile can override GPG/PYTHON, so import it if it was given
1974     var_isset FTPLCFG && {
1975       set_config
1976       [ -r "$CONF" ] && . "$CONF" || warning "Cannot import config '$CONF'."
1977     }
1978     version_info_using
1979     exit 0
1980     ;;
1981   # fallthrough.. we got a command that needs an existing profile
1982   *)
1983     # if we reach here, user either forgot profile or chose wrong profileless command
1984     if [ ${#@} -le 1 ]; then
1985       error "\
1986  Missing or wrong parameters. 
1987  Only the commands 
1988    changelog, create, usage, txt2man, version
1989  can be called without selecting an existing profile first.
1990  Your command was '$cmd'.
1991 
1992  Hint: Run '$ME usage' to get help."
1993     fi
1994 esac
1995 
1996 
1997 # Hello world
1998 echo "Start $ME v$ME_VERSION, time is $(date_fix '%F %T')."
1999 
2000 # check system environment
2001 
2002 # is duplicity avail
2003 lookup duplicity || error_path "duplicity missing. installed und available in path?"
2004 # init, exec duplicity version check info
2005 duplicity_version_get
2006 
2007 # check for certain important helper programs
2008 # TODO: we should probably check for duplicity and $PYTHON (if set) here too
2009 for f in awk grep ; do
2010   lookup "$f" || \
2011     error_path "$f missing. installed und available in path?"
2012 done
2013 
2014 ### read configuration
2015 set_config
2016 # check validity
2017 if [ ! -d "$CONFDIR" ]; then 
2018     error "Selected profile '$FTPLCFG' does not resolve to a profile folder in
2019 '$CONFDIR'.
2020 
2021 Hints:
2022 Select one of the available profiles: $(for d in "$(dirname "$CONFDIR")"/*/; do [ -e "$d" ] || [ -L "$d" ] || continue; printf "$sep'$(basename "$d")'"; sep=",";done)
2023  Use '$ME <name> create' to create a new profile.
2024  Use '$ME usage' to get usage help."
2025 elif [ ! -x "$CONFDIR" ]; then
2026     error "\
2027 Profile folder in '$CONFDIR' cannot be accessed.
2028 
2029 Hint: 
2030  Check the filesystem permissions and set directory accessible e.g. 'chmod 700'."
2031 elif [ ! -f "$CONF" ] ; then
2032   error "'$CONF' not found."
2033 elif [ ! -r "$CONF" ] ; then
2034   error "'$CONF' not readable."
2035 else
2036   . "$CONF"
2037   #KEYFILE="${KEYFILE//.asc/${GPG_KEY:+.$GPG_KEY}.asc}"
2038   TEMP_DIR=${TEMP_DIR:-'/tmp'}
2039   # backward compatibility: old TARGET_PW overrides silently new TARGET_PASS if set
2040   if var_isset 'TARGET_PW'; then
2041     TARGET_PASS="${TARGET_PW}"
2042   fi
2043 fi
2044 echo "Using profile '$CONFDIR'."
2045 
2046 # secure config dir, if needed w/ warning
2047 secureconf
2048 
2049 # split TARGET in handy variables
2050 TARGET_SPLIT_URL=$(echo "$TARGET" | awk '{ \
2051   target=$0; match(target,/^([^\/:]+):\/\//); \
2052   prot=substr(target,RSTART,RLENGTH);\
2053   rest=substr(target,RSTART+RLENGTH); \
2054   if (credsavail=match(rest,/^[^@]*@/)){\
2055     creds=substr(rest,RSTART,RLENGTH-1);\
2056     credcount=split(creds,cred,":");\
2057     rest=substr(rest,RLENGTH+1);\
2058     # split creds with regexp\
2059     match(creds,/^([^:]+)/);\
2060     user=substr(creds,RSTART,RLENGTH);\
2061     pass=substr(creds,RSTART+1+RLENGTH);\
2062   };\
2063   # filter quotes or escape them\
2064   gsub(/[\047\042]/,"",prot);\
2065   gsub(/[\047\042]/,"",rest);\
2066   gsub(/[\047]/,"\047\\\047\047",creds);\
2067   print "TARGET_URL_PROT=\047"prot"\047\n"\
2068          "TARGET_URL_HOSTPATH=\047"rest"\047\n"\
2069          "TARGET_URL_CREDS=\047"creds"\047\n";\
2070    if(user){\
2071      gsub(/[\047]/,"\047\\\047\047",user);\
2072      print "TARGET_URL_USER=\047"user"\047\n"}\
2073    if(pass){\
2074      gsub(/[\047]/,"\047\\\047\047",pass);\
2075      print "TARGET_URL_PASS=$(url_decode \047"pass"\047)\n"}\
2076   }')
2077 eval "${TARGET_SPLIT_URL}"
2078 
2079 # fetch commmand from parameters ########################################################
2080 # Hint: cmds is also used to check if authentification info sufficient in the next step 
2081 cmds="$2"; shift 2
2082 
2083 # complain if command(s) missing
2084 [ -z $cmds ] && error "  No command given.
2085 
2086   Hint: 
2087     Use '$ME usage' to get usage help."
2088 
2089 # process params
2090 for param in "$@"; do
2091   #echo !$param!
2092   case "$param" in
2093     # enable ftplicity preview mode
2094     '--preview')
2095       PREVIEW=1
2096       ;;
2097     # interpret duplicity disable encr switch
2098     '--disable-encryption')
2099       GPG_KEY='disabled'
2100       ;;
2101     *)
2102       if [ `echo "$param" | grep -e "^-"` ] || \
2103          [ `echo "$last_param" | grep -e "^-"` ] ; then
2104         # forward parameter[/option pairs] to duplicity
2105         dupl_opts["${#dupl_opts[@]}"]=${param}
2106       else
2107         # anything else must be a parameter (e.g. for fetch, ...)
2108         ftpl_pars["${#ftpl_pars[@]}"]=${param}
2109       fi
2110       last_param=${param}
2111       ;;
2112   esac
2113 done
2114 
2115 # plausibility check config - VARS & KEY ################################################
2116 # check if src, trg, trg pw
2117 # auth info sufficient 
2118 # gpg key, gpg pwd (might be empty) set in config
2119 # OR key in local gpg db
2120 # OR key can be imported from keyfile 
2121 # OR fail
2122 if [ -z "$SOURCE" ] || [ "$SOURCE" == "${DEFAULT_SOURCE}" ]; then
2123  error " Source Path (setting SOURCE) not set or still default value in conf file 
2124  '$CONF'."
2125 
2126 elif [ -z "$TARGET" ] || [ "$TARGET" == "${DEFAULT_TARGET}" ]; then
2127  error " Backup Target (setting TARGET) not set or still default value in conf file 
2128  '$CONF'."
2129 
2130 elif var_isset 'TARGET_USER' && var_isset 'TARGET_URL_USER' && \
2131      [ "${TARGET_USER}" != "${TARGET_URL_USER}" ]; then
2132  error " TARGET_USER ('${TARGET_USER}') _and_ user in TARGET url ('${TARGET_URL_USER}') 
2133  are configured with different values. There can be only one.
2134  
2135  Hint: Remove conflicting setting."
2136 
2137 elif var_isset 'TARGET_PASS' && var_isset 'TARGET_URL_PASS' && \
2138      [ "${TARGET_PASS}" != "${TARGET_URL_PASS}" ]; then
2139  error " TARGET_PASS ('${TARGET_PASS}') _and_ password in TARGET url ('${TARGET_URL_PASS}') 
2140  are configured with different values. There can be only one.
2141  
2142  Hint: Remove conflicting setting."
2143 fi
2144 
2145 # GPG config plausibility check1 (disabled check) #############################
2146 if gpg_disabled; then
2147   : # encryption disabled, all is well
2148 elif [ -z "${GPG_KEY}${GPG_KEYS_ENC}${GPG_KEY_SIGN}" ] && ! var_isset 'GPG_PW'; then
2149   warning "GPG_KEY, GPG_KEYS_ENC, GPG_KEY_SIGN and GPG_PW are empty/not set in conf file 
2150 '$CONF'.
2151 Will disable encryption for duplicity now.
2152 
2153 Hint: 
2154  If you really want to use _no_ encryption you can disable this warning by 
2155  setting GPG_KEY='disabled' in conf file."
2156   GPG_KEY='disabled'
2157 fi
2158 
2159 # GPG availability check (now we know if gpg is really needed)#################
2160 if ! gpg_disabled; then 
2161   gpg_avail || error_path "gpg '$(gpg_binary)' missing. installed und available in path?"
2162 fi
2163 
2164 # Output versions info ########################################################
2165 using_info
2166 
2167 # GPG create key settings, config check2 (needs gpg) ##########################
2168 if gpg_disabled; then
2169   : # the following tests are not necessary
2170 else
2171 
2172 # we test this early as any invocation gpg2.1+ starts gpg-agent automatically
2173 GPG_AGENT_ERR=$(gpg_agent_avail ; echo $?)
2174 
2175 # enc key still default?
2176 if [ "$GPG_KEY" == "${DEFAULT_GPG_KEY}" ]; then 
2177   error_gpg "Encryption Key GPG_KEY still default in conf file 
2178 '$CONF'."
2179 fi
2180 
2181 # create array of gpg encr keys, for further processing
2182 OIFS="$IFS" IFS=$'\n'
2183 GPG_KEYS_ENC_ARRAY=( $( gpg_split_keyset ${GPG_KEY},${GPG_KEYS_ENC} ) )
2184 IFS="$OIFS"
2185 
2186 # pw set? 
2187 # symmetric needs one, always
2188 if gpg_symmetric && ( [ -z "$GPG_PW" ] || [ "$GPG_PW" == "${DEFAULT_GPG_PW}" ] ) \
2189   ; then
2190   error_gpg "Encryption passphrase GPG_PW (needed for symmetric encryption) 
2191 is empty/not set or still default value in conf file 
2192 '$CONF'."
2193 fi
2194 # this is a technicality, we can only pump one pass via pipe into gpg
2195 # but symmetric already always needs one for encryption
2196 if gpg_symmetric && var_isset GPG_PW && var_isset GPG_PW_SIGN &&\
2197   [ -n "$GPG_PW_SIGN" ] && [ "$GPG_PW" != "$GPG_PW_SIGN" ]; then
2198   error_gpg "GPG_PW _and_ GPG_PW_SIGN are defined but not identical in config
2199 '$CONF'.
2200 This is unfortunately impossible. For details see duplicity manpage, 
2201 section 'A Note On Symmetric Encryption And Signing'.
2202 
2203 Tip: Separate signing keys may have empty passwords e.g. GPG_PW_SIGN=''.
2204 Tip2: Use gpg-agent."
2205 fi
2206 
2207 # test - GPG KEY AVAILABILITY ##################################################
2208 
2209 # check gpg public keys availability, try import if needed
2210 for (( i = 0 ; i < ${#GPG_KEYS_ENC_ARRAY[@]} ; i++ )); do
2211   KEY_ID="${GPG_KEYS_ENC_ARRAY[$i]}"
2212   # test availability, try to import, retest
2213   if ! gpg_pub_avail "${KEY_ID}"; then
2214     echo "Encryption public key '${KEY_ID}' not in keychain. Try to import from profile."
2215     gpg_import "${KEY_ID}" PUB
2216     gpg_key_cache RESET "${KEY_ID}"
2217     gpg_pub_avail "${KEY_ID}" || { \
2218       gpg_testing && error_gpg \
2219       "Needed public gpg key '${KEY_ID}' is not available in keychain." \
2220       "Doublecheck if the above key is listed by 'gpg --list-keys' or available
2221   as gpg key file '$(basename "$(gpg_keyfile "${KEY_ID}")")' in the profile folder.
2222   If not you can put it there and $ME_NAME will autoimport it on the next run.
2223   Alternatively import it manually as the user you plan to run $ME_NAME with."
2224     }
2225   else
2226     echo "Public key '${KEY_ID}' found in keychain."
2227   fi
2228 done
2229 
2230 # check gpg encr secret encryption keys availability and fail
2231 # if none is available after a round of importing trials
2232 gpg_key_decryptable || \
2233 {
2234   echo "Missing secret keys for decryption in keychain."
2235   for (( i = 0 ; i < ${#GPG_KEYS_ENC_ARRAY[@]} ; i++ )); do
2236     KEY_ID="${GPG_KEYS_ENC_ARRAY[$i]}"
2237     # test availability, try to import, retest
2238     if ! gpg_sec_avail "${KEY_ID}"; then
2239     echo "Try to import secret key '${KEY_ID}' from profile."
2240       gpg_import "${KEY_ID}" SEC
2241       gpg_key_cache RESET "${KEY_ID}"
2242     fi
2243   done
2244   gpg_key_decryptable || \
2245   {
2246     gpg_testing && error_gpg_test "None of the configured keys '$(join "','" "${GPG_KEYS_ENC_ARRAY[@]}")' \
2247 has a secret key in the keychain. Decryption will be impossible!"
2248   }
2249 }
2250 
2251 # gpg secret sign key availability
2252 # if none set, autoset first encryption key as sign key
2253 if ! gpg_signing; then
2254   echo "Signing disabled per configuration."
2255 # try first key, if one set
2256 elif ! var_isset 'GPG_KEY_SIGN'; then
2257   KEY_ID="${GPG_KEYS_ENC_ARRAY[0]}"
2258   if [ -z "${KEY_ID}" ]; then
2259     echo "Signing disabled. No GPG_KEY entries in config."
2260     GPG_KEY_SIGN='disabled'
2261   else  
2262     # use avail OR try import OR fail
2263     if gpg_sec_avail "${KEY_ID}"; then
2264       GPG_KEY_SIGN="${KEY_ID}"
2265     else
2266       echo "Signing secret key '${KEY_ID}' not found."
2267       gpg_import "${KEY_ID}" SEC
2268       gpg_key_cache RESET "${KEY_ID}"
2269       if gpg_sec_avail "${KEY_ID}"; then
2270         GPG_KEY_SIGN="${KEY_ID}"
2271       fi
2272     fi
2273 
2274     # interpret sign key setting
2275     if var_isset 'GPG_KEY_SIGN'; then
2276       echo "Autoset found secret key of first GPG_KEY entry '${KEY_ID}' for signing."
2277     else
2278       echo "Signing disabled. First GPG_KEY entry's '${KEY_ID}' private key is missing."
2279       GPG_KEY_SIGN='disabled'
2280     fi
2281   fi
2282 else
2283   KEY_ID="${GPG_KEY_SIGN}"
2284   if ! gpg_sec_avail "${KEY_ID}"; then
2285     inform "Secret signing key defined in setting GPG_KEY_SIGN='${KEY_ID}' not found.\nTry to import."
2286     gpg_import "${KEY_ID}" SEC
2287     gpg_key_cache RESET "${KEY_ID}"
2288     gpg_sec_avail "${KEY_ID}" || error_gpg_key "${KEY_ID}" "Private"
2289   fi
2290 fi
2291 
2292 # using GPG_AGENT_ERR set early above, try to autoenable gpg-agent or issue some warnings
2293 # key enc can deal without, but might profit from gpg-agent
2294 # if GPG_PW is not set alltogether
2295 # if signing key is different from first (main) enc key (we can only pipe one pass into gpg)
2296 if ! gpg_symmetric && \
2297    ( ! var_isset GPG_PW || \
2298      ( gpg_signing && ! var_isset GPG_PW_SIGN && [ "$GPG_KEY_SIGN" != "${GPG_KEYS_ENC_ARRAY[0]}" ] ) ); then
2299 
2300   if [ "$GPG_AGENT_ERR" -eq 1 ]; then
2301     warning "Cannot use gpg-agent. GPG_AGENT_INFO not set."
2302   elif [ "$GPG_AGENT_ERR" -eq 2 ]; then
2303     warning "Cannot use gpg-agent! GPG_AGENT_INFO contains stale pid."
2304   elif [ "$GPG_AGENT_ERR" -eq 3 ]; then
2305     warning "No running gpg-agent found although GPG_PW or GPG_PW_SIGN (enc != sign key) not set."
2306   else
2307     echo "Enable gpg-agent usage. Running gpg-agent instance found and GPG_PW or GPG_PW_SIGN (enc != sign key) not set."
2308     GPG_USEAGENT="--use-agent"
2309   fi
2310 fi
2311 
2312 # end GPG config plausibility check2 
2313 fi
2314 
2315 # config plausibility check - SPACE ###########################################
2316 
2317 # is tmp is a folder and writable
2318 CMD_MSG="Checking TEMP_DIR '${TEMP_DIR}' is a folder and writable"
2319 run_cmd test -d $(qw "$TEMP_DIR") '&&' test -w $(qw "$TEMP_DIR")
2320 if [ "$?" != "0" ]; then
2321     error "Temporary file space '$TEMP_DIR' is not a directory or writable."
2322 fi
2323 
2324 
2325 # get volsize, default duplicity volume size is 25MB since v0.5.07
2326 #VOLSIZE=${VOLSIZE:-25}
2327 # double if asynch is on
2328 #echo $@ $DUPL_PARAMS | grep -q -e '--asynchronous-upload' && FACTOR=2 || FACTOR=1
2329 
2330 # TODO: check for enough (async= upload space and WARN only
2331 #       use function tmp_space 
2332 #echo TODO: reimplent tmp space check
2333 
2334 
2335 # test - GPG SANITY #####################################################################
2336 # if encryption is disabled, skip this whole section
2337 if gpg_disabled; then
2338   echo -e "Test - En/Decryption skipped. (GPG='disabled')"
2339 elif ! gpg_testing; then 
2340   echo -e "Test - En/Decryption skipped. (GPG_TEST='disabled')"
2341 else
2342 
2343 GPG_TEST_PREFIX="$TEMP_DIR/${ME_NAME}.$$.$(date_fix %s)"
2344 function cleanup_gpgtest { 
2345   echo -en "Cleanup - Delete '${GPG_TEST_PREFIX}_*'"
2346   rm "${GPG_TEST_PREFIX}"_* 2>/dev/null && echo "(OK)" || echo "(FAILED)"
2347 }
2348 
2349 # signing enabled?
2350 if gpg_signing; then
2351   CMD_PARAM_SIGN="--sign --default-key $(qw ${GPG_KEY_SIGN})"
2352   CMD_MSG_SIGN="Sign with '${GPG_KEY_SIGN}'"
2353 fi
2354 
2355 # using keys
2356 if [ ${#GPG_KEYS_ENC_ARRAY[@]} -gt 0 ]; then
2357 
2358   for KEY_ID in "${GPG_KEYS_ENC_ARRAY[@]}"; do
2359     CMD_PARAMS="$CMD_PARAMS -r $(qw ${KEY_ID})"
2360   done
2361   # check encrypting
2362   CMD_MSG="Test - Encrypt to '$(join "','" "${GPG_KEYS_ENC_ARRAY[@]}")'${CMD_MSG_SIGN:+ & $CMD_MSG_SIGN}"
2363   run_cmd $(gpg_pass_pipein GPG_PW_SIGN GPG_PW) gpg $CMD_PARAM_SIGN $(gpg_param_passwd GPG_PW_SIGN GPG_PW) $CMD_PARAMS $GPG_USEAGENT --status-fd 1 $GPG_OPTS -o $(qw "${GPG_TEST_PREFIX}_ENC") -e $(qw "$ME_LONG")
2364   CMD_ERR=$?
2365 
2366   if [ "$CMD_ERR" != "0" ]; then 
2367     KEY_NOTRUST=$(echo "$CMD_OUT"|awk '/^\[GNUPG:\] INV_RECP 10/ { print $4 }')
2368     [ -n "$KEY_NOTRUST" ] && HINT="Key '${KEY_NOTRUST}' seems to be untrusted. If you really trust this key try to
2369   'gpg --edit-key "$KEY_NOTRUST"' and raise the trust level to ultimate. If you
2370   can trust all of your keys set GPG_OPTS='--trust-model always' in conf file."
2371     error_gpg_test "Encryption failed (Code $CMD_ERR).${CMD_OUT:+\n$CMD_OUT}" "$HINT"
2372   fi
2373 
2374   # check decrypting
2375   CMD_MSG="Test - Decrypt"
2376   gpg_key_decryptable || CMD_DISABLED="No matching secret key available."
2377   run_cmd $(gpg_pass_pipein GPG_PW) gpg $(gpg_param_passwd GPG_PW) $GPG_OPTS -o $(qw "${GPG_TEST_PREFIX}_DEC") $GPG_USEAGENT -d $(qw "${GPG_TEST_PREFIX}_ENC")
2378   CMD_ERR=$?
2379 
2380   if [ "$CMD_ERR" != "0" ]; then 
2381     error_gpg_test "Decryption failed.${CMD_OUT:+\n$CMD_OUT}"
2382   fi
2383 
2384 # symmetric only
2385 else
2386   # check encrypting
2387   CMD_MSG="Test - Encryption with passphrase${CMD_MSG_SIGN:+ & $CMD_MSG_SIGN}"
2388   run_cmd $(gpg_pass_pipein GPG_PW) gpg $GPG_OPTS $CMD_PARAM_SIGN --passphrase-fd 0 -o $(qw "${GPG_TEST_PREFIX}_ENC") --batch -c $(qw "$ME_LONG")
2389   CMD_ERR=$?
2390   if [ "$CMD_ERR" != "0" ]; then 
2391     error_gpg_test "Encryption failed.${CMD_OUT:+\n$CMD_OUT}"
2392   fi
2393 
2394   # check decrypting
2395   CMD_MSG="Test - Decryption with passphrase"
2396   run_cmd $(gpg_pass_pipein GPG_PW) gpg $GPG_OPTS --passphrase-fd 0 -o $(qw "${GPG_TEST_PREFIX}_DEC") --batch -d $(qw "${GPG_TEST_PREFIX}_ENC")
2397   CMD_ERR=$?
2398   if [ "$CMD_ERR" != "0" ]; then 
2399     error_gpg_test "Decryption failed.${CMD_OUT:+\n$CMD_OUT}"
2400   fi
2401 fi
2402 
2403 # compare original w/ decryptginal
2404 CMD_MSG="Test - Compare"
2405 [ -r "${GPG_TEST_PREFIX}_DEC" ] || CMD_DISABLED="File not found. Nothing to compare."
2406 run_cmd "test \"\$(cat '$ME_LONG')\" = \"\$(cat '${GPG_TEST_PREFIX}_DEC')\""
2407 CMD_ERR=$?
2408 if [ "$CMD_ERR" = "0" ]; then 
2409   cleanup_gpgtest
2410 else
2411   error_gpg_test "Comparision failed.${CMD_OUT:+\n$CMD_OUT}"
2412 fi
2413 
2414 fi # end disabled
2415 
2416 ## an empty line
2417 #echo
2418 
2419 # Exclude file is needed, create it if necessary
2420 [ -f "$EXCLUDE" ] || touch "$EXCLUDE"
2421 
2422 # export only used keys, if bkp not already exists ######################################
2423 gpg_export_if_needed "${GPG_KEYS_ENC_ARRAY[@]}" "$(gpg_signing && echo $GPG_KEY_SIGN)"
2424 
2425 
2426 # command execution #####################################################################
2427 
2428 # urldecode url vars into plain text
2429 var_isset 'TARGET_URL_USER' && TARGET_URL_USER="$(url_decode "$TARGET_URL_USER")"
2430 var_isset 'TARGET_URL_PASS' && TARGET_URL_PASS="$(url_decode "$TARGET_URL_PASS")"
2431 
2432 # defined TARGET_USER&PASS vars replace their URL pendants 
2433 # (double defs already dealt with)
2434 var_isset 'TARGET_USER' && TARGET_URL_USER="$TARGET_USER"
2435 var_isset 'TARGET_PASS' && TARGET_URL_PASS="$TARGET_PASS"
2436 
2437 TARGET_URL_PROT_lowercase="$(tolower "${TARGET_URL_PROT%%:*}")"
2438 
2439 # issue some warnings
2440 case "$TARGET_URL_PROT_lowercase" in
2441   'cf+http')
2442     # info on missing AUTH_URL
2443     if ! var_isset 'CLOUDFILES_AUTHURL'; then
2444       inform "No CLOUDFILES_AUTHURL exported (in conf).
2445 Will use default which is probably rackspace."
2446     fi
2447     ;;
2448    'swift')
2449     # info on possibly missing AUTH_URL
2450        var_isset 'SWIFT_AUTHURL' ||\
2451        warning "\
2452 Swift will probably fail because the conf var SWIFT_AUTHURL was not exported!"
2453     ;;
2454   'rsync')
2455     # everything in url (this backend does not support pass in env var)
2456     # this is obsolete from version 0.6.10 (buggy), hopefully fixed in 0.6.11
2457     # print warning older version is detected
2458     duplicity_version_lt 610 &&
2459       warning "\
2460 Duplicity version '$DUPL_VERSION' does not support providing the password as 
2461 env var for rsync backend. For security reasons you should consider to 
2462 update to a version greater than '0.6.10' of duplicity."
2463     ;;
2464 esac
2465 
2466 
2467 # for all protocols we put username in url and pass into env var 
2468 # for sec�rity reasons, we url_encode username to protect special chars
2469 # first sortout backends with special ways to handle password
2470 case "$TARGET_URL_PROT_lowercase" in
2471   'imap'|'imaps')
2472     var_isset 'TARGET_URL_PASS' && BACKEND_PARAMS="IMAP_PASSWORD=$(qw "${TARGET_URL_PASS}")"
2473     ;;
2474   *)
2475     # rest uses FTP_PASS var
2476     var_isset 'TARGET_URL_PASS' && \
2477       BACKEND_PARAMS="FTP_PASSWORD=$(qw "${TARGET_URL_PASS}")"
2478     ;;
2479 esac
2480 # insert url encoded username into target url if needed
2481 if var_isset 'TARGET_URL_USER' && [ "$TARGET_URL_PROT_lowercase" != "file" ]; then
2482   BACKEND_URL="${TARGET_URL_PROT}$(url_encode "${TARGET_URL_USER}")@${TARGET_URL_HOSTPATH}"
2483 else
2484   BACKEND_URL="$TARGET"
2485 fi
2486 
2487 
2488 # protect eval from special chars in url (e.g. open ')' in password, 
2489 # spaces in path, quotes) happens above in duplify() via quotewrap()
2490 SOURCE="$SOURCE"
2491 BACKEND_URL="$BACKEND_URL"
2492 EXCLUDE="$EXCLUDE"
2493 # since 0.7.03 --exclude-globbing-filelist is deprecated
2494 EXCLUDE_PARAM="--exclude$(duplicity_version_lt 703 && echo -globbing)-filelist" 
2495 
2496 # replace magic separators to command equivalents (+=and,-=or,[=groupIn,]=groupOut)
2497 cmds=$(awk -v cmds="$cmds" "BEGIN{ \
2498   gsub(/\+/,\"_and_\",cmds);\
2499   gsub(/\-/,\"_or_\",cmds);\
2500   gsub(/\[/,\"_groupIn_\",cmds);\
2501   gsub(/\]/,\"_groupOut_\",cmds);\
2502   print cmds}")
2503 
2504 # split commands by '_', preserve spaces even if not allowed :)
2505 IFS='_' read -ra CMDS_IN <<< "$(tolower "$cmds")"
2506 
2507 # convert cmds to array, 
2508 # post process, translate batch commands
2509 # ATTENTION: commands are lowercase from here on out
2510 declare -a CMDS
2511 for cmd in "${CMDS_IN[@]}"; do
2512   case "$cmd" in
2513     # backup -> [pre_bkp_post]
2514     'backup')
2515       CMDS=("${CMDS[@]}" groupin pre bkp post groupout)
2516     ;;
2517     # purgeAuto -> [purge purgeFull purgeIncr] depending on set conf vars 
2518     'purgeauto')
2519       purgeAuto=${MAX_AGE:+ purge}${MAX_FULL_BACKUPS:+ purgefull}${MAX_FULLS_WITH_INCRS:+ purgeincr}
2520       [[ -z "$purgeAuto" ]] && error "Command 'fullAuto' was given but neither of the purge conf vars configured."
2521       CMDS=("${CMDS[@]}" groupin $purgeAuto groupout)
2522     ;;
2523     *)
2524       CMDS=("${CMDS[@]}" "$cmd")
2525     ;;
2526   esac
2527 done
2528 #echo $(IFS=',';echo "${CMDS[*]}")
2529 
2530 unset FTPL_ERR
2531 
2532 # run CMDS
2533 for cmd in "${CMDS[@]}";
2534 do
2535 
2536 ## init
2537 # raise index in cmd array for pre/post param
2538 var_isset 'CMD_NO' && CMD_NO=$((++CMD_NO)) || CMD_NO=0
2539 
2540 unset CMD_VALUE CMD_NEXT CMD_PREV CND_NEXT CND_PREV
2541 
2542 # get next cmd,cnd vars
2543 nextno=$(( $CMD_NO ))
2544 while ! var_isset 'CMD_NEXT'
2545 do
2546   nextno=$(($nextno+1))
2547   if [ "$nextno" -lt "${#CMDS[@]}" ]; then
2548     CMD_VALUE=${CMDS[$nextno]}
2549     is_condition "$CMD_VALUE" && CND_NEXT="$CMD_VALUE" && continue
2550     is_groupMarker "$CMD_VALUE" && continue
2551     CMD_NEXT="$CMD_VALUE"
2552   else
2553     CMD_NEXT='END'
2554   fi
2555 done
2556 
2557 # get prev cnd, cnds are skipped pseudocmds
2558 prevno=$(( $CMD_NO ));
2559 while ! var_isset 'CND_PREV'
2560 do
2561   prevno=$(($prevno-1))
2562   if [ "$prevno" -ge 0 ]; then
2563     CMD_VALUE=${CMDS[$prevno]}
2564     is_condition "$CMD_VALUE" && CND_PREV="$CMD_VALUE" && break
2565     is_command "$CMD_VALUE" && break
2566   else
2567     break
2568   fi
2569 done
2570 
2571 # get prev cmd command minus skipped commands, only executed
2572 prevno=$(( $CMD_NO - ${CMD_SKIPPED-0} ));
2573 while ! var_isset 'CMD_PREV'
2574 do
2575   prevno=$(($prevno-1))
2576   if [ "$prevno" -ge 0 ]; then
2577     CMD_VALUE=${CMDS[$prevno]}
2578     is_condition "$CMD_VALUE" && CND_PREV="$CMD_VALUE" && continue
2579     is_groupMarker "$CMD_VALUE" && continue
2580     CMD_PREV="$CMD_VALUE"
2581   else
2582     CMD_PREV='START'
2583   fi
2584 done
2585 
2586 function get_cmd_skip_count {
2587   # find closing bracket, get group skip count
2588   local nextno=$CMD_NO
2589   local GRP_OPEN=0
2590   local GRP_SKIP=0
2591   local CMD_VALUE
2592   while [ "$nextno" -lt "${#CMDS[@]}" ]
2593   do
2594     nextno=$(($nextno+1))
2595     CMD_VALUE=${CMDS[$nextno]}
2596     GRP_SKIP=$(( ${GRP_SKIP} + 1 ));
2597     if is_command "$CMD_VALUE" && [ "$GRP_OPEN" -lt 1 ]; then
2598         break;
2599     elif [ "$CMD_VALUE" == 'groupin' ]; then
2600       GRP_OPEN=$(( ${GRP_OPEN} + 1 ))
2601     elif [ "$CMD_VALUE" == 'groupout' ]; then
2602       GRP_OPEN=$(( ${GRP_OPEN} - 1 ))
2603       if [ "$GRP_OPEN" -lt 1 ]; then
2604         break;
2605       fi
2606     fi
2607   done
2608 
2609   echo $GRP_SKIP;
2610 }
2611 
2612 # decision time: are we skipping already or dealing with condition "commands" or other non-cmds?
2613 unset SKIP_NOW
2614 if var_isset 'CMD_SKIP' && [ $CMD_SKIP -gt 0 ]; then
2615   # skip cnd/grp cmds silently
2616   is_command "$cmd" && echo -e "\n--- Skipping command $(toupper $cmd) ! ---"
2617   CMD_SKIP=$(($CMD_SKIP - 1))
2618   SKIP_NOW="yes"
2619 elif ! var_isset 'PREVIEW' && [ "$cmd" == 'and' ] && [ "$CMD_ERR" -ne "0" ]; then
2620   CMD_SKIP=$(get_cmd_skip_count)
2621   # incl. this "cmd"
2622   CMD_SKIP=$(( $CMD_SKIP + 1 ))
2623   unset CMD_SKIPPED
2624   SKIP_NOW="yes"
2625 elif ! var_isset 'PREVIEW' && [ "$cmd" == 'or' ] && [ "$CMD_ERR" -eq "0" ]; then
2626   CMD_SKIP=$(get_cmd_skip_count)
2627   # incl. this "cmd"
2628   CMD_SKIP=$(( $CMD_SKIP + 1 ))
2629   unset CMD_SKIPPED
2630   SKIP_NOW="yes"
2631 elif is_condition "$cmd" || is_groupMarker "$cmd"; then
2632   unset 'CMD_SKIP';
2633   SKIP_NOW="yes"
2634 fi
2635 
2636 # let's do the skip now
2637 if [ -n "$SKIP_NOW" ]; then
2638   # sum up how many commands we actually skipped for the prev var routines
2639   CMD_SKIPPED=$((${CMD_SKIPPED-0} + 1))
2640   continue
2641 fi
2642 
2643 # save start time
2644 RUN_START=$(nsecs)
2645 
2646 # export some useful env vars for external scripts/programs to use
2647 export PROFILE CONFDIR SOURCE TARGET_URL_PROT TARGET_URL_HOSTPATH \
2648        TARGET_URL_USER TARGET_URL_PASS \
2649        GPG_KEYS_ENC=$(join "\n" "${GPG_KEYS_ENC_ARRAY[@]}") GPG_KEY_SIGN \
2650        GPG_PW CMD_PREV CMD_NEXT CMD_ERR CND_PREV CND_NEXT\
2651        RUN_START
2652 
2653 # user info
2654 echo; separator "Start running command $(toupper $cmd) at $(datefull_from_nsecs $RUN_START)"
2655 
2656 case "$(tolower $cmd)" in
2657   'pre'|'post')
2658     if [ "$cmd" == 'pre' ]; then
2659       script=$PRE
2660     else
2661       script=$POST
2662     fi
2663     # script execution in a subshell, protect us from failures/var overwrites
2664     ( run_script "$script" )
2665     ;;
2666   'bkp')
2667     duplify backup -- "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \
2668           "$SOURCE" "$BACKEND_URL"
2669     ;;
2670   'incr')
2671     duplify incremental -- "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \
2672           "$SOURCE" "$BACKEND_URL"
2673     ;;
2674   'full')
2675     duplify full -- "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \
2676           "$SOURCE" "$BACKEND_URL"
2677     ;;
2678   'verify')
2679     TIME="${ftpl_pars[0]:+"-t ${ftpl_pars[0]}"}"
2680     duplify verify -- $TIME "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \
2681           "$BACKEND_URL" "$SOURCE"
2682     ;;
2683   'verifypath')
2684     TIME="${ftpl_pars[2]:+"-t ${ftpl_pars[2]}"}"
2685     IN_PATH="${ftpl_pars[0]}"; OUT_PATH="${ftpl_pars[1]}"; 
2686     ( [ -z "$IN_PATH" ] || [ -z "$OUT_PATH" ] ) && error "  Missing parameter <rel_bkp_path> or <local_path> for verifyPath.
2687   
2688   Hint: 
2689     Syntax is -> $ME <profile> verifyPath <rel_bkp_path> <local_path> [<age>]"
2690 
2691     duplify verify -- $TIME "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \
2692           --path-to-restore "$IN_PATH" "$BACKEND_URL" "$OUT_PATH"
2693     ;;
2694   'list')
2695     # time param exists since 0.5.10+
2696     TIME="${ftpl_pars[0]:+"-t ${ftpl_pars[0]}"}"
2697     duplify list-current-files -- $TIME "${dupl_opts[@]}" "$BACKEND_URL"
2698     ;;
2699   'cleanup')
2700     duplify cleanup -- "${dupl_opts[@]}" "$BACKEND_URL"
2701     ;;
2702   'purge')
2703     MAX_AGE=${ftpl_pars[0]:-$MAX_AGE}
2704     [ -z "$MAX_AGE" ] && error "  Missing parameter <max_age>. Can be set in profile or as command line parameter."
2705     
2706     duplify remove-older-than "${MAX_AGE}" \
2707           -- "${dupl_opts[@]}" "$BACKEND_URL"
2708     ;;
2709   'purgefull')
2710     MAX_FULL_BACKUPS=${ftpl_pars[0]:-$MAX_FULL_BACKUPS}
2711     [ -z "$MAX_FULL_BACKUPS" ] && error "  Missing parameter <max_full_backups>. Can be set in profile or as command line parameter."
2712   
2713     duplify remove-all-but-n-full "${MAX_FULL_BACKUPS}" \
2714           -- "${dupl_opts[@]}" "$BACKEND_URL"
2715     ;;
2716   'purgeincr')
2717     MAX_FULLS_WITH_INCRS=${ftpl_pars[0]:-$MAX_FULLS_WITH_INCRS}
2718     [ -z "$MAX_FULLS_WITH_INCRS" ] && error "  Missing parameter <max_fulls_with_incrs>. Can be set in profile or as command line parameter."
2719   
2720     duplify remove-all-inc-of-but-n-full "${MAX_FULLS_WITH_INCRS}" \
2721           -- "${dupl_opts[@]}" "$BACKEND_URL"
2722     ;;
2723   'restore')
2724     OUT_PATH="${ftpl_pars[0]}"; TIME="${ftpl_pars[1]:-now}";
2725     [ -z "$OUT_PATH" ] && error "  Missing parameter target_path for restore.
2726   
2727   Hint: 
2728     Syntax is -> $ME <profile> restore <target_path> [<age>]"
2729 
2730     duplify restore -- -t "$TIME" "${dupl_opts[@]}" "$BACKEND_URL" "$OUT_PATH"
2731     ;;
2732   'fetch')
2733     IN_PATH="${ftpl_pars[0]}"; OUT_PATH="${ftpl_pars[1]}"; 
2734     TIME="${ftpl_pars[2]:-now}";
2735     ( [ -z "$IN_PATH" ] || [ -z "$OUT_PATH" ] ) && error "  Missing parameter <src_path> or <target_path> for fetch.
2736 
2737   Hint: 
2738     Syntax is -> $ME <profile> fetch <src_path> <target_path> [<age>]"
2739 
2740     duplify restore -- --restore-time "$TIME" "${dupl_opts[@]}" \
2741               --path-to-restore "$IN_PATH" "$BACKEND_URL" "$OUT_PATH"
2742     ;;
2743   'status')
2744     duplify collection-status -- "${dupl_opts[@]}" "$BACKEND_URL"
2745     ;;    
2746   *)
2747     error "  Unknown command '$cmd'.
2748 
2749   Hint:
2750     Use '$ME usage' to get usage help."
2751     ;;
2752 esac
2753 
2754 CMD_ERR=$?
2755 RUN_END=$(nsecs)
2756 RUNTIME=$(( $RUN_END - $RUN_START ))
2757 
2758 # print message on error; set error code
2759 if [ "$CMD_ERR" -ne 0 ]; then
2760   error_print "$(datefull_from_nsecs $RUN_END) Task '$(echo $cmd|awk '$0=toupper($0)')' failed with exit code '$CMD_ERR'."
2761   FTPL_ERR=1
2762 fi
2763 
2764 separator "Finished state $(error_to_string $CMD_ERR) at $(datefull_from_nsecs $RUN_END) - \
2765 Runtime $(printf "%02d:%02d:%02d.%03d" $((RUNTIME/1000000000/60/60)) $((RUNTIME/1000000000/60%60)) $((RUNTIME/1000000000%60)) $((RUNTIME/1000000%1000)) )"
2766 
2767 done
2768 
2769 exit ${FTPL_ERR}