Sourcecode / SVN
(Redirected from Duply-code)
Jump to navigation
Jump to search
Contents
duply sourcecode
duply's svn repository is not the most recent source for the code. Usually the svn only gets updated on releases. But not always. Development snapshots are taken daily. These are not guaranteed to work, but might contain unreleased fixes. Check the snapshot's changelog header accordingly.
SVN access
Browse the repository
https://sourceforge.net/p/ftplicity/code/HEAD/tree/duply/trunk/
SVN access
svn co https://svn.code.sf.net/p/ftplicity/code/duply/trunk duply
SVN Log
https://sourceforge.net/p/ftplicity/code/commit_browser
Latest Development Snapshot
mod time 2020-07-07
plain/text -> duply.sh
1 #!/usr/bin/env bash
2 #
3 ################################################################################
4 # duply (grown out of ftplicity), is a shell front end to duplicity that #
5 # simplifies the usage by managing settings for backup jobs in profiles. #
6 # It supports executing multiple commands in a batch mode to enable single #
7 # line cron entries and executes pre/post backup scripts. #
8 # Since version 1.5.0 all duplicity backends are supported. Hence the name #
9 # changed from ftplicity to duply. #
10 # See http://duply.net or http://ftplicity.sourceforge.net/ for more info. #
11 # (c) 2006 Christiane Ruetten, Heise Zeitschriften Verlag, Germany #
12 # (c) 2008-2019 Edgar Soldin (changes since version 1.3) #
13 ################################################################################
14 # LICENSE: #
15 # This program is licensed under GPLv2. #
16 # Please read the accompanying license information in gpl-2.0.txt. #
17 ################################################################################
18 # TODO/IDEAS/KNOWN PROBLEMS:
19 # - possibility to restore time frames (incl. deleted files)
20 # realizable by listing each backup and restore from
21 # oldest to the newest, problem: not performant
22 # - search file in all backups function and show available
23 # versions with backups date (list old avail since 0.6.06)
24 # - edit profile opens conf file in vi
25 # - implement log-fd interpretation
26 # - add a duplicity option check against the options pending
27 # deprecation since 0.5.10 namely --time-separator
28 # --short-filenames
29 # --old-filenames
30 # - add 'exclude_<command>' list usage e.g. exclude_verify
31 # - featreq 25: a download/install duplicity option
32 # - hint on install software if a piece is missing
33 # - import/export profile from/to .tgz function !!!
34 #
35 # CHANGELOG:
36 # 2.2.2 (24.02.2020)
37 # - bugfix 120: Failures in "Autoset trust of key" during restore
38 # because of gpg2.2 fingerprint output change
39 #
40 # 2.2.1 (22.01.2020)
41 # - featreq 46: Example systemd units & Howto, courtesy of Jozef Riha
42 # - featreq 47: Clarify message about keeping the profile, also by Jozef Riha
43 # - fix abbreviation spelling of 'e.g.'
44 #
45 # 2.2 (30.12.2018)
46 # - featreq 44: implement grouping for batch commands
47 # new separators are [] (square brackets) or groupIn/groupOut
48 # command 'backup' translates now to [pre_bkp_post] to be skipped as
49 # one block in case a condition was set in the batch instruction
50 #
51 # 2.1 (23.07.2018)
52 # - be more verbose when duplicity version detection fails
53 # - using info shows python binary's path for easier identification now
54 # - reworked python interpreter handling, it's either
55 # configured per PYTHON var
56 # unconfigured, parsed from duplicity shebang
57 # or set to current duplicity default 'python2' (was 'python' until now)
58 # - do not quotewrap strings because of slashes (e.g. paths) anymore
59 # - bugfix: improved in/exclude stripping from conf DUPL_PARAMS
60 #
61 # 2.0.4 (20.02.2018)
62 # - bugfix 114: "duply usage is not current" wrt. purgeFull/Incr
63 # - bugfix 115: typo in error message - "Not GPG_KEY entries" should be "No"
64 # - bugfix 117: no duply_ prefix when ARCH_DIR is set in conf
65 # - bugfix debian 882159: duply: occasionally shows negative runtimes
66 #
67 # 2.0.3 (29.08.2017)
68 # - bugfix: "line 2231: CMDS: bad array subscript"
69 # - bugfix 112: "env: illegal option -- u" on MacOSX
70 #
71 # 2.0.2 (23.05.2017)
72 # - bugfix: never insert creds into file:// targets
73 # - bugfix: avail profiles hint sometimes shortend the names by one char
74 # - bugfix 108: CMD_NEXT variable should ignore conditional commands (and, or)
75 # - export condition before/after next/prev command as CND_PREV,CND_NEXT now
76 # - bugfix 97: Unknown command should be ERROR, not WARNING
77 #
78 # 2.0.1 (16.11.2016)
79 # - bugfix 104: Duply 2.0 sets wrong archive dir, --name was always 'duply_'
80 #
81 # 2.0 (27.10.2016)
82 # made this a major version change, as we broke backward compatibility anyway
83 # (see last change in v1.10). got complaints that rightfully pointed out
84 # that should only come w/ a major version change. so, here we go ;)
85 # if your backend stops working w/ this version create a new profile and
86 # export the env vars needed as described in the comments of the conf file
87 # directly above the SOURCE setting.
88 # Changes:
89 # - making sure multi spaces in TARGET survive awk processing
90 # - new env var PROFILE exported to scripts
91 # - fix 102: expose a unique timestamp variable for pre/post scripts
92 # actually a featreq. exporting RUN_START nanosec unix timestamp
93 # - fix 101: GPG_AGENT_INFO is 'bogus' (thx Thomas Harning Jr.)
94 # - fix 96: duply cannot handle two consecutive spaces in paths
95 #
96 # 1.11.3 (29.5.2016)
97 # - fix wrong "WARNING: No running gpg-agent ..." when sign key was not set
98 #
99 # 1.11.2 (11.2.2016)
100 # - fix "gpg: unsafe" version print out
101 # - bugfix 91: v1.11 [r47] broke asymmetric encryption when using GPG_KEYS_ENC
102 # - bugfix 90: S3: TARGET_USER/PASS have no effect, added additional
103 # documentation about needed env vars to template conf file
104 #
105 # 1.11.1 (18.12.2015)
106 # - bugfix 89: "Duply has trouble with PYTHON-interpreter" on OSX homebrew
107 # - reverted duply's default PYTHON to 'python'
108 #
109 # 1.11 (24.11.2015)
110 # - remove obsolete --ssh-askpass routine
111 # - add PYTHON conf var to allow global override of used python interpreter
112 # - enforced usage of "python2" in PATH as default interpreter for internal
113 # use _and_ to run duplicity (setup.py changed the shebang to the fixed
114 # path /usr/bin/python until 0.7.05, which we circumvent this way)
115 # - featreq 36: support gpg-connect-agent as a means to detect if an agent is
116 # running (thx Thomas Harning Jr.), used gpg-agent for detection though
117 # - quotewrapped run_cmd parameters to protect it from spaces e.g. in TMP path
118 # - key export routine respects gpg-agent usage now
119 #
120 # 1.10.1 (19.8.2015)
121 # - bugfix 86: Duply+Swift outputs warning
122 # - bugfix 87: Swift fails without BACKEND_URL
123 #
124 # 1.10 (31.7.2015)
125 # - featreq 37: busybox issues - fix awk, grep version detection,
126 # fix grep failure because --color=never switch is unsupported
127 # (thx Thomas Harning Jr. for reporting and helping to debug/fix it)
128 # - bugfix 81: --exclude-globbing-filelist is deprecated since 0.7.03
129 # (thx Joachim Wiedorn, also for maintaining the debian package)
130 # - implemented base-/dirname as bash functions
131 # - featreq 31 " Support for duplicity Azure backend " - ignored a
132 # contributed patch by Scott McKenzie and instead opted for removing almost
133 # all code that deals with special env vars required by backends.
134 # adding and modifying these results in too much overhead so i dropped this
135 # feature. the future alternative for users is to consult the duplicity
136 # manpage and add the needed export definitions to the conf file.
137 # appended a commented example to the template conf below the auth section.
138 #
139 # 1.9.2 (21.6.2015)
140 # - bugfix: exporting keys with gpg2.1 works now (thx Philip Jocks)
141 # - documented GPG_OPTS needed for gpg2.1 to conf template (thx Troy Engel)
142 # - bugfix 82: GREP_OPTIONS=--color=always disrupted time calculation
143 # - added GPG conf var (see conf template for details)
144 # - added grep version output as it is an integral needed binary
145 # - added PYTHONPATH printout in version output
146 #
147 # 1.9.1 (13.10.2014)
148 # - export CMD_ERR now for scripts to detect if CMD_PREV failed/succeeded
149 # - bugfix: CMD_PREV contained command even if it was skipped
150 #
151 # 1.9.0 (24.8.2014)
152 # - bugfix: env vars were not exported when external script was executable
153 # - rework GPG_KEY handling, allow virtually anything now (uid, keyid etc.)
154 # see gpg manpage, section "How to specify a user ID"
155 # let gpg complain when the delivered values are invalid for whatever reason
156 # - started to rework tmp space checking, exposed folder & writable check
157 # TODO: reimplement enough file space available checking
158 #
159 # 1.8.0 (13.7.2014)
160 # - add command verifyPath to expose 'verify --file-to-restore' action
161 # - add time parameter support to verify command
162 # - add section time formats to usage output
163 #
164 # 1.7.4 (24.6.2014)
165 # - remove ubuntu one support, service is discontinued
166 # - featreq 31: add authenticated swift (contributed by Justus Seifert)
167 #
168 # 1.7.3 (3.4.2014)
169 # - bugfix: test routines, gpg2 asked for passphrase although GPG_PW was set
170 #
171 # 1.7.2 (1.4.2014 "April,April")
172 # - bugfix: debian Bug#743190 "duply no longer allows restoration without
173 # gpg passphrase in conf file"
174 # GPG_AGENT_INFO env var is now needed to trigger --use-agent
175 # - bugfix: gpg keyenc test routines didn't work if GPG_PW was not set
176 #
177 # 1.7.1 (30.3.2014)
178 # - bugfix: purge-* commands renamed to purgeFull, purgeIncr due to
179 # incompatibility with new minus batch separator
180 #
181 # 1.7.0 (20.3.2014)
182 # - disabled gpg key id plausibility check, too many valid possibilities
183 # - featreq 7 "Halt if precondition fails":
184 # added and(+), or(-) batch command(separator) support
185 # - featreq 26 "pre/post script with shebang line":
186 # if a script is flagged executable it's executed in a subshell
187 # now as opposed to sourced to bash, which is the default
188 # - bugfix: do not check if dpbx, swift credentials are set anymore
189 # - bugfix: properly escape profile name, archdir if used as arguments
190 # - add DUPL_PRECMD conf setting for use with e.g. trickle
191 #
192 # 1.6.0 (1.1.2014)
193 # - support gs backend
194 # - support dropbox backend
195 # - add gpg-agent support to gpg test routines
196 # - autoenable --use-agent if passwords were not defined in config
197 # - GPG_OPTS are now honored everywhere, keyrings or complete gpg
198 # homedir can thus be configured to be located anywhere
199 # - always import both secret and public key if avail from config profile
200 # - new explanatory comments in initial exclude file
201 # - bugfix 7: Duply only imports one key at a time
202 #
203 # 1.5.11 (19.07.2013)
204 # - purge-incr command for remove-all-inc-of-but-n-full feature added
205 # patch provided by Moritz Augsburger, thanks!
206 # - documented version command in man page
207 #
208 # 1.5.10 (26.03.2013)
209 # - minor indent and documentation fixes
210 # - bugfix: exclude filter failed on ubuntu, mawk w/o posix char class support
211 # - bugfix: fix url_decoding generally and for python3
212 # - bugfix 3609075: wrong script results in status line (thx David Epping)
213 #
214 # 1.5.9 (22.11.2012)
215 # - bugfix 3588926: filter --exclude* params for restore/fetch ate too much
216 # - restore/fetch now also ignores --include* or --exclude='foobar'
217 #
218 # 1.5.8 (26.10.2012)
219 # - bugfix 3575487: implement proper cloud files support
220 #
221 # 1.5.7 (10.06.2012)
222 # - bugfix 3531450: Cannot use space in target URL (file:///) anymore
223 #
224 # 1.5.6 (24.5.2012)
225 # - commands purge, purge-full have no default value anymore for security
226 # reasons; instead max value can be given via cmd line or must be set
227 # in profile; else an error is shown.
228 # - minor man page modifications
229 #
230 # versioning scheme will be simplified to [major].[minor].[patch] version
231 # with the next version raise
232 #
233 # 1.5.5.5 (4.2.2012)
234 # - bugfix 3479605: SEL context confused profile folder's permission check
235 # - colon ':' in url passphrase got ignored, added python driven url_decoding
236 # for user & pass to better process special chars
237 #
238 # 1.5.5.4 (16.10.2011)
239 # - bugfix 3421268: SFTP passwords from conf ignored and always prompted for
240 # - add support for separate sign passphrase (needs duplicity 0.6.14+)
241 #
242 # 1.5.5.3 (1.10.2011)
243 # - bugfix 3416690: preview threw echo1 error
244 # - fix unknown cmds error usage & friends if more than 2 params were given
245 #
246 # 1.5.5.2 (23.9.2011)
247 # - bugfix 3409643: ssh key auth did ask for passphrase (--ssh-askpass ?)
248 # - bugfix: mawk does not support \W and did not split multikey definitions
249 # - all parameters should survive single (') and double (") quotes now
250 #
251 # 1.5.5.1 (7.6.2011)
252 # - featreq 3311881: add ftps as supported by duplicity 0.6.13 (thx mape2k)
253 # - bugfix 3312208: signing detection broke symmetric gpg test routine
254 #
255 # 1.5.5 (2.5.2011)
256 # - bugfix: fetch problem with space char in path, escape all params
257 # containing non word chars
258 # - list available profiles, if given profile cannot be found
259 # - added --use-agent configuration hint
260 # - bugfix 3174133: --exclude* params in conf DUPL_PARAMS broke
261 # fetch/restore
262 # - version command now prints out 'using installed' info
263 # - featreq 3166169: autotrust imported keys, based on code submitted by
264 # Martin Ellis - imported keys are now automagically trusted ultimately
265 # - new txt2man feature to create manpages for package maintainers
266 #
267 # 1.5.4.2 (6.1.2011)
268 # - new command changelog
269 # - bugfix 3109884: freebsd awk segfaulted on printf '%*', use print again
270 # - bugfix: freebsd awk hangs on 'awk -W version'
271 # - bugfix 3150244: mawk does not know '--version'
272 # - minor help text improvements
273 # - new env vars CMD_PREV,CMD_NEXT replacing CMD env var for scripts
274 #
275 # 1.5.4.1 (4.12.2010)
276 # - output awk, python, bash version now in prolog
277 # - shebang uses /usr/bin/env now for freebsd compatibility,
278 # bash not in /bin/bash
279 # - new --disable-encryption parameter,
280 # to override profile encr settings for one run
281 # - added exclude-if-present setting to conf template
282 # - bug 3126972: GPG_PW only needed for signing/symmetric encryption
283 # (even though duplicity still needs it)
284 #
285 # 1.5.4 (15.11.2010)
286 # - as of 1.5.3 already, new ARCH_DIR config option
287 # - multiple key support
288 # - ftplicity-Feature Requests-2994929: separate encryption and signing key
289 # - key signing of symmetric encryption possible (duplicity patch committed)
290 # - gpg tests disable switch
291 # - gpg tests now previewable and more intelligent
292 #
293 # 1.5.3 (1.11.2010)
294 # - bugfix 3056628: improve busybox compatibility, grep did not have -m param
295 # - bugfix 2995408: allow empty password for PGP key
296 # - bugfix 2996459: Duply erroneously escapes '-' symbol in username
297 # - url_encode function is now pythonized
298 # - rsync uses FTP_PASSWORD now if duplicity 0.6.10+ , else issue warning
299 # - feature 3059262: Make pre and post aware of parameters,
300 # internal parameters + CMD of pre or post
301 #
302 # 1.5.2.3 (16.4.2010)
303 # - bugfix: date again, should now work virtually anywhere
304 #
305 # 1.5.2.2 (3.4.2010)
306 # - minor bugfix: duplicity 0.6.8b version string now parsable
307 # - added INSTALL.txt
308 #
309 # 1.5.2.1 (23.3.2010)
310 # - bugfix: date formatting is awked now and should work on all platforms
311 #
312 # 1.5.2 (2.3.2010)
313 # - bugfix: errors print to STD_ERR now, failed tasks print an error message
314 # - added --name=duply_<profile> for duplicity 0.6.01+ to name cache folder
315 # - simplified & cleaned profileless commands, removed second instance
316 # - generalized separator time routines
317 # - added support for --no-encryption (GPG_KEY='disabled'), see conf examples
318 # - minor fixes
319 #
320 # 1.5.1.5 (5.2.2010)
321 # - bugfix: added special handling of credentials for rsync, imap(s)
322 #
323 # 1.5.1.4 (7.1.2010)
324 # - bugfix: nsecs defaults now to zeroes if date does not deliver [0-9]{9}
325 # - check if ncftp binary is available if url protocol is ftp
326 # - bugfix: duplicity output is now printed to screen directly to resolve
327 # 'mem alloc problem' bug report
328 # - bugfix: passwords will not be in the url anymore to solve the 'duply shows
329 # sensitive data in process listing' bug report
330 #
331 # 1.5.1.3 (24.12.2009) 'merry xmas'
332 # - bugfix: gpg pass now apostrophed to allow space and friends
333 # - bugfix: credentials are now url encoded to allow special chars in them
334 # a note about url encoding has been added to the conf template
335 #
336 # 1.5.1.2 (1.11.2009)
337 # - bugfix: open parenthesis in password broke duplicity execution
338 # - bugfix: ssh/scp backend does not always need credentials e.g. key auth
339 #
340 # 1.5.1.1 (21.09.2009)
341 # - bugfix: fixed s3[+http] TARGET_PASS not needed routine
342 # - bugfix: TYPO in duply 1.5.1 prohibited the use of /etc/duply
343 # see https://sourceforge.net/tracker/index.php?func=detail&
344 # aid=2864410&group_id=217745&atid=1041147
345 #
346 # 1.5.1 (21.09.2009) - duply (fka. ftplicity)
347 # - first things first: ftplicity (being able to support all backends since
348 # some time) will be called duply (fka. ftplicity) from now on. The addendum
349 # is for the time being to circumvent confusion.
350 # - bugfix: exit code is 1 (error) not 0 (success), if at least on duplicity
351 # command failed
352 # - s3[+http] now supported natively by translating user/pass to access_key/
353 # secret_key environment variables needed by duplicity s3 boto backend
354 # - bugfix: additional output lines do not confuse version check anymore
355 # - list command supports now age parameter (patch by stefan on feature
356 # request tracker)
357 # - bugfix: option/param pairs are now correctly passed on to duplicity
358 # - bugfix: s3[+http] needs no TARGET_PASS if command is read only
359 #
360 # 1.5.0.2 (31.07.1009)
361 # - bugfix: insert password in target url didn't work with debian mawk
362 # related to previous bug report
363 #
364 # 1.5.0.1 (23.07.2009)
365 # - bugfix: gawk gensub dependency raised an error on debian's default mawk
366 # replaced with match/substr command combination (bug report)
367 # https://sf.net/tracker/?func=detail&atid=1041147&aid=2825388&
368 # group_id=217745
369 #
370 # 1.5.0 (01.07.2009)
371 # - removed ftp limitation, all duplicity backends should work now
372 # - bugfix: date for separator failed on openwrt busybox date, added a
373 # detecting workaround, milliseconds are not available w/ busybox date
374 #
375 # 1.4.2.1 (14.05.2009)
376 # - bugfix: free temp space detection failed with lvm, fixed awk parse routine
377 #
378 # 1.4.2 (22.04.2009)
379 # - gpg keys are now exported as gpgkey.[id].asc , the suffix reflects the
380 # armored ascii nature, the id helps if the key is switched for some reason
381 # im/export routines are updated accordingly (import is backward compatible
382 # to the old profile/gpgkey files)
383 # - profile argument is treated as path if it contains slashes
384 # (for details see usage)
385 # - non-ftplicity options (all but --preview currently) are now passed
386 # on to duplicity
387 # - removed need for stat in secure_conf, it is ls based now
388 # - added profile folder readable check
389 # - added gpg version & home info output
390 # - awk utility availability is now checked, because it was mandatory already
391 # - tmp space is now checked on writability and space requirement
392 # test fails on less than 25MB or configured $VOLSIZE,
393 # test warns if there is less than two times $VOLSIZE because
394 # that's required for --asynchronous-upload option
395 # - gpg functionality is tested now before executing duplicity
396 # test drive contains encryption, decryption, comparison, cleanup
397 # this is meant to detect non trusted or other gpg errors early
398 # - added possibility of doing symmetric encryption with duplicity
399 # set GPG_KEY="" or simply comment it out
400 # - added hints in config template on the depreciation of
401 # --short-filenames, --time-separator duplicity options
402 #
403 # new versioning scheme 1.4.2b => 1.4.2,
404 # beta b's are replaced by a patch count number e.g. 1.4.2.1 will be assigned
405 # to the first bug fixing version and 1.4.2.2 to the second and so on
406 # also the releases will now have a release date formatted (Day.Month.Year)
407 #
408 # 1.4.1b1 - bugfix: ftplicity changed filesystem permission of a folder
409 # named exactly as the profile if existing in executing dir
410 # - improved plausibility checking of config and profile folder
411 # - secure_conf only acts if needed and prints a warning now
412 #
413 # 1.4.1b - introduce status (duplicity collection-status) command
414 # - pre/post script output printed always now, not only on errors
415 # - new config parameter GPG_OPTS to pass gpg options
416 # added examples & comments to profile template conf
417 # - reworked separator times, added duration display
418 # - added --preview switch, to preview generated command lines
419 # - disabled MAX_AGE, MAX_FULL_BACKUPS, VERBOSITY in generated
420 # profiles because they have reasonable defaults now if not set
421 #
422 # 1.4.0b1 - bugfix: incr forces incremental backups on duplicity,
423 # therefore backup translates to pre_bkp_post now
424 # - bugfix: new command bkp, which represents duplicity's
425 # default action (incr or full if full_if_older matches
426 # or no earlier backup chain is found)
427 #
428 # new versioning scheme 1.4 => 1.4.0, added new minor revision number
429 # this is meant to slow down the rapid version growing but still keep
430 # versions cleanly separated.
431 # only additional features will raise the new minor revision number.
432 # all releases start as beta, each bugfix release will raise the beta
433 # count, usually new features arrive before a version 'ripes' to stable
434 #
435 # 1.4.0b
436 # 1.4b - added startup info on version, time, selected profile
437 # - added time output to separation lines
438 # - introduced: command purge-full implements duplicity's
439 # remove-all-but-n-full functionality (patch by unknown),
440 # uses config variable $MAX_FULL_BACKUPS (default = 1)
441 # - purge config var $MAX_AGE defaults to 1M (month) now
442 # - command full does not execute pre/post anymore
443 # use batch command pre_full_post if needed
444 # - introduced batch mode cmd1_cmd2_etc
445 # (in turn removed the bvp command)
446 # - unknown/undefined command issues a warning/error now
447 # - bugfix: version check works with 0.4.2 and older now
448 # 1.3b3 - introduced pre/post commands to execute/debug scripts
449 # - introduced bvp (backup, verify, purge)
450 # - bugfix: removed need for awk gensub, now mawk compatible
451 # 1.3b2 - removed pre/post need executable bit set
452 # - profiles now under ~/.ftplicity as folders
453 # - root can keep profiles in /etc/ftplicity, folder must be
454 # created by hand, existing profiles must be moved there
455 # - removed ftplicity in path requirement
456 # - bugfix: bash < v.3 did not know '=~'
457 # - bugfix: purge works again
458 # 1.3 - introduces multiple profiles support
459 # - modified some script errors/docs
460 # - reordered gpg key check import routine
461 # - added 'gpg key id not set' check
462 # - added error_gpg (adds how to setup gpg key howto)
463 # - bugfix: duplicity 0.4.4RC4+ parameter syntax changed
464 # - duplicity_version_check routine introduced
465 # - added time separator, shortnames, volsize, full_if_older
466 # duplicity options to config file (inspired by stevie
467 # from http://weareroot.de)
468 # 1.1.1 - bugfix: encryption reactivated
469 # 1.1 - introduced config directory
470 # 1.0 - first release
471 ################################################################################
472
473 # utility functions overriding binaries
474
475 # wrap grep to override possible env set GREP_OPTIONS=--color=always
476 function grep {
477 command env "GREP_OPTIONS=" grep "$@"
478 }
479
480 # implement basename in plain bash
481 function basename {
482 local stripped="${1%/}"
483 echo "${stripped##*/}"
484 }
485
486 # implement dirname in plain bash
487 function dirname {
488 echo ${1%/*}
489 }
490
491 # implement basic which in plain bash
492 function which {
493 type -p "$@"
494 }
495
496 # check availability of executables via file name or file paths
497 function lookup {
498 local bin="$1"
499 # look for file names in path via bash hash OR
500 # look for executables at given relative/absolute location
501 ( [ "${bin##*/}" == "$bin" ] && hash "$bin" 2>/dev/null ) || [ -x "$bin" ]
502 }
503
504 # the python binary to use, exit code 0 when configured, else 1
505 function python_binary {
506 # if unset, parse from duplicity shebang
507 if ! var_isset 'PYTHON'; then
508 duplicity_python_binary_parse;
509 echo $DUPL_PYTHON_BIN;
510 return 1;
511 else
512 # tell if PYTHON was configured manually
513 echo $PYTHON;
514 return 0
515 fi
516 }
517
518 # important definitions #######################################################
519
520 ME_LONG="$0"
521 ME="$(basename $0)"
522 ME_NAME="${ME%%.*}"
523 ME_VERSION="2.2.2"
524 ME_WEBSITE="http://duply.net"
525
526 # default config values
527 DEFAULT_SOURCE='/path/of/source'
528 DEFAULT_TARGET='scheme://user[:password]@host[:port]/[/]path'
529 DEFAULT_TARGET_USER='_backend_username_'
530 DEFAULT_TARGET_PASS='_backend_password_'
531 DEFAULT_GPG='gpg'
532 DEFAULT_GPG_KEY='_KEY_ID_'
533 DEFAULT_GPG_PW='_GPG_PASSWORD_'
534 DEFAULT_PYTHON='python2'
535
536 # function definitions ##########################
537
538 function set_config { # sets global config vars
539 local CONFHOME_COMPAT="$HOME/.ftplicity"
540 local CONFHOME="$HOME/.duply"
541 local CONFHOME_ETC_COMPAT="/etc/ftplicity"
542 local CONFHOME_ETC="/etc/duply"
543
544 # confdir can be delivered as path (must contain /)
545 if [ `echo $FTPLCFG | grep /` ] ; then
546 CONFDIR=$(readlink -f $FTPLCFG 2>/dev/null || \
547 ( echo $FTPLCFG|grep -v '^/' 1>/dev/null 2>&1 \
548 && echo $(pwd)/${FTPLCFG} ) || \
549 echo ${FTPLCFG})
550 # or DEFAULT in home/.duply folder (NEW)
551 elif [ -d "${CONFHOME}" ]; then
552 CONFDIR="${CONFHOME}/${FTPLCFG}"
553 # or in home/.ftplicity folder (OLD)
554 elif [ -d "${CONFHOME_COMPAT}" ]; then
555 CONFDIR="${CONFHOME_COMPAT}/${FTPLCFG}"
556 warning_oldhome "${CONFHOME_COMPAT}" "${CONFHOME}"
557 # root can put profiles under /etc/duply (NEW) if path exists
558 elif [ -d "${CONFHOME_ETC}" ] && [ "$EUID" -eq 0 ]; then
559 CONFDIR="${CONFHOME_ETC}/${FTPLCFG}"
560 # root can keep profiles under /etc/ftplicity (OLD) if path exists
561 elif [ -d "${CONFHOME_ETC_COMPAT}" ] && [ "$EUID" -eq 0 ]; then
562 CONFDIR="${CONFHOME_ETC_COMPAT}/${FTPLCFG}"
563 warning_oldhome "${CONFHOME_ETC_COMPAT}" "${CONFHOME_ETC}"
564 # hmm no profile folder there, then use default for error later
565 else
566 CONFDIR="${CONFHOME}/${FTPLCFG}" # continue, will fail later in main
567 fi
568
569 # remove trailing slash, get profile name etc.
570 CONFDIR="${CONFDIR%/}"
571 PROFILE="${CONFDIR##*/}"
572 CONF="$CONFDIR/conf"
573 PRE="$CONFDIR/pre"
574 POST="$CONFDIR/post"
575 EXCLUDE="$CONFDIR/exclude"
576 KEYFILE="$CONFDIR/gpgkey.asc"
577 }
578
579 function version_info { # print version information
580 cat <<END
581 $ME_NAME version $ME_VERSION
582 ($ME_WEBSITE)
583 END
584 }
585
586 function version_info_using {
587 cat <<END
588 $(version_info)
589
590 $(using_info)
591 END
592 }
593
594 function using_info {
595 # init needed vars into global name space
596 lookup duplicity && { duplicity_python_binary_parse; duplicity_version_get; }
597 local NOTFOUND="INVALID"
598 local AWK_VERSION GREP_VERSION PYTHON_RUNNER
599 # freebsd awk (--version only), debian mawk (-W version only), deliver '' so awk does not wait for input
600 AWK_VERSION=$( lookup awk && (awk --version 2>/dev/null || awk -W version 2>&1) | awk 'NR<=2&&tolower($0)~/(busybox|awk)/{success=1;print;exit} END{if(success<1) print "unknown"}' || echo "$NOTFOUND" )
601 GREP_VERSION=$( lookup grep && grep --version 2>&1 | awk 'NR<=2&&tolower($0)~/(busybox|grep.*[0-9]+\.[0-9]+)/{success=1;print;exit} END{if(success<1) print "unknown"}' || echo "$NOTFOUND" )
602 PYTHON_RUNNER=$(python_binary)
603 local PYTHON_VERSION=$(lookup "$PYTHON_RUNNER" && "$PYTHON_RUNNER" -V 2>&1| awk '{print tolower($0);exit}' || echo "'$PYTHON_RUNNER' $NOTFOUND" )
604 local GPG_INFO=$(gpg_avail && gpg --version 2>&1| awk '/^gpg.*[0-9\.]+$/&&length(v)<1{v=$1" "$3}/^Home:/{h=" ("$0")"}END{print v""h}' || echo "gpg $NOTFOUND")
605 local BASH_VERSION=$(bash --version | awk 'NR==1{IGNORECASE=1;sub(/GNU bash, version[ ]+/,"",$0);print $0}')
606 # print out
607 echo -e "Using installed duplicity version ${DUPL_VERSION:-$NOTFOUND}\
608 ${PYTHON_VERSION+, $PYTHON_VERSION ${PYTHON_RUNNER:+($(which "$PYTHON_RUNNER"))}${PYTHONPATH:+ 'PYTHONPATH=$PYTHONPATH'}}\
609 ${GPG_INFO:+, $GPG_INFO}${AWK_VERSION:+, awk '${AWK_VERSION}'}${GREP_VERSION:+, grep '${GREP_VERSION}'}\
610 ${BASH_VERSION:+, bash '${BASH_VERSION}'}."
611 }
612
613 function usage_info { # print usage information
614
615 cat <<USAGE_EOF
616 VERSION:
617 $(version_info)
618
619 DESCRIPTION:
620 Duply deals as a wrapper for the mighty duplicity magic.
621 It simplifies running duplicity with cron or on command line by:
622
623 - keeping recurring settings in profiles per backup job
624 - enabling batch operations e.g. backup_verify+purge
625 - executing pre/post scripts (different actions possible
626 depending on previous or next command or it's exit status)
627 - precondition checking for flawless duplicity operation
628
629 For each backup job one configuration profile must be created.
630 The profile folder will be stored under '~/.${ME_NAME}/<profile>'
631 (where ~ is the current users home directory).
632 Hint:
633 If the folder '/etc/${ME_NAME}' exists, the profiles for the super
634 user root will be searched & created there.
635
636 USAGE:
637 first time usage (profile creation):
638 $ME <profile> create
639
640 general usage in single or batch mode (see EXAMPLES):
641 $ME <profile> <command>[[_|+|-]<command>[_|+|-]...] [<options> ...]
642
643 For batches the conditional separators can also be written as pseudo commands
644 and(+), or(-). See SEPARATORS for details.
645
646 Non $ME options are passed on to duplicity (see OPTIONS).
647 All conf parameters can also be defined in the environment instead.
648
649 PROFILE:
650 Indicated by a path or a profile name (<profile>), which is resolved
651 to '~/.${ME_NAME}/<profile>' (~ expands to environment variable \$HOME).
652
653 Superuser root can place profiles under '/etc/${ME_NAME}'. Simply create
654 the folder manually before running $ME_NAME as superuser.
655 Note:
656 Already existing profiles in root's home folder will cease to work
657 unless they are moved to the new location manually.
658
659 example 1: $ME humbug backup
660
661 Alternatively a _path_ might be used e.g. useful for quick testing,
662 restoring or exotic locations. Shell expansion should work as usual.
663 Hint:
664 The path must contain at least one path separator '/',
665 e.g. './test' instead of only 'test'.
666
667 example 2: $ME ~/.${ME_NAME}/humbug backup
668
669 SEPARATORS:
670 _ (underscore)
671 neutral separator
672 + (plus sign), _and_
673 conditional AND
674 the next command will only be executed if the previous succeeded
675 - (minus sign), _or_
676 conditional OR
677 the next command will only be executed if the previous failed
678 [] (square brackets), _groupIn_/_groupOut_
679 enables grouping of commands
680
681 example:
682 'pre+[bkp-verify]_post' translates to
683 'pre_and_groupIn_bkp_or_verify_groupOut_post'
684
685 COMMANDS:
686 usage get usage help text
687
688 and/or/groupIn/groupOut
689 pseudo commands used in batches (see SEPARATORS above)
690
691 create creates a configuration profile
692 backup backup with pre/post script execution (batch: [pre_bkp_post]),
693 full (if full_if_older matches or no earlier backup is found)
694 incremental (in all other cases)
695 pre/post execute '<profile>/$(basename "$PRE")', '<profile>/$(basename "$POST")' scripts
696 bkp as above but without executing pre/post scripts
697 full force full backup
698 incr force incremental backup
699 list [<age>]
700 list all files in backup (as it was at <age>, default: now)
701 status prints backup sets and chains currently in repository
702 verify [<age>] [--compare-data]
703 list files changed, since age if given
704 verifyPath <rel_path_in_bkp> <local_path> [<age>] [--compare-data]
705 list changes of a file or folder path in backup compared to a
706 local path, since age if given
707 restore <target_path> [<age>]
708 restore the complete backup to <target_path> [as it was at <age>]
709 fetch <src_path> <target_path> [<age>]
710 fetch single file/folder from backup [as it was at <age>]
711 purge [<max_age>] [--force]
712 list outdated backup files (older than \$MAX_AGE)
713 [use --force to actually delete these files]
714 purgeFull [<max_full_backups>] [--force]
715 list outdated backup files (\$MAX_FULL_BACKUPS being the number of
716 full backups and associated incrementals to keep, counting in
717 reverse chronological order)
718 [use --force to actually delete these files]
719 purgeIncr [<max_fulls_with_incrs>] [--force]
720 list outdated incremental backups (\$MAX_FULLS_WITH_INCRS being
721 the number of full backups which associated incrementals will be
722 kept, counting in reverse chronological order)
723 [use --force to actually delete these files]
724 cleanup [--force]
725 list broken backup chain files archives (e.g. after unfinished run)
726 [use --force to actually delete these files]
727
728 changelog print changelog / todo list
729 txt2man feature for package maintainers - create a manpage based on the
730 usage output. download txt2man from http://mvertes.free.fr/, put
731 it in the PATH and run '$ME txt2man' to create a man page.
732 version show version information of $ME_NAME and needed programs
733
734 OPTIONS:
735 --force passed to duplicity (see commands:
736 purge, purgeFull, purgeIncr, cleanup)
737 --preview do nothing but print out generated duplicity command lines
738 --disable-encryption
739 disable encryption, overrides profile settings
740
741 TIME FORMATS:
742 For all time related parameters like age, max_age etc.
743 Refer to the duplicity manpage for all available formats. Here some examples:
744 2002-01-25T07:00:00+02:00 (full date time format string)
745 2002/3/5 (date string YYYY/MM/DD)
746 12D (interval, 12 days ago)
747 1h78m (interval, 1 hour 78 minutes ago)
748
749 PRE/POST SCRIPTS:
750 Some useful internal duply variables are exported to the scripts.
751
752 PROFILE, CONFDIR, SOURCE, TARGET_URL_<PROT|HOSTPATH|USER|PASS>,
753 GPG_<KEYS_ENC|KEY_SIGN|PW>, CMD_ERR, RUN_START,
754 CMD_<PREV|NEXT> (previous/next command),
755 CND_<PREV|NEXT> (condition before/after)
756
757 The CMD_* variables were introduced to allow different actions according to
758 the command the scripts were attached to e.g. 'pre_bkp_post_pre_verify_post'
759 will call the pre script two times, with CMD_NEXT variable set to 'bkp'
760 on the first and to 'verify' on the second run.
761 CMD_ERR holds the exit code of the CMD_PREV .
762
763 EXAMPLES:
764 create profile 'humbug':
765 $ME humbug create (don't forget to edit this new conf file)
766 backup 'humbug' now:
767 $ME humbug backup
768 list available backup sets of profile 'humbug':
769 $ME humbug status
770 list and delete outdated backups of 'humbug':
771 $ME humbug purge --force
772 restore latest backup of 'humbug' to /mnt/restore:
773 $ME humbug restore /mnt/restore
774 restore /etc/passwd of 'humbug' from 4 days ago to /root/pw:
775 $ME humbug fetch etc/passwd /root/pw 4D
776 (see "duplicity manpage", section TIME FORMATS)
777 a one line batch job on 'humbug' for cron execution:
778 $ME humbug backup_verify_purge --force
779 batch job to run a full backup with pre/post scripts:
780 $ME humbug pre_full_post
781
782 FILES:
783 in profile folder '~/.${ME_NAME}/<profile>' or '/etc/${ME_NAME}'
784 conf profile configuration file
785 pre,post pre/post scripts (see above for details)
786 gpgkey.*.asc exported GPG key files
787 exclude a globbing list of included or excluded files/folders
788 (see "duplicity manpage", section FILE SELECTION)
789
790 $(hint_profile)
791
792 SEE ALSO:
793 duplicity man page:
794 duplicity(1) or http://duplicity.nongnu.org/duplicity.1.html
795 USAGE_EOF
796 }
797
798 # to check call 'duply txt2man | man -l -'
799 function usage_txt2man {
800 usage_info | \
801 awk '/^^[^[:lower:][:space:]][^[:lower:]]+$/{gsub(/[^[:upper:]]/," ",$0)}{print}' |\
802 txt2man -t"$(toupper "${ME_NAME}")" -s1 -r"${ME_NAME}-${ME_VERSION}" -v'User Manuals'
803 }
804
805 function changelog {
806 cat $ME_LONG | awk '/^#####/{on=on+1}(on==3){sub(/^#( )?/,"",$0);print}'
807 }
808
809 function create_config {
810 if [ ! -d "$CONFDIR" ] ; then
811 mkdir -p "$CONFDIR" || error "Couldn't create config '$CONFDIR'."
812 # create initial config file
813 cat <<EOF >"$CONF"
814 # gpg encryption settings, simple settings:
815 # GPG_KEY='disabled' - disables encryption alltogether
816 # GPG_KEY='<key1>[,<key2>]'; GPG_PW='pass' - encrypt with keys,
817 # sign if secret key of key1 is available use GPG_PW for sign & decrypt
818 # Note: you can specify keys via all methods described in gpg manpage,
819 # section "How to specify a user ID", escape commas (,) via backslash (\)
820 # e.g. 'Mueller, Horst', 'Bernd' -> 'Mueller\, Horst, Bernd'
821 # as they are used to separate the entries
822 # GPG_PW='passphrase' - symmetric encryption using passphrase only
823 GPG_KEY='${DEFAULT_GPG_KEY}'
824 GPG_PW='${DEFAULT_GPG_PW}'
825 # gpg encryption settings in detail (extended settings)
826 # the above settings translate to the following more specific settings
827 # GPG_KEYS_ENC='<keyid1>[,<keyid2>,...]' - list of pubkeys to encrypt to
828 # GPG_KEY_SIGN='<keyid1>|disabled' - a secret key for signing
829 # GPG_PW='<passphrase>' - needed for signing, decryption and symmetric
830 # encryption. If you want to deliver different passphrases for e.g.
831 # several keys or symmetric encryption plus key signing you can use
832 # gpg-agent. Simply make sure that GPG_AGENT_INFO is set in environment.
833 # also see "A NOTE ON SYMMETRIC ENCRYPTION AND SIGNING" in duplicity manpage
834 # notes on en/decryption
835 # private key and passphrase will only be needed for decryption or signing.
836 # decryption happens on restore and incrementals (compare archdir contents).
837 # for security reasons it makes sense to separate the signing key from the
838 # encryption keys. https://answers.launchpad.net/duplicity/+question/107216
839 #GPG_KEYS_ENC='<pubkey1>,<pubkey2>,...'
840 #GPG_KEY_SIGN='<prvkey>'
841 # set if signing key passphrase differs from encryption (key) passphrase
842 # NOTE: available since duplicity 0.6.14, translates to SIGN_PASSPHRASE
843 #GPG_PW_SIGN='<signpass>'
844
845 # uncomment and set a file path or name force duply to use this gpg executable
846 # available in duplicity 0.7.04 and above (currently unreleased 06/2015)
847 #GPG='/usr/local/gpg-2.1/bin/gpg'
848
849 # gpg options passed from duplicity to gpg process (default='')
850 # e.g. "--trust-model pgp|classic|direct|always"
851 # or "--compress-algo=bzip2 --bzip2-compress-level=9"
852 # or "--personal-cipher-preferences AES256,AES192,AES..."
853 # or "--homedir ~/.duply" - keep keyring and gpg settings duply specific
854 # or "--pinentry-mode loopback" - needed for GPG 2.1+ _and_
855 # also enable allow-loopback-pinentry in your .gnupg/gpg-agent.conf
856 #GPG_OPTS=''
857
858 # disable preliminary tests with the following setting
859 #GPG_TEST='disabled'
860
861 # backend, credentials & location of the backup target (URL-Format)
862 # generic syntax is
863 # scheme://[user[:password]@]host[:port]/[/]path
864 # e.g.
865 # sftp://bob:secret@backupserver.com//home/bob/dupbkp
866 # for details and available backends see duplicity manpage, section URL Format
867 # http://duplicity.nongnu.org/duplicity.1.html#sect7
868 # BE AWARE:
869 # some backends (cloudfiles, S3 etc.) need additional env vars to be set to
870 # work properly, read after the TARGET definition for more details.
871 # ATTENTION:
872 # characters other than A-Za-z0-9.-_.~ in the URL have to be
873 # replaced by their url encoded pendants, see
874 # http://en.wikipedia.org/wiki/Url_encoding
875 # if you define the credentials as TARGET_USER, TARGET_PASS below $ME
876 # will try to url_encode them for you if the need arises.
877 TARGET='${DEFAULT_TARGET}'
878 # optionally the username/password can be defined as extra variables
879 # setting them here _and_ in TARGET results in an error
880 # ATTENTION:
881 # there are backends that do not support the user/pass auth scheme.
882 # prominent examples are S3, Azure, Cloudfiles. when in doubt consult the
883 # duplicity manpage. usually there is a NOTE section explaining if and which
884 # env vars should be set.
885 #TARGET_USER='${DEFAULT_TARGET_USER}'
886 #TARGET_PASS='${DEFAULT_TARGET_PASS}'
887 # e.g. for cloud files backend it might look like this (uncomment for use!)
888 #export CLOUDFILES_USERNAME='someuser'
889 #export CLOUDFILES_APIKEY='somekey'
890 #export CLOUDFILES_AUTHURL ='someurl'
891 # the following is an incomplete list (<backend>: comma separated env vars list)
892 # Azure: AZURE_ACCOUNT_NAME, AZURE_ACCOUNT_KEY
893 # Cloudfiles: CLOUDFILES_USERNAME, CLOUDFILES_APIKEY, CLOUDFILES_AUTHURL
894 # Google Cloud Storage: GS_ACCESS_KEY_ID, GS_SECRET_ACCESS_KEY
895 # Pydrive: GOOGLE_DRIVE_ACCOUNT_KEY, GOOGLE_DRIVE_SETTINGS
896 # S3: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
897 # Swift: SWIFT_USERNAME, SWIFT_PASSWORD, SWIFT_AUTHURL,
898 # SWIFT_TENANTNAME OR SWIFT_PREAUTHURL, SWIFT_PREAUTHTOKEN
899
900 # base directory to backup
901 SOURCE='${DEFAULT_SOURCE}'
902
903 # a command that runs duplicity e.g.
904 # shape bandwidth use via trickle
905 # "trickle -s -u 640 -d 5120" # 5Mb up, 40Mb down"
906 #DUPL_PRECMD=""
907
908 # override the used python interpreter, defaults to
909 # - parsed result of duplicity's shebang or 'python2'
910 # e.g. "python2" or "/usr/bin/python2.7"
911 #PYTHON="python"
912
913 # exclude folders containing exclusion file (since duplicity 0.5.14)
914 # Uncomment the following two lines to enable this setting.
915 #FILENAME='.duplicity-ignore'
916 #DUPL_PARAMS="\$DUPL_PARAMS --exclude-if-present '\$FILENAME'"
917
918 # Time frame for old backups to keep, Used for the "purge" command.
919 # see duplicity man page, chapter TIME_FORMATS)
920 #MAX_AGE=1M
921
922 # Number of full backups to keep. Used for the "purgeFull" command.
923 # See duplicity man page, action "remove-all-but-n-full".
924 #MAX_FULL_BACKUPS=1
925
926 # Number of full backups for which incrementals will be kept for.
927 # Used for the "purgeIncr" command.
928 # See duplicity man page, action "remove-all-inc-of-but-n-full".
929 #MAX_FULLS_WITH_INCRS=1
930
931 # activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
932 # forces a full backup if last full backup reaches a specified age, for the
933 # format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
934 # Uncomment the following two lines to enable this setting.
935 #MAX_FULLBKP_AGE=1M
936 #DUPL_PARAMS="\$DUPL_PARAMS --full-if-older-than \$MAX_FULLBKP_AGE "
937
938 # sets duplicity --volsize option (available since v0.4.3.RC7)
939 # set the size of backup chunks to VOLSIZE MB instead of the default 25MB.
940 # VOLSIZE must be number of MB's to set the volume size to.
941 # Uncomment the following two lines to enable this setting.
942 #VOLSIZE=50
943 #DUPL_PARAMS="\$DUPL_PARAMS --volsize \$VOLSIZE "
944
945 # verbosity of output (error 0, warning 1-2, notice 3-4, info 5-8, debug 9)
946 # default is 4, if not set
947 #VERBOSITY=5
948
949 # temporary file space. at least the size of the biggest file in backup
950 # for a successful restoration process. (default is '/tmp', if not set)
951 #TEMP_DIR=/tmp
952
953 # Modifies archive-dir option (since 0.6.0) Defines a folder that holds
954 # unencrypted meta data of the backup, enabling new incrementals without the
955 # need to decrypt backend metadata first. If empty or deleted somehow, the
956 # private key and it's password are needed.
957 # NOTE: This is confidential data. Put it somewhere safe. It can grow quite
958 # big over time so you might want to put it not in the home dir.
959 # default '~/.cache/duplicity/duply_<profile>/'
960 # if set '\${ARCH_DIR}/<profile>'
961 #ARCH_DIR=/some/space/safe/.duply-cache
962
963 # DEPRECATED setting
964 # sets duplicity --time-separator option (since v0.4.4.RC2) to allow users
965 # to change the time separator from ':' to another character that will work
966 # on their system. HINT: For Windows SMB shares, use --time-separator='_'.
967 # NOTE: '-' is not valid as it conflicts with date separator.
968 # ATTENTION: only use this with duplicity < 0.5.10, since then default file
969 # naming is compatible and this option is pending depreciation
970 #DUPL_PARAMS="\$DUPL_PARAMS --time-separator _ "
971
972 # DEPRECATED setting
973 # activates duplicity --short-filenames option, when uploading to a file
974 # system that can't have filenames longer than 30 characters (e.g. Mac OS 8)
975 # or have problems with ':' as part of the filename (e.g. Microsoft Windows)
976 # ATTENTION: only use this with duplicity < 0.5.10, later versions default file
977 # naming is compatible and this option is pending depreciation
978 #DUPL_PARAMS="\$DUPL_PARAMS --short-filenames "
979
980 # more duplicity command line options can be added in the following way
981 # don't forget to leave a separating space char at the end
982 #DUPL_PARAMS="\$DUPL_PARAMS --put_your_options_here "
983
984 EOF
985
986 # create initial exclude file
987 cat <<EOF >"$EXCLUDE"
988 # although called exclude, this file is actually a globbing file list
989 # duplicity accepts some globbing patterns, even including ones here
990 # here is an example, this incl. only 'dir/bar' except it's subfolder 'foo'
991 # - dir/bar/foo
992 # + dir/bar
993 # - **
994 # for more details see duplicity manpage, section File Selection
995 # http://duplicity.nongnu.org/duplicity.1.html#sect9
996
997 EOF
998
999 # Hints on first usage
1000 cat <<EOF
1001
1002 Congratulations. You just created the profile '$FTPLCFG'.
1003 The initial config file has been created as
1004 '$CONF'.
1005 You should now adjust this config file to your needs.
1006
1007 $(hint_profile)
1008
1009 EOF
1010 fi
1011
1012 }
1013
1014 # used in usage AND create_config
1015 function hint_profile {
1016 cat <<EOF
1017 IMPORTANT:
1018 Copy the _whole_ profile folder after the first backup to a safe place.
1019 It contains everything (duply related) needed to restore your backups.
1020
1021 Pay attention to (possibly later added) external files such as credentials
1022 or auth files (e.g. netrc, .megarc, ssh keys) or environment variables
1023 (e.g. DPBX_ACCESS_TOKEN).
1024 It is good policy to place those in the profile folder if possible at all.
1025 e.g. in case of 'multi://' target the config .json file
1026 Env vars should be added to duply profiles' conf file.
1027
1028 Keep access to these files restricted as they contain information (gpg key,
1029 passphrases etc.) to access and modify your backups.
1030
1031 Finally:
1032 You should attempt a restore from an unrelated host to be sure you really
1033 have everything needed for restoration.
1034
1035 Repeat these steps after _all_ configuration changes. Some configuration
1036 options are crucial for restoration.
1037
1038 EOF
1039 }
1040
1041 function separator {
1042 echo "--- $@ ---"
1043 }
1044
1045 function inform {
1046 echo -e "\nINFO:\n\n$@\n"
1047 }
1048
1049 function warning {
1050 echo -e "\nWARNING:\n\n$@\n"
1051 }
1052
1053 function warning_oldhome {
1054 local old=$1 new=$2
1055 warning " ftplicity changed name to duply since you created your profiles.
1056 Please rename the old folder
1057 '$old'
1058 to
1059 '$new'
1060 and this warning will disappear.
1061 If you decide not to do so profiles will _only_ work from the old location."
1062 }
1063
1064 function error_print {
1065 echo -e "$@" >&2
1066 }
1067
1068 function error {
1069 error_print "\nSorry. A fatal ERROR occured:\n\n$@\n"
1070 exit -1
1071 }
1072
1073 function error_gpg {
1074 [ -n "$2" ] && local hint="\n $2\n\n "
1075
1076 error "$1
1077
1078 Hint${hint:+s}:
1079 ${hint}Maybe you have not created a gpg key yet (e.g. gpg --gen-key)?
1080 Don't forget the used _password_ as you will need it.
1081 When done enter the 8 digit id & the password in the profile conf file.
1082
1083 The key id can be found doing a 'gpg --list-keys'. In the example output
1084 below the key id would be FFFFFFFF for the public key.
1085
1086 pub 1024D/FFFFFFFF 2007-12-17
1087 uid duplicity
1088 sub 2048g/899FE27F 2007-12-17
1089 "
1090 }
1091
1092 function error_gpg_key {
1093 local KEY_ID="$1"
1094 local KIND="$2"
1095 error_gpg "${KIND} gpg key '${KEY_ID}' cannot be found." \
1096 "Doublecheck if the above key is listed by 'gpg --list-keys' or available
1097 as gpg key file '$(basename "$(gpg_keyfile "${KEY_ID}")")' in the profile folder.
1098 If not you can put it there and $ME_NAME will autoimport it on the next run.
1099 Alternatively import it manually as the user you plan to run $ME_NAME with."
1100 }
1101
1102 function error_gpg_test {
1103 [ -n "$2" ] && local hint="\n $2\n\n "
1104
1105 error "$1
1106
1107 Hint${hint:+s}:
1108 ${hint}This error means that gpg is probably misconfigured or not working
1109 correctly. The error message above should help to solve the problem.
1110 However, if for some reason $ME_NAME should misinterpret the situation you
1111 can define GPG_TEST='disabled' in the conf file to bypass the test.
1112 Please do not forget to report the bug in order to resolve the problem
1113 in future versions of $ME_NAME.
1114 "
1115 }
1116
1117 function error_path {
1118 error "$@
1119 PATH='$PATH'
1120 "
1121 }
1122
1123 function error_to_string {
1124 [ -n "$1" ] && [ "$1" -eq 0 ] && echo "OK" || echo "FAILED 'code $1'"
1125 }
1126
1127 function duplicity_version_get {
1128 # nothing to do, just print
1129 var_isset DUPL_VERSION && return
1130
1131 local DUPL_VERSION_OUT DUPL_VERSION_AWK PYTHON_BIN CMD='duplicity'
1132 # only run with a user specific python if configured (running by default
1133 # breaks homebrew as they place a shell wrapper for duplicity in path)
1134 PYTHON_BIN="$(python_binary)" &&\
1135 CMD="$(qw "$PYTHON_BIN") $(which $CMD)"
1136 CMD="$CMD --version 2>&1"
1137 DUPL_VERSION_OUT=$(eval "$CMD")
1138 DUPL_VERSION=`echo $DUPL_VERSION_OUT | awk '/^duplicity /{print $2; exit;}'`
1139 #DUPL_VERSION='0.7.03' #'0.6.08b' #,0.4.4.RC4,0.6.08b
1140 DUPL_VERSION_VALUE=0
1141 DUPL_VERSION_AWK=$(awk -v v="$DUPL_VERSION" 'BEGIN{
1142 if (match(v,/[^\.0-9]+[0-9]*$/)){
1143 rest=substr(v,RSTART,RLENGTH);v=substr(v,0,RSTART-1);}
1144 if (pos=match(rest,/RC([0-9]+)$/)) rc=substr(rest,pos+2)
1145 split(v,f,"[. ]"); if(f[1]f[2]f[3]~/^[0-9]+$/) vvalue=f[1]*10000+f[2]*100+f[3]; else vvalue=0
1146 print "#"v"_"rest"("rc"):"f[1]"-"f[2]"-"f[3]
1147 print "DUPL_VERSION_VALUE=\047"vvalue"\047"
1148 print "DUPL_VERSION_RC=\047"rc"\047"
1149 print "DUPL_VERSION_SUFFIX=\047"rest"\047"
1150 }')
1151 eval "$DUPL_VERSION_AWK"
1152 #echo -e ",$DUPL_VERSION,$DUPL_VERSION_VALUE,$DUPL_VERSION_RC,$DUPL_VERSION_SUFFIX,"
1153
1154 # doublecheck findings and report error
1155 if [ $DUPL_VERSION_VALUE -eq 0 ]; then
1156 inform "duplicity version check failed (please report, this is a bug)
1157 the command
1158 $CMD
1159 resulted in
1160 $DUPL_VERSION_OUT
1161 "
1162 elif [ $DUPL_VERSION_VALUE -le 404 ] && [ ${DUPL_VERSION_RC:-4} -lt 4 ]; then
1163 error "The installed version $DUPL_VERSION is incompatible with $ME_NAME v$ME_VERSION.
1164 You should upgrade your version of duplicity to at least v0.4.4RC4 or
1165 use the older ftplicity version 1.1.1 from $ME_WEBSITE."
1166 fi
1167 }
1168
1169 function duplicity_version_ge {
1170 [ "$DUPL_VERSION_VALUE" -ge "$1" ]
1171 }
1172
1173 function duplicity_version_lt {
1174 ! duplicity_version_ge "$1"
1175 }
1176
1177 # parse interpreter from duplicity shebang
1178 function duplicity_python_binary_parse {
1179 # cached result
1180 ( var_isset 'PYTHON' || var_isset 'DUPL_PYTHON_BIN' ) && return
1181
1182 # parse it or warn
1183 local DUPL_BIN=$(which duplicity)
1184 DUPL_PYTHON_BIN=$(awk 'NR==1&&/^#!/{sub(/^#!( *\/usr\/bin\/env *)?/,""); print}' < "$DUPL_BIN")
1185 if ! echo "$DUPL_PYTHON_BIN" | grep -q -i 'python'; then
1186 warning "Could not parse the python interpreter used from duplicity ($DUPL_BIN). Result was '$DUPL_PYTHON_BIN'.
1187 Will assume it is '$DEFAULT_PYTHON'."
1188 DUPL_PYTHON_BIN="$DEFAULT_PYTHON"
1189 fi
1190 }
1191
1192 function run_script { # run pre/post scripts
1193 local ERR=0
1194 local SCRIPT="$1"
1195 if [ ! -z "$PREVIEW" ] ; then
1196 echo "$([ ! -x "$SCRIPT" ] && echo ". ")$SCRIPT"
1197 elif [ -r "$SCRIPT" ] ; then
1198 echo -n "Running '$SCRIPT' "
1199 if [ -x "$SCRIPT" ]; then
1200 OUT=$("$SCRIPT" 2>&1)
1201 ERR=$?
1202 else
1203 OUT=$(. "$SCRIPT" 2>&1)
1204 ERR=$?
1205 fi
1206 [ $ERR -eq "0" ] && echo "- OK" || echo "- FAILED (code $ERR)"
1207 echo -en ${OUT:+"Output: $OUT\n"} ;
1208 else
1209 echo "Skipping n/a script '$SCRIPT'."
1210 fi
1211 return $ERR
1212 }
1213
1214 function run_cmd {
1215 # run or print escaped cmd string
1216 local CMD_ERR=0
1217 if [ -n "$PREVIEW" ]; then
1218 CMD_OUT=$( echo "$@ 2>&1" )
1219 CMD_MSG="-- Run cmd -- $CMD_MSG --\n$CMD_OUT"
1220 elif [ -n "$CMD_DISABLED" ]; then
1221 CMD_MSG="$CMD_MSG (DISABLED) - $CMD_DISABLED"
1222 else
1223 echo -n -e "$CMD_MSG"
1224 CMD_OUT=` eval "$@" 2>&1 `
1225 CMD_ERR=$?
1226 if [ "$CMD_ERR" = "0" ]; then
1227 CMD_MSG=" (OK)"
1228 else
1229 CMD_MSG=" (FAILED)"
1230 fi
1231 fi
1232 echo -e "$CMD_MSG"
1233 # reset
1234 unset CMD_DISABLED CMD_MSG
1235 return $CMD_ERR
1236 }
1237
1238 function qw { quotewrap "$@"; }
1239
1240 function quotewrap {
1241 local param="$@"
1242 # quote strings having non word chars (e.g. spaces)
1243 if echo "$param" | awk '/[^A-Za-z0-9_\.\-\/]/{exit 0}{exit 1}'; then
1244 echo "$param" | awk '{\
1245 gsub(/[\047]/,"\047\\\047\047",$0);\
1246 gsub(/[\042]/,"\047\\\042\047",$0);\
1247 print "\047"$0"\047"}'
1248 return
1249 fi
1250 echo $param
1251 }
1252
1253 function duplicity_params_global {
1254 # already done? return
1255 var_isset 'DUPL_PARAMS_GLOBAL' && return
1256 local DUPL_ARG_ENC
1257
1258 # use key only if set in config, else leave it to symmetric encryption
1259 if gpg_disabled; then
1260 local DUPL_PARAM_ENC='--no-encryption'
1261 else
1262 local DUPL_PARAM_ENC=$(gpg_prefix_keyset '--encrypt-key' "${GPG_KEYS_ENC_ARRAY[@]}")
1263 gpg_signing && local DUPL_PARAM_SIGN=$(gpg_prefix_keyset '--sign-key' "$GPG_KEY_SIGN")
1264 # interpret password settings
1265 var_isset 'GPG_PW' && DUPL_ARG_ENC="PASSPHRASE=$(qw "${GPG_PW}")"
1266 var_isset 'GPG_PW_SIGN' && DUPL_ARG_ENC="${DUPL_ARG_ENC} SIGN_PASSPHRASE=$(qw "${GPG_PW_SIGN}")"
1267 fi
1268
1269 local GPG_OPTS=${GPG_OPTS:+"--gpg-options $(qw "${GPG_OPTS}")"}
1270
1271 # set name for dupl archive folder, since 0.6.0
1272 if duplicity_version_ge 601; then
1273 local DUPL_ARCHDIR=''
1274 if var_isset 'ARCH_DIR'; then
1275 DUPL_ARCHDIR="--archive-dir $(qw "${ARCH_DIR}")"
1276 # reuse erronously duply_ prefixed folders from bug #117
1277 if [ -d "$ARCH_DIR/duply_${PROFILE}" ]; then
1278 DUPL_ARCHDIR="${DUPL_ARCHDIR} --name $(qw "duply_${PROFILE}")"
1279 else
1280 DUPL_ARCHDIR="${DUPL_ARCHDIR} --name $(qw "${PROFILE}")"
1281 fi
1282 else
1283 DUPL_ARCHDIR="--name $(qw "duply_${PROFILE}")"
1284 fi
1285 fi
1286
1287 DUPL_PARAMS_GLOBAL="${DUPL_ARCHDIR} ${DUPL_PARAM_ENC} \
1288 ${DUPL_PARAM_SIGN} --verbosity '${VERBOSITY:-4}' \
1289 ${GPG_OPTS}"
1290
1291 DUPL_VARS_GLOBAL="TMPDIR='$TEMP_DIR' \
1292 ${DUPL_ARG_ENC}"
1293 }
1294
1295 # function to filter the DUPL_PARAMS var from user conf
1296 function duplicity_params_conf {
1297 # reuse cmd var from main loop
1298 ## in/exclude parameters are currently not supported on restores
1299 if [ "$cmd" = "fetch" ] || [ "$cmd" = "restore" ] || [ "$cmd" = "status" ]; then
1300 # filter exclude params from fetch/restore/status
1301 eval "stripXcludes $DUPL_PARAMS"
1302 return
1303 fi
1304
1305 # nothing done, print unchanged
1306 echo "$DUPL_PARAMS"
1307 }
1308
1309 # strip in/exclude parameters from param string
1310 function stripXcludes {
1311 local STRIPNEXT OUT;
1312 for p in "$@"; do
1313 if [ -n "$STRIPNEXT" ]; then
1314 unset STRIPNEXT
1315 # strip the value of previous parameter
1316 continue
1317 elif echo "$p" | awk '/^\-\-(in|ex)clude(\-[a-zA-Z]+)?$/{exit 0;}{exit 1;}'; then
1318 # strips e.g. --include /foo/bar
1319 STRIPNEXT="yes"
1320 continue
1321 elif echo "$p" | awk '/^\-\-(in|ex)clude(\-[a-zA-Z]+)?=/{exit 0;}{exit 1;}'; then
1322 # strips e.g. --include=/foo/bar
1323 continue
1324 fi
1325
1326 OUT="$OUT $(qw "$p")"
1327 done
1328 echo "$OUT"
1329 }
1330
1331 function duplify { # the actual wrapper function
1332 local PARAMSNOW DUPL_CMD DUPL_CMD_PARAMS
1333
1334 # put command (with params) first in duplicity parameters
1335 for param in "$@" ; do
1336 # split cmd from params (everything before splitchar --)
1337 if [ "$param" == "--" ] ; then
1338 PARAMSNOW=1
1339 else
1340 # wrap in quotes to protect from spaces
1341 [ ! $PARAMSNOW ] && \
1342 DUPL_CMD="$DUPL_CMD $(qw "$param")" \
1343 || \
1344 DUPL_CMD_PARAMS="$DUPL_CMD_PARAMS $(qw "$param")"
1345 fi
1346 done
1347
1348 # init global duplicity parameters same for all tasks
1349 duplicity_params_global
1350
1351 local RUN=eval BIN=duplicity DUPL_BIN PYTHON_BIN
1352 # run in cmd line preview mode if requested
1353 var_isset 'PREVIEW' && RUN=echo
1354 # try to resolve duplicity path for usage with python interpreter
1355 DUPL_BIN=$(which "$BIN") || DUPL_BIN="$BIN"
1356 # only run with a user specific python if configured (running by default
1357 # breaks homebrew as they place a shell wrapper for duplicity in path)
1358 PYTHON_BIN="$(python_binary)" &&\
1359 BIN="$(qw "$PYTHON_BIN") $(qw "$DUPL_BIN")"
1360
1361 $RUN "${DUPL_VARS_GLOBAL} ${BACKEND_PARAMS}\
1362 ${DUPL_PRECMD} $BIN $DUPL_CMD $DUPL_PARAMS_GLOBAL $(duplicity_params_conf)\
1363 $GPG_USEAGENT $(gpg_custom_binary) $DUPL_CMD_PARAMS"
1364
1365 local ERR=$?
1366 return $ERR
1367 }
1368
1369 function secureconf { # secure the configuration dir
1370 #PERMS=$(ls -la $(dirname $CONFDIR) | grep -e " $(basename $CONFDIR)\$" | awk '{print $1}')
1371 local PERMS="$(ls -la "$CONFDIR/." | awk 'NR==2{print $1}')"
1372 if [ "${PERMS/#drwx------*/OK}" != 'OK' ] ; then
1373 chmod u+rwX,go= "$CONFDIR"; local ERR=$?
1374 warning "The profile's folder
1375 '$CONFDIR'
1376 permissions are not safe ($PERMS). Secure them now. - ($(error_to_string $ERR))"
1377 fi
1378 }
1379
1380 # params are $1=timeformatstring (default like date output), $2=epoch seconds since 1.1.1970 (default now)
1381 function date_fix {
1382 local DEFAULTFORMAT='%a %b %d %H:%M:%S %Z %Y'
1383 local date
1384 #[ "$1" == "%N" ] && return #test the no nsec test below
1385 # gnu date with -d @epoch
1386 date=$(date ${2:+-d @$2} ${1:++"$1"} 2> /dev/null) && \
1387 echo $date && return
1388 # date bsd,osx with -r epoch
1389 date=$(date ${2:+-r $2} ${1:++"$1"} 2> /dev/null) && \
1390 echo $date && return
1391 # date busybox with -d epoch -D %s
1392 date=$(date ${2:+-d $2 -D %s} ${1:++"$1"} 2> /dev/null) && \
1393 echo $date && return
1394 ## some date commands do not support giving a time w/o setting it systemwide (irix,solaris,others?)
1395 # python fallback
1396 date=$("$(python_binary)" -c "import time;print time.strftime('${1:-$DEFAULTFORMAT}',time.localtime(${2}))" 2> /dev/null) && \
1397 echo $date && return
1398 # awk fallback
1399 date=$(awk "BEGIN{print strftime(\"${1:-$DEFAULTFORMAT}\"${2:+,$2})}" 2> /dev/null) && \
1400 echo $date && return
1401 # perl fallback
1402 date=$(perl -e "use POSIX qw(strftime);\$date = strftime(\"${1:-$DEFAULTFORMAT}\",localtime(${2}));print \"\$date\n\";" 2> /dev/null) && \
1403 echo $date && return
1404 # error
1405 echo "ERROR"
1406 return 1
1407 }
1408
1409 function nsecs {
1410 local NSECS
1411 # test if date supports nanosecond output
1412 if ! var_isset NSECS_DISABLED; then
1413 NSECS=$(date_fix %N 2> /dev/null | head -1 |grep -e "^[[:digit:]]\{9\}$")
1414 [ -n "$NSECS" ] && NSECS_DISABLED=0 || NSECS_DISABLED=1
1415 fi
1416
1417 # add 9 digits, not all date(s) deliver nsecs e.g. busybox date
1418 if [ "$NSECS_DISABLED" == "1" ]; then
1419 date_fix %s000000000
1420 else
1421 date_fix %s%N
1422 fi
1423 }
1424
1425 function nsecs_to_sec {
1426 echo $(($1/1000000000)).$(printf "%03d" $(($1/1000000%1000)) )
1427 }
1428
1429 function datefull_from_nsecs {
1430 date_from_nsecs $1 '%F %T'
1431 }
1432
1433 function date_from_nsecs {
1434 local FORMAT=${2:-%T}
1435 local TIME=$(nsecs_to_sec $1)
1436 local SECS=${TIME%.*}
1437 local DATE=$(date_fix "%T" ${SECS:-0})
1438 echo $DATE.${TIME#*.}
1439 }
1440
1441 function var_isset {
1442 if [ -z "$1" ]; then
1443 echo "ERROR: function var_isset needs a string as parameter"
1444 elif eval "[ \"\${$1}\" == 'not_set' ]" || eval "[ \"\${$1-not_set}\" != 'not_set' ]"; then
1445 return 0
1446 fi
1447 return 1
1448 }
1449
1450 function is_condition {
1451 local CMD=$(tolower "$1")
1452 [ "$CMD" == 'and' ] || [ "$CMD" == 'or' ]
1453 }
1454
1455 function is_groupMarker {
1456 local CMD=$(tolower "$1")
1457 [ "$CMD" == 'groupin' ] || [ "$CMD" == 'groupout' ]
1458 }
1459
1460 function is_command {
1461 local CMD=$(tolower "$1")
1462 ! is_condition "$CMD" && ! is_groupMarker "$CMD"
1463 }
1464
1465 function url_encode {
1466 # utilize python, silently do nothing on error - because no python no duplicity
1467 OUT=$("$(python_binary)" -c "
1468 try: import urllib.request as urllib
1469 except ImportError: import urllib
1470 print(urllib.${2}quote('$1'));
1471 " 2>/dev/null ); ERR=$?
1472 [ "$ERR" -eq 0 ] && echo $OUT || echo $1
1473 }
1474
1475 function url_decode {
1476 # reuse function above with a simple string param hack
1477 url_encode "$1" "un"
1478 }
1479
1480 function toupper {
1481 echo "$@"|awk '$0=toupper($0)'
1482 }
1483
1484 function tolower {
1485 echo "$@"|awk '$0=tolower($0)'
1486 }
1487
1488 function isnumber {
1489 case "$*" in
1490 ''|*[!0-9]*) return 1;;
1491 *) return 0;;
1492 esac
1493 }
1494
1495 #function tmp_space {
1496 #
1497 # if ! isnumber $VOLSIZE; then
1498 # inform "failed to determine free space (please report, this is a bug)"
1499 # return
1500 # fi
1501 #
1502 # get free temp space
1503 # TEMP_FREE="$(df -P -k "$TEMP_DIR" 2>/dev/null | awk 'END{pos=(NF-2);if(pos>0) print $pos;}')"
1504 # # check for free space or FAIL
1505 # if [ $((${TEMP_FREE:-0}-${VOLSIZE:-0}*1024)) -lt 0-lt 0 ]; then
1506 # error "Temporary file space '$TEMP_DIR' free space is smaller ($((TEMP_FREE/1024))MB)
1507 #than one duplicity volume (${VOLSIZE}MB).
1508 #
1509 # Hint: Free space or change TEMP_DIR setting."
1510 #fi
1511 #
1512 #}
1513
1514 function gpg_disabled {
1515 echo "${GPG_KEY}" | grep -iq -e '^disabled$'
1516 }
1517
1518 # usage: join SEPARATOR "entry1" "entry2"
1519 function join {
1520 local SEP="$1" ENTRY OUT; shift;
1521 for ENTRY in "$@"; do
1522 ENTRY=${ENTRY//$SEP/\\$SEP}
1523 [ -z "$OUT" ] && OUT=$ENTRY || OUT="$OUT$SEP$ENTRY"
1524 done
1525 echo $OUT
1526 }
1527
1528 function gpg_signing {
1529 echo ${GPG_KEY_SIGN} | grep -v -q -e '^disabled$'
1530 }
1531
1532 # parameter key id, key_type
1533 function gpg_keyfile {
1534 local GPG_KEY=$(gpg_key_legalize $1) TYPE="$2"
1535 local KEYFILE="${KEYFILE//.asc/${GPG_KEY:+.$GPG_KEY}.asc}"
1536 echo "${KEYFILE//.asc/${TYPE:+.$(tolower $TYPE)}.asc}"
1537 }
1538
1539 # parameter key id
1540 function gpg_import {
1541 local i FILE FOUND=0 KEY_ID="$1" KEY_TYPE="$2" KEY_FP="" ERR=0
1542 # create a list of legacy key file names and current naming scheme
1543 # we always import pub and sec if they are avail in conf folder
1544 local KEYFILES=( "$CONFDIR/gpgkey" $(gpg_keyfile "$KEY_ID") \
1545 $(gpg_keyfile "$KEY_ID" PUB) $(gpg_keyfile "$KEY_ID" SEC))
1546
1547 # Try autoimport from existing old gpgkey files
1548 # and new gpgkey.XXX.asc files (since v1.4.2)
1549 # and even newer gpgkey.XXX.[pub|sec].asc
1550 for (( i = 0 ; i < ${#KEYFILES[@]} ; i++ )); do
1551 FILE=${KEYFILES[$i]}
1552 if [ -f "$FILE" ]; then
1553 FOUND=1
1554
1555 CMD_MSG="Import keyfile '$FILE' to keyring"
1556 run_cmd gpg $GPG_OPTS --batch --import $(qw "$FILE")
1557 if [ "$?" != "0" ]; then
1558 warning "Import failed.${CMD_OUT:+\n$CMD_OUT}"
1559 ERR=1
1560 # continue with next
1561 continue
1562 fi
1563 fi
1564 done
1565
1566 if [ "$FOUND" -eq 0 ]; then
1567 warning "No keyfile for '$KEY_ID' found in profile\n'$CONFDIR'."
1568 fi
1569
1570 # try to set trust automagically
1571 CMD_MSG="Autoset trust of key '$KEY_ID' to ultimate"
1572 run_cmd echo $(gpg_fingerprint "$KEY_ID"):6: \| gpg $GPG_OPTS --import-ownertrust --batch --logger-fd 1
1573 if [ "$?" = "0" ] && [ -z "$PREVIEW" ]; then
1574 # success on all levels, we're done
1575 return $ERR
1576 fi
1577
1578 # failover: user has to set trust manually
1579 echo -e "For $ME_NAME to work you have to set the trust level
1580 with the command \"trust\" to \"ultimate\" (5) now.
1581 Exit the edit mode of gpg with \"quit\"."
1582 CMD_MSG="Running gpg to manually edit key '$KEY_ID'"
1583 run_cmd sleep 5\; gpg $GPG_OPTS --edit-key $(qw "$KEY_ID")
1584
1585 return $ERR
1586 }
1587
1588 # see 'How to specify a user ID' on gpg manpage
1589 function gpg_fingerprint {
1590 gpg $GPG_OPTS --fingerprint "$1" 2>&1 | \
1591 awk 'NR==2{sub(/^.*=/,"");gsub(/[ \t]/,""); if ( $0 !~ /^[A-F0-9]+$/ || length($0) != 40 ) exit 1; print}'
1592 }
1593
1594 function gpg_export_if_needed {
1595 local SUCCESS FILE KEY_TYPE
1596 local TMPFILE="$TEMP_DIR/${ME_NAME}.$$.$(date_fix %s).gpgexp"
1597 for KEY_ID in "$@"; do
1598 # check if already exported, do it if not
1599 for KEY_TYPE in PUB SEC; do
1600 FILE="$(gpg_keyfile "$KEY_ID" $KEY_TYPE)"
1601 if [ ! -f "$FILE" ] && eval gpg_$(tolower $KEY_TYPE)_avail \"$KEY_ID\"; then
1602 # exporting
1603 CMD_MSG="Backup $KEY_TYPE key '$KEY_ID' to profile."
1604 # gpg2.1 insists on passphrase here, gpg2.0- happily exports w/o it
1605 # we pipe an empty string when GPG_PW is not set to avoid gpg silently waiting for input
1606 run_cmd $(gpg_pass_pipein GPG_PW_SIGN GPG_PW) gpg $GPG_OPTS $GPG_USEAGENT $(gpg_param_passwd GPG_PW_SIGN GPG_PW) --armor --export"$(test "SEC" = "$KEY_TYPE" && echo -secret-keys)" $(qw "$KEY_ID") '>>' $(qw "$TMPFILE")
1607 CMD_ERR=$?
1608
1609 if [ "$CMD_ERR" = "0" ]; then
1610 CMD_MSG="Write file '"$(basename "$FILE")"'"
1611 run_cmd mv $(qw "$TMPFILE") $(qw "$FILE")
1612 fi
1613
1614 if [ "$CMD_ERR" != "0" ]; then
1615 warning "Backup failed.${CMD_OUT:+\n$CMD_OUT}"
1616 else
1617 SUCCESS=1
1618 fi
1619
1620 # cleanup
1621 rm $(qw "$TMPFILE") 1>/dev/null 2>&1
1622 fi
1623 done
1624 done
1625
1626 [ -n "$SUCCESS" ] && inform "$ME_NAME exported new keys to your profile.
1627 You should backup your changed profile folder now and store it in a safe place."
1628 }
1629
1630 # replace all non-alnum chars with underscore (for file operations)
1631 function gpg_key_legalize {
1632 echo $* | awk '{gsub(/[^a-zA-Z0-9]/,"_",$0); print}'
1633 }
1634
1635 function gpg_key_cache {
1636 local RES
1637 local MODE=$1
1638 shift
1639 local PREFIX="GPG_KEY"
1640 local SUFFIX=$(gpg_key_legalize "$@")
1641 local KEYID="$*"
1642 local CACHE="${PREFIX}_${MODE}_${SUFFIX}"
1643 if [ "$MODE" = "RESET" ]; then
1644 eval unset ${PREFIX}_PUB_$SUFFIX ${PREFIX}_SEC_$SUFFIX
1645 return 255
1646 elif ! var_isset "$CACHE"; then
1647 if [ "$MODE" = "PUB" ]; then
1648 RES=$(gpg $GPG_OPTS --list-key "$KEYID" > /dev/null 2>&1; echo -n $?)
1649 elif [ "$MODE" = "SEC" ]; then
1650 RES=$(gpg $GPG_OPTS --list-secret-key "$KEYID" > /dev/null 2>&1; echo -n $?)
1651 else
1652 return 255
1653 fi
1654 eval $CACHE=$RES
1655 fi
1656 eval return \$$CACHE
1657 }
1658
1659 function gpg_pub_avail {
1660 gpg_key_cache PUB "$@"
1661 }
1662
1663 function gpg_sec_avail {
1664 gpg_key_cache SEC "$@"
1665 }
1666
1667 function gpg_key_format {
1668 echo $1 | grep -q '^[0-9a-fA-F]\{8\}$'
1669 }
1670
1671 # splits a comma separated line into lines, respects escaped commas
1672 function gpg_split_keyset {
1673 local LIST
1674 LIST=$(echo "$@" | awk '{ gsub(/,/,"\n",$0); gsub(/\\\n/,",",$0); print $0 }')
1675 echo -e "$LIST"
1676 }
1677
1678 function gpg_prefix_keyset {
1679 local PREFIX="$1" OUT=""
1680 shift
1681 for KEY_ID in "$@"; do
1682 OUT="${OUT} $PREFIX $(qw ${KEY_ID})"
1683 done
1684 echo $OUT
1685 }
1686
1687 # grep a variable from conf text file (currently not used)
1688 function gpg_passwd {
1689 [ -r "$CONF" ] && \
1690 awk '/^[ \t]*GPG_PW[ \t=]/{\
1691 sub(/^[ \t]*GPG_PW[ \t]*=*/,"",$0);\
1692 gsub(/^[ \t]*[\047"]|[\047"][ \t]*$/,"",$0);\
1693 print $0; exit}' "$CONF"
1694 }
1695
1696 # return success if at least one secret key is available
1697 function gpg_key_decryptable {
1698 local KEY_ID
1699 for KEY_ID in "${GPG_KEYS_ENC_ARRAY[@]}"; do
1700 gpg_sec_avail "$KEY_ID" && return 0
1701 done
1702 return 1
1703 }
1704
1705 function gpg_symmetric {
1706 [ -z "${GPG_KEY}${GPG_KEYS_ENC_ARRAY}" ]
1707 }
1708
1709 # checks for max two params if they are set, typically GPG_PW & GPG_PW_SIGN
1710 function gpg_param_passwd {
1711 var_isset GPG_USEAGENT && exit 1
1712
1713 if ( [ -n "$1" ] && var_isset "$1" ) || ( [ -n "$2" ] && var_isset "$2" ); then
1714 echo "--passphrase-fd 0 --batch"
1715 fi
1716 }
1717
1718 # select the earlist defined and create an "echo <value> |" string
1719 function gpg_pass_pipein {
1720 var_isset GPG_USEAGENT && exit 1
1721
1722 for var in "$@"
1723 do
1724 if var_isset "$var"; then
1725 echo "echo $(qw $(eval echo \$$var)) |"
1726 return 0
1727 fi
1728 done
1729
1730 return 1
1731 }
1732
1733 # checks if gpg-agent is available, returns error code
1734 # 0 on success
1735 # 1 if GPG_AGENT_INFO is not set (unused, should probably be merged w/ 3)
1736 # 2 if GPG_AGENT_INFO is stale
1737 # 3 cannot connect to gpg-agent
1738 function gpg_agent_avail {
1739 # GPG_AGENT_INFO is deprecated in gpg2.1,
1740 # first try to connect to a possibly running agent here
1741 local ERR=3
1742 gpg-agent > /dev/null 2>&1 && return 0
1743
1744 # detect stale pid in legacy GPG_AGENT_INFO env var
1745 if var_isset GPG_AGENT_INFO; then
1746 # check if a pid matching process is running at all
1747 local GPG_AGENT_PID=$(echo $GPG_AGENT_INFO|awk -F: '{print $2}')
1748 if isnumber "$GPG_AGENT_PID"; then
1749 ps -p "$GPG_AGENT_PID" > /dev/null 2>&1 || ERR=2
1750 fi
1751 fi
1752
1753 return $ERR
1754 }
1755
1756 function gpg_custom_binary {
1757 var_isset GPG && [ "$GPG" != "$DEFAULT_GPG" ] &&\
1758 echo "--gpg-binary $(qw "$GPG")"
1759 }
1760
1761 function gpg_binary {
1762 local BIN
1763 var_isset GPG && BIN="$GPG" || BIN="$DEFAULT_GPG"
1764 echo "$BIN"
1765 }
1766
1767 function gpg_avail {
1768 lookup "$(gpg_binary)"
1769 }
1770
1771 # enforce the use of our selected gpg binary
1772 function gpg {
1773 command "$(gpg_binary)" "$@"
1774 }
1775 export -f gpg
1776
1777 # start of script #######################################################################
1778
1779 # confidentiality first, all we create is only readable by us
1780 umask 077
1781
1782 # check if ftplicity is there & executable
1783 [ -n "$ME_LONG" ] && [ -x "$ME_LONG" ] || error "$ME missing. Executable & available in path? ($ME_LONG)"
1784
1785 if [ ${#@} -eq 1 ]; then
1786 cmd="${1}"
1787 else
1788 FTPLCFG="${1}" ; cmd="${2}"
1789 fi
1790
1791 # deal with command before profile validation calls
1792 # show requested version
1793 # OR requested usage info
1794 # OR create a profile
1795 # OR fall through
1796 ##if [ ${#@} -le 2 ]; then
1797 case "$cmd" in
1798 changelog)
1799 changelog
1800 exit 0
1801 ;;
1802 create)
1803 set_config
1804 if [ -d "$CONFDIR" ]; then
1805 error "The profile '$FTPLCFG' already exists in
1806 '$CONFDIR'.
1807
1808 Hint:
1809 If you _really_ want to create a new profile by this name you will
1810 have to manually delete the existing profile folder first."
1811 exit 1
1812 else
1813 create_config
1814 exit 0
1815 fi
1816 ;;
1817 txt2man)
1818 set_config
1819 usage_txt2man
1820 exit 0
1821 ;;
1822 usage|-help|--help|-h|-H)
1823 set_config
1824 usage_info
1825 exit 0
1826 ;;
1827 version|-version|--version|-v|-V)
1828 # profile can override GPG/PYTHON, so import it if it was given
1829 var_isset FTPLCFG && {
1830 set_config
1831 [ -r "$CONF" ] && . "$CONF" || warning "Cannot import config '$CONF'."
1832 }
1833 version_info_using
1834 exit 0
1835 ;;
1836 # fallthrough.. we got a command that needs an existing profile
1837 *)
1838 # if we reach here, user either forgot profile or chose wrong profileless command
1839 if [ ${#@} -le 1 ]; then
1840 error "\
1841 Missing or wrong parameters.
1842 Only the commands
1843 changelog, create, usage, txt2man, version
1844 can be called without selecting an existing profile first.
1845 Your command was '$cmd'.
1846
1847 Hint: Run '$ME usage' to get help."
1848 fi
1849 esac
1850
1851
1852 # Hello world
1853 echo "Start $ME v$ME_VERSION, time is $(date_fix '%F %T')."
1854
1855 # check system environment
1856
1857 # is duplicity avail
1858 lookup duplicity || error_path "duplicity missing. installed und available in path?"
1859 # init, exec duplicity version check info
1860 duplicity_python_binary_parse
1861 duplicity_version_get
1862
1863 # check for certain important helper programs
1864 for f in awk grep "$(python_binary)"; do
1865 lookup "$f" || \
1866 error_path "$f missing. installed und available in path?"
1867 done
1868
1869 ### read configuration
1870 set_config
1871 # check validity
1872 if [ ! -d "$CONFDIR" ]; then
1873 error "Selected profile '$FTPLCFG' does not resolve to a profile folder in
1874 '$CONFDIR'.
1875
1876 Hints:
1877 Select one of the available profiles: $(for d in "$(dirname "$CONFDIR")"/*/; do [ -e "$d" ] || [ -L "$d" ] || continue; printf "$sep'$(basename "$d")'"; sep=",";done)
1878 Use '$ME <name> create' to create a new profile.
1879 Use '$ME usage' to get usage help."
1880 elif [ ! -x "$CONFDIR" ]; then
1881 error "\
1882 Profile folder in '$CONFDIR' cannot be accessed.
1883
1884 Hint:
1885 Check the filesystem permissions and set directory accessible e.g. 'chmod 700'."
1886 elif [ ! -f "$CONF" ] ; then
1887 error "'$CONF' not found."
1888 elif [ ! -r "$CONF" ] ; then
1889 error "'$CONF' not readable."
1890 else
1891 . "$CONF"
1892 #KEYFILE="${KEYFILE//.asc/${GPG_KEY:+.$GPG_KEY}.asc}"
1893 TEMP_DIR=${TEMP_DIR:-'/tmp'}
1894 # backward compatibility: old TARGET_PW overrides silently new TARGET_PASS if set
1895 if var_isset 'TARGET_PW'; then
1896 TARGET_PASS="${TARGET_PW}"
1897 fi
1898 fi
1899 echo "Using profile '$CONFDIR'."
1900
1901 # secure config dir, if needed w/ warning
1902 secureconf
1903
1904 # split TARGET in handy variables
1905 TARGET_SPLIT_URL=$(echo "$TARGET" | awk '{ \
1906 target=$0; match(target,/^([^\/:]+):\/\//); \
1907 prot=substr(target,RSTART,RLENGTH);\
1908 rest=substr(target,RSTART+RLENGTH); \
1909 if (credsavail=match(rest,/^[^@]*@/)){\
1910 creds=substr(rest,RSTART,RLENGTH-1);\
1911 credcount=split(creds,cred,":");\
1912 rest=substr(rest,RLENGTH+1);\
1913 # split creds with regexp\
1914 match(creds,/^([^:]+)/);\
1915 user=substr(creds,RSTART,RLENGTH);\
1916 pass=substr(creds,RSTART+1+RLENGTH);\
1917 };\
1918 # filter quotes or escape them\
1919 gsub(/[\047\042]/,"",prot);\
1920 gsub(/[\047\042]/,"",rest);\
1921 gsub(/[\047]/,"\047\\\047\047",creds);\
1922 print "TARGET_URL_PROT=\047"prot"\047\n"\
1923 "TARGET_URL_HOSTPATH=\047"rest"\047\n"\
1924 "TARGET_URL_CREDS=\047"creds"\047\n";\
1925 if(user){\
1926 gsub(/[\047]/,"\047\\\047\047",user);\
1927 print "TARGET_URL_USER=\047"user"\047\n"}\
1928 if(pass){\
1929 gsub(/[\047]/,"\047\\\047\047",pass);\
1930 print "TARGET_URL_PASS=$(url_decode \047"pass"\047)\n"}\
1931 }')
1932 eval "${TARGET_SPLIT_URL}"
1933
1934 # fetch commmand from parameters ########################################################
1935 # Hint: cmds is also used to check if authentification info sufficient in the next step
1936 cmds="$2"; shift 2
1937
1938 # complain if command(s) missing
1939 [ -z $cmds ] && error " No command given.
1940
1941 Hint:
1942 Use '$ME usage' to get usage help."
1943
1944 # process params
1945 for param in "$@"; do
1946 #echo !$param!
1947 case "$param" in
1948 # enable ftplicity preview mode
1949 '--preview')
1950 PREVIEW=1
1951 ;;
1952 # interpret duplicity disable encr switch
1953 '--disable-encryption')
1954 GPG_KEY='disabled'
1955 ;;
1956 *)
1957 if [ `echo "$param" | grep -e "^-"` ] || \
1958 [ `echo "$last_param" | grep -e "^-"` ] ; then
1959 # forward parameter[/option pairs] to duplicity
1960 dupl_opts["${#dupl_opts[@]}"]=${param}
1961 else
1962 # anything else must be a parameter (e.g. for fetch, ...)
1963 ftpl_pars["${#ftpl_pars[@]}"]=${param}
1964 fi
1965 last_param=${param}
1966 ;;
1967 esac
1968 done
1969
1970 # plausibility check config - VARS & KEY ################################################
1971 # check if src, trg, trg pw
1972 # auth info sufficient
1973 # gpg key, gpg pwd (might be empty) set in config
1974 # OR key in local gpg db
1975 # OR key can be imported from keyfile
1976 # OR fail
1977 if [ -z "$SOURCE" ] || [ "$SOURCE" == "${DEFAULT_SOURCE}" ]; then
1978 error " Source Path (setting SOURCE) not set or still default value in conf file
1979 '$CONF'."
1980
1981 elif [ -z "$TARGET" ] || [ "$TARGET" == "${DEFAULT_TARGET}" ]; then
1982 error " Backup Target (setting TARGET) not set or still default value in conf file
1983 '$CONF'."
1984
1985 elif var_isset 'TARGET_USER' && var_isset 'TARGET_URL_USER' && \
1986 [ "${TARGET_USER}" != "${TARGET_URL_USER}" ]; then
1987 error " TARGET_USER ('${TARGET_USER}') _and_ user in TARGET url ('${TARGET_URL_USER}')
1988 are configured with different values. There can be only one.
1989
1990 Hint: Remove conflicting setting."
1991
1992 elif var_isset 'TARGET_PASS' && var_isset 'TARGET_URL_PASS' && \
1993 [ "${TARGET_PASS}" != "${TARGET_URL_PASS}" ]; then
1994 error " TARGET_PASS ('${TARGET_PASS}') _and_ password in TARGET url ('${TARGET_URL_PASS}')
1995 are configured with different values. There can be only one.
1996
1997 Hint: Remove conflicting setting."
1998 fi
1999
2000 # GPG config plausibility check1 (disabled check) #############################
2001 if gpg_disabled; then
2002 : # encryption disabled, all is well
2003 elif [ -z "${GPG_KEY}${GPG_KEYS_ENC}${GPG_KEY_SIGN}" ] && ! var_isset 'GPG_PW'; then
2004 warning "GPG_KEY, GPG_KEYS_ENC, GPG_KEY_SIGN and GPG_PW are empty/not set in conf file
2005 '$CONF'.
2006 Will disable encryption for duplicity now.
2007
2008 Hint:
2009 If you really want to use _no_ encryption you can disable this warning by
2010 setting GPG_KEY='disabled' in conf file."
2011 GPG_KEY='disabled'
2012 fi
2013
2014 # GPG availability check (now we know if gpg is really needed)#################
2015 if ! gpg_disabled; then
2016 gpg_avail || error_path "gpg '$(gpg_binary)' missing. installed und available in path?"
2017 fi
2018
2019 # Output versions info ########################################################
2020 using_info
2021
2022 # GPG create key settings, config check2 (needs gpg) ##########################
2023 if gpg_disabled; then
2024 : # the following tests are not necessary
2025 else
2026
2027 # we test this early as any invocation gpg2.1+ starts gpg-agent automatically
2028 GPG_AGENT_ERR=$(gpg_agent_avail ; echo $?)
2029
2030 # enc key still default?
2031 if [ "$GPG_KEY" == "${DEFAULT_GPG_KEY}" ]; then
2032 error_gpg "Encryption Key GPG_KEY still default in conf file
2033 '$CONF'."
2034 fi
2035
2036 # create array of gpg encr keys, for further processing
2037 OIFS="$IFS" IFS=$'\n'
2038 GPG_KEYS_ENC_ARRAY=( $( gpg_split_keyset ${GPG_KEY},${GPG_KEYS_ENC} ) )
2039 IFS="$OIFS"
2040
2041 # pw set?
2042 # symmetric needs one, always
2043 if gpg_symmetric && ( [ -z "$GPG_PW" ] || [ "$GPG_PW" == "${DEFAULT_GPG_PW}" ] ) \
2044 ; then
2045 error_gpg "Encryption passphrase GPG_PW (needed for symmetric encryption)
2046 is empty/not set or still default value in conf file
2047 '$CONF'."
2048 fi
2049 # this is a technicality, we can only pump one pass via pipe into gpg
2050 # but symmetric already always needs one for encryption
2051 if gpg_symmetric && var_isset GPG_PW && var_isset GPG_PW_SIGN &&\
2052 [ -n "$GPG_PW_SIGN" ] && [ "$GPG_PW" != "$GPG_PW_SIGN" ]; then
2053 error_gpg "GPG_PW _and_ GPG_PW_SIGN are defined but not identical in config
2054 '$CONF'.
2055 This is unfortunately impossible. For details see duplicity manpage,
2056 section 'A Note On Symmetric Encryption And Signing'.
2057
2058 Tip: Separate signing keys may have empty passwords e.g. GPG_PW_SIGN=''.
2059 Tip2: Use gpg-agent."
2060 fi
2061
2062 # check gpg encr public keys availability
2063 for (( i = 0 ; i < ${#GPG_KEYS_ENC_ARRAY[@]} ; i++ )); do
2064 KEY_ID="${GPG_KEYS_ENC_ARRAY[$i]}"
2065 # test availability, try to import, retest
2066 if ! gpg_pub_avail "${KEY_ID}"; then
2067 echo "Encryption public key '${KEY_ID}' not found."
2068 gpg_import "${KEY_ID}" PUB
2069 gpg_key_cache RESET "${KEY_ID}"
2070 gpg_pub_avail "${KEY_ID}" || error_gpg_key "${KEY_ID}" "Public"
2071 fi
2072 done
2073
2074 # gpg secret sign key availability
2075 # if none set, autoset first encryption key as sign key
2076 if ! gpg_signing; then
2077 echo "Signing disabled per configuration."
2078 # try first key, if one set
2079 elif ! var_isset 'GPG_KEY_SIGN'; then
2080 KEY_ID="${GPG_KEYS_ENC_ARRAY[0]}"
2081 if [ -z "${KEY_ID}" ]; then
2082 echo "Signing disabled. No GPG_KEY entries in config."
2083 GPG_KEY_SIGN='disabled'
2084 else
2085 # use avail OR try import OR fail
2086 if gpg_sec_avail "${KEY_ID}"; then
2087 GPG_KEY_SIGN="${KEY_ID}"
2088 else
2089 gpg_import "${KEY_ID}" SEC
2090 gpg_key_cache RESET "${KEY_ID}"
2091 if gpg_sec_avail "${KEY_ID}"; then
2092 GPG_KEY_SIGN="${KEY_ID}"
2093 fi
2094 fi
2095
2096 # interpret sign key setting
2097 if var_isset 'GPG_KEY_SIGN'; then
2098 echo "Autoset found secret key of first GPG_KEY entry '${KEY_ID}' for signing."
2099 else
2100 echo "Signing disabled. First GPG_KEY entry's '${KEY_ID}' private key is missing."
2101 GPG_KEY_SIGN='disabled'
2102 fi
2103 fi
2104 else
2105 KEY_ID="${GPG_KEY_SIGN}"
2106 if ! gpg_sec_avail "${KEY_ID}"; then
2107 inform "Secret signing key defined in setting GPG_KEY_SIGN='${KEY_ID}' not found.\nTry to import."
2108 gpg_import "${KEY_ID}" SEC
2109 gpg_key_cache RESET "${KEY_ID}"
2110 gpg_sec_avail "${KEY_ID}" || error_gpg_key "${KEY_ID}" "Private"
2111 fi
2112 fi
2113
2114 # using GPG_AGENT_ERR set early above, try to autoenable gpg-agent or issue some warnings
2115 # key enc can deal without, but might profit from gpg-agent
2116 # if GPG_PW is not set alltogether
2117 # if signing key is different from first (main) enc key (we can only pipe one pass into gpg)
2118 if ! gpg_symmetric && \
2119 ( ! var_isset GPG_PW || \
2120 ( gpg_signing && ! var_isset GPG_PW_SIGN && [ "$GPG_KEY_SIGN" != "${GPG_KEYS_ENC_ARRAY[0]}" ] ) ); then
2121
2122 if [ "$GPG_AGENT_ERR" -eq 1 ]; then
2123 warning "Cannot use gpg-agent. GPG_AGENT_INFO not set."
2124 elif [ "$GPG_AGENT_ERR" -eq 2 ]; then
2125 warning "Cannot use gpg-agent! GPG_AGENT_INFO contains stale pid."
2126 elif [ "$GPG_AGENT_ERR" -eq 3 ]; then
2127 warning "No running gpg-agent found although GPG_PW or GPG_PW_SIGN (enc != sign key) not set."
2128 else
2129 echo "Enable gpg-agent usage. Running gpg-agent instance found and GPG_PW or GPG_PW_SIGN (enc != sign key) not set."
2130 GPG_USEAGENT="--use-agent"
2131 fi
2132 fi
2133
2134 # end GPG config plausibility check2
2135 fi
2136
2137 # config plausibility check - SPACE ###########################################
2138
2139 # is tmp is a folder and writable
2140 CMD_MSG="Checking TEMP_DIR '${TEMP_DIR}' is a folder and writable"
2141 run_cmd test -d $(qw "$TEMP_DIR") '&&' test -w $(qw "$TEMP_DIR")
2142 if [ "$?" != "0" ]; then
2143 error "Temporary file space '$TEMP_DIR' is not a directory or writable."
2144 fi
2145
2146
2147 # get volsize, default duplicity volume size is 25MB since v0.5.07
2148 VOLSIZE=${VOLSIZE:-25}
2149 # double if asynch is on
2150 echo $@ $DUPL_PARAMS | grep -q -e '--asynchronous-upload' && FACTOR=2 || FACTOR=1
2151
2152 # TODO: check for enough (async= upload space and WARN only
2153 # use function tmp_space
2154 #echo TODO: reimplent tmp space check
2155
2156
2157 # test - GPG SANITY #####################################################################
2158 # if encryption is disabled, skip this whole section
2159 if gpg_disabled; then
2160 echo -e "Test - En/Decryption skipped. (GPG disabled)"
2161 elif [ "$GPG_TEST" = "disabled" ]; then
2162 echo -e "Test - En/Decryption skipped. (Testing disabled)"
2163 else
2164
2165 GPG_TEST="$TEMP_DIR/${ME_NAME}.$$.$(date_fix %s)"
2166 function cleanup_gpgtest {
2167 echo -en "Cleanup - Delete '${GPG_TEST}_*'"
2168 rm "${GPG_TEST}"_* 2>/dev/null && echo "(OK)" || echo "(FAILED)"
2169 }
2170
2171 # signing enabled?
2172 if gpg_signing; then
2173 CMD_PARAM_SIGN="--sign --default-key $(qw ${GPG_KEY_SIGN})"
2174 CMD_MSG_SIGN="Sign with '${GPG_KEY_SIGN}'"
2175 fi
2176
2177 # using keys
2178 if [ ${#GPG_KEYS_ENC_ARRAY[@]} -gt 0 ]; then
2179
2180 for KEY_ID in "${GPG_KEYS_ENC_ARRAY[@]}"; do
2181 CMD_PARAMS="$CMD_PARAMS -r $(qw ${KEY_ID})"
2182 done
2183 # check encrypting
2184 CMD_MSG="Test - Encrypt to '$(join "','" "${GPG_KEYS_ENC_ARRAY[@]}")'${CMD_MSG_SIGN:+ & $CMD_MSG_SIGN}"
2185 run_cmd $(gpg_pass_pipein GPG_PW_SIGN GPG_PW) gpg $CMD_PARAM_SIGN $(gpg_param_passwd GPG_PW_SIGN GPG_PW) $CMD_PARAMS $GPG_USEAGENT --status-fd 1 $GPG_OPTS -o $(qw "${GPG_TEST}_ENC") -e $(qw "$ME_LONG")
2186 CMD_ERR=$?
2187
2188 if [ "$CMD_ERR" != "0" ]; then
2189 KEY_NOTRUST=$(echo "$CMD_OUT"|awk '/^\[GNUPG:\] INV_RECP 10/ { print $4 }')
2190 [ -n "$KEY_NOTRUST" ] && HINT="Key '${KEY_NOTRUST}' seems to be untrusted. If you really trust this key try to
2191 'gpg --edit-key "$KEY_NOTRUST"' and raise the trust level to ultimate. If you
2192 can trust all of your keys set GPG_OPTS='--trust-model always' in conf file."
2193 error_gpg_test "Encryption failed (Code $CMD_ERR).${CMD_OUT:+\n$CMD_OUT}" "$HINT"
2194 fi
2195
2196 # check decrypting
2197 CMD_MSG="Test - Decrypt"
2198 gpg_key_decryptable || CMD_DISABLED="No matching secret key available."
2199 run_cmd $(gpg_pass_pipein GPG_PW) gpg $(gpg_param_passwd GPG_PW) $GPG_OPTS -o $(qw "${GPG_TEST}_DEC") $GPG_USEAGENT -d $(qw "${GPG_TEST}_ENC")
2200 CMD_ERR=$?
2201
2202 if [ "$CMD_ERR" != "0" ]; then
2203 error_gpg_test "Decryption failed.${CMD_OUT:+\n$CMD_OUT}"
2204 fi
2205
2206 # symmetric only
2207 else
2208 # check encrypting
2209 CMD_MSG="Test - Encryption with passphrase${CMD_MSG_SIGN:+ & $CMD_MSG_SIGN}"
2210 run_cmd $(gpg_pass_pipein GPG_PW) gpg $GPG_OPTS $CMD_PARAM_SIGN --passphrase-fd 0 -o $(qw "${GPG_TEST}_ENC") --batch -c $(qw "$ME_LONG")
2211 CMD_ERR=$?
2212 if [ "$CMD_ERR" != "0" ]; then
2213 error_gpg_test "Encryption failed.${CMD_OUT:+\n$CMD_OUT}"
2214 fi
2215
2216 # check decrypting
2217 CMD_MSG="Test - Decryption with passphrase"
2218 run_cmd $(gpg_pass_pipein GPG_PW) gpg $GPG_OPTS --passphrase-fd 0 -o $(qw "${GPG_TEST}_DEC") --batch -d $(qw "${GPG_TEST}_ENC")
2219 CMD_ERR=$?
2220 if [ "$CMD_ERR" != "0" ]; then
2221 error_gpg_test "Decryption failed.${CMD_OUT:+\n$CMD_OUT}"
2222 fi
2223 fi
2224
2225 # compare original w/ decryptginal
2226 CMD_MSG="Test - Compare"
2227 [ -r "${GPG_TEST}_DEC" ] || CMD_DISABLED="File not found. Nothing to compare."
2228 run_cmd "test \"\$(cat '$ME_LONG')\" = \"\$(cat '${GPG_TEST}_DEC')\""
2229 CMD_ERR=$?
2230 if [ "$CMD_ERR" = "0" ]; then
2231 cleanup_gpgtest
2232 else
2233 error_gpg_test "Comparision failed.${CMD_OUT:+\n$CMD_OUT}"
2234 fi
2235
2236 fi # end disabled
2237
2238 ## an empty line
2239 #echo
2240
2241 # Exclude file is needed, create it if necessary
2242 [ -f "$EXCLUDE" ] || touch "$EXCLUDE"
2243
2244 # export only used keys, if bkp not already exists ######################################
2245 gpg_export_if_needed "${GPG_KEYS_ENC_ARRAY[@]}" "$(gpg_signing && echo $GPG_KEY_SIGN)"
2246
2247
2248 # command execution #####################################################################
2249
2250 # urldecode url vars into plain text
2251 var_isset 'TARGET_URL_USER' && TARGET_URL_USER="$(url_decode "$TARGET_URL_USER")"
2252 var_isset 'TARGET_URL_PASS' && TARGET_URL_PASS="$(url_decode "$TARGET_URL_PASS")"
2253
2254 # defined TARGET_USER&PASS vars replace their URL pendants
2255 # (double defs already dealt with)
2256 var_isset 'TARGET_USER' && TARGET_URL_USER="$TARGET_USER"
2257 var_isset 'TARGET_PASS' && TARGET_URL_PASS="$TARGET_PASS"
2258
2259 TARGET_URL_PROT_lowercase="$(tolower "${TARGET_URL_PROT%%:*}")"
2260
2261 # issue some warnings
2262 case "$TARGET_URL_PROT_lowercase" in
2263 'cf+http')
2264 # info on missing AUTH_URL
2265 if ! var_isset 'CLOUDFILES_AUTHURL'; then
2266 inform "No CLOUDFILES_AUTHURL exported (in conf).
2267 Will use default which is probably rackspace."
2268 fi
2269 ;;
2270 'swift')
2271 # info on possibly missing AUTH_URL
2272 var_isset 'SWIFT_AUTHURL' ||\
2273 warning "\
2274 Swift will probably fail because the conf var SWIFT_AUTHURL was not exported!"
2275 ;;
2276 'rsync')
2277 # everything in url (this backend does not support pass in env var)
2278 # this is obsolete from version 0.6.10 (buggy), hopefully fixed in 0.6.11
2279 # print warning older version is detected
2280 duplicity_version_lt 610 &&
2281 warning "\
2282 Duplicity version '$DUPL_VERSION' does not support providing the password as
2283 env var for rsync backend. For security reasons you should consider to
2284 update to a version greater than '0.6.10' of duplicity."
2285 ;;
2286 esac
2287
2288
2289 # for all protocols we put username in url and pass into env var
2290 # for sec�rity reasons, we url_encode username to protect special chars
2291 # first sortout backends with special ways to handle password
2292 case "$TARGET_URL_PROT_lowercase" in
2293 'imap'|'imaps')
2294 var_isset 'TARGET_URL_PASS' && BACKEND_PARAMS="IMAP_PASSWORD=$(qw "${TARGET_URL_PASS}")"
2295 ;;
2296 *)
2297 # rest uses FTP_PASS var
2298 var_isset 'TARGET_URL_PASS' && \
2299 BACKEND_PARAMS="FTP_PASSWORD=$(qw "${TARGET_URL_PASS}")"
2300 ;;
2301 esac
2302 # insert url encoded username into target url if needed
2303 if var_isset 'TARGET_URL_USER' && [ "$TARGET_URL_PROT_lowercase" != "file" ]; then
2304 BACKEND_URL="${TARGET_URL_PROT}$(url_encode "${TARGET_URL_USER}")@${TARGET_URL_HOSTPATH}"
2305 else
2306 BACKEND_URL="$TARGET"
2307 fi
2308
2309
2310 # protect eval from special chars in url (e.g. open ')' in password,
2311 # spaces in path, quotes) happens above in duplify() via quotewrap()
2312 SOURCE="$SOURCE"
2313 BACKEND_URL="$BACKEND_URL"
2314 EXCLUDE="$EXCLUDE"
2315 # since 0.7.03 --exclude-globbing-filelist is deprecated
2316 EXCLUDE_PARAM="--exclude$(duplicity_version_lt 703 && echo -globbing)-filelist"
2317
2318 # translate backup to batch command
2319 cmds=${cmds//backup/groupIn_pre_bkp_post_groupOut}
2320
2321 # replace magic separators to command equivalents (+=and,-=or,[=groupIn,]=groupOut)
2322 cmds=$(awk -v cmds="$cmds" "BEGIN{ \
2323 gsub(/\+/,\"_and_\",cmds);\
2324 gsub(/\-/,\"_or_\",cmds);\
2325 gsub(/\[/,\"_groupIn_\",cmds);\
2326 gsub(/\]/,\"_groupOut_\",cmds);\
2327 print cmds}")
2328 # convert cmds to array, lowercase for safety
2329 declare -a CMDS
2330 CMDS=( $(awk "BEGIN{ cmds=tolower(\"$cmds\"); gsub(/_/,\" \",cmds); print cmds }") )
2331
2332 unset FTPL_ERR
2333
2334 # run cmds
2335 for cmd in ${CMDS[*]};
2336 do
2337
2338 ## init
2339 # raise index in cmd array for pre/post param
2340 var_isset 'CMD_NO' && CMD_NO=$((++CMD_NO)) || CMD_NO=0
2341
2342 unset CMD_VALUE CMD_NEXT CMD_PREV CND_NEXT CND_PREV
2343
2344 # get next cmd,cnd vars
2345 nextno=$(( $CMD_NO ))
2346 while ! var_isset 'CMD_NEXT'
2347 do
2348 nextno=$(($nextno+1))
2349 if [ "$nextno" -lt "${#CMDS[@]}" ]; then
2350 CMD_VALUE=${CMDS[$nextno]}
2351 is_condition "$CMD_VALUE" && CND_NEXT="$CMD_VALUE" && continue
2352 is_groupMarker "$CMD_VALUE" && continue
2353 CMD_NEXT="$CMD_VALUE"
2354 else
2355 CMD_NEXT='END'
2356 fi
2357 done
2358
2359 # get prev cnd, cnds are skipped pseudocmds
2360 prevno=$(( $CMD_NO ));
2361 while ! var_isset 'CND_PREV'
2362 do
2363 prevno=$(($prevno-1))
2364 if [ "$prevno" -ge 0 ]; then
2365 CMD_VALUE=${CMDS[$prevno]}
2366 is_condition "$CMD_VALUE" && CND_PREV="$CMD_VALUE" && break
2367 is_command "$CMD_VALUE" && break
2368 else
2369 break
2370 fi
2371 done
2372
2373 # get prev cmd command minus skipped commands, only executed
2374 prevno=$(( $CMD_NO - ${CMD_SKIPPED-0} ));
2375 while ! var_isset 'CMD_PREV'
2376 do
2377 prevno=$(($prevno-1))
2378 if [ "$prevno" -ge 0 ]; then
2379 CMD_VALUE=${CMDS[$prevno]}
2380 is_condition "$CMD_VALUE" && CND_PREV="$CMD_VALUE" && continue
2381 is_groupMarker "$CMD_VALUE" && continue
2382 CMD_PREV="$CMD_VALUE"
2383 else
2384 CMD_PREV='START'
2385 fi
2386 done
2387
2388 function get_cmd_skip_count {
2389 # find closing bracket, get group skip count
2390 local nextno=$CMD_NO
2391 local GRP_OPEN=0
2392 local GRP_SKIP=0
2393 local CMD_VALUE
2394 while [ "$nextno" -lt "${#CMDS[@]}" ]
2395 do
2396 nextno=$(($nextno+1))
2397 CMD_VALUE=${CMDS[$nextno]}
2398 GRP_SKIP=$(( ${GRP_SKIP} + 1 ));
2399 if is_command "$CMD_VALUE" && [ "$GRP_OPEN" -lt 1 ]; then
2400 break;
2401 elif [ "$CMD_VALUE" == 'groupin' ]; then
2402 GRP_OPEN=$(( ${GRP_OPEN} + 1 ))
2403 elif [ "$CMD_VALUE" == 'groupout' ]; then
2404 GRP_OPEN=$(( ${GRP_OPEN} - 1 ))
2405 if [ "$GRP_OPEN" -lt 1 ]; then
2406 break;
2407 fi
2408 fi
2409 done
2410
2411 echo $GRP_SKIP;
2412 }
2413
2414 # decision time: are we skipping already or dealing with condition "commands" or other non-cmds?
2415 unset SKIP_NOW
2416 if var_isset 'CMD_SKIP' && [ $CMD_SKIP -gt 0 ]; then
2417 # skip cnd/grp cmds silently
2418 is_command "$cmd" && echo -e "\n--- Skipping command $(toupper $cmd) ! ---"
2419 CMD_SKIP=$(($CMD_SKIP - 1))
2420 SKIP_NOW="yes"
2421 elif ! var_isset 'PREVIEW' && [ "$cmd" == 'and' ] && [ "$CMD_ERR" -ne "0" ]; then
2422 CMD_SKIP=$(get_cmd_skip_count)
2423 # incl. this "cmd"
2424 CMD_SKIP=$(( $CMD_SKIP + 1 ))
2425 unset CMD_SKIPPED
2426 SKIP_NOW="yes"
2427 elif ! var_isset 'PREVIEW' && [ "$cmd" == 'or' ] && [ "$CMD_ERR" -eq "0" ]; then
2428 CMD_SKIP=$(get_cmd_skip_count)
2429 # incl. this "cmd"
2430 CMD_SKIP=$(( $CMD_SKIP + 1 ))
2431 unset CMD_SKIPPED
2432 SKIP_NOW="yes"
2433 elif is_condition "$cmd" || is_groupMarker "$cmd"; then
2434 unset 'CMD_SKIP';
2435 SKIP_NOW="yes"
2436 fi
2437
2438 # let's do the skip now
2439 if [ -n "$SKIP_NOW" ]; then
2440 # sum up how many commands we actually skipped for the prev var routines
2441 CMD_SKIPPED=$((${CMD_SKIPPED-0} + 1))
2442 continue
2443 fi
2444
2445 # save start time
2446 RUN_START=$(nsecs)
2447
2448 # export some useful env vars for external scripts/programs to use
2449 export PROFILE CONFDIR SOURCE TARGET_URL_PROT TARGET_URL_HOSTPATH \
2450 TARGET_URL_USER TARGET_URL_PASS \
2451 GPG_KEYS_ENC=$(join "\n" "${GPG_KEYS_ENC_ARRAY[@]}") GPG_KEY_SIGN \
2452 GPG_PW CMD_PREV CMD_NEXT CMD_ERR CND_PREV CND_NEXT\
2453 RUN_START
2454
2455 # user info
2456 echo; separator "Start running command $(toupper $cmd) at $(date_from_nsecs $RUN_START)"
2457
2458 case "$(tolower $cmd)" in
2459 'pre'|'post')
2460 if [ "$cmd" == 'pre' ]; then
2461 script=$PRE
2462 else
2463 script=$POST
2464 fi
2465 # script execution in a subshell, protect us from failures/var overwrites
2466 ( run_script "$script" )
2467 ;;
2468 'bkp')
2469 duplify -- "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \
2470 "$SOURCE" "$BACKEND_URL"
2471 ;;
2472 'incr')
2473 duplify incr -- "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \
2474 "$SOURCE" "$BACKEND_URL"
2475 ;;
2476 'full')
2477 duplify full -- "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \
2478 "$SOURCE" "$BACKEND_URL"
2479 ;;
2480 'verify')
2481 TIME="${ftpl_pars[0]:+"-t ${ftpl_pars[0]}"}"
2482 duplify verify -- $TIME "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \
2483 "$BACKEND_URL" "$SOURCE"
2484 ;;
2485 'verifypath')
2486 TIME="${ftpl_pars[2]:+"-t ${ftpl_pars[2]}"}"
2487 IN_PATH="${ftpl_pars[0]}"; OUT_PATH="${ftpl_pars[1]}";
2488 ( [ -z "$IN_PATH" ] || [ -z "$OUT_PATH" ] ) && error " Missing parameter <rel_bkp_path> or <local_path> for verifyPath.
2489
2490 Hint:
2491 Syntax is -> $ME <profile> verifyPath <rel_bkp_path> <local_path> [<age>]"
2492
2493 duplify verify -- $TIME "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \
2494 --file-to-restore "$IN_PATH" "$BACKEND_URL" "$OUT_PATH"
2495 ;;
2496 'list')
2497 # time param exists since 0.5.10+
2498 TIME="${ftpl_pars[0]:+"-t ${ftpl_pars[0]}"}"
2499 duplify list-current-files -- $TIME "${dupl_opts[@]}" "$BACKEND_URL"
2500 ;;
2501 'cleanup')
2502 duplify cleanup -- "${dupl_opts[@]}" "$BACKEND_URL"
2503 ;;
2504 'purge')
2505 MAX_AGE=${ftpl_pars[0]:-$MAX_AGE}
2506 [ -z "$MAX_AGE" ] && error " Missing parameter <max_age>. Can be set in profile or as command line parameter."
2507
2508 duplify remove-older-than "${MAX_AGE}" \
2509 -- "${dupl_opts[@]}" "$BACKEND_URL"
2510 ;;
2511 'purgefull')
2512 MAX_FULL_BACKUPS=${ftpl_pars[0]:-$MAX_FULL_BACKUPS}
2513 [ -z "$MAX_FULL_BACKUPS" ] && error " Missing parameter <max_full_backups>. Can be set in profile or as command line parameter."
2514
2515 duplify remove-all-but-n-full "${MAX_FULL_BACKUPS}" \
2516 -- "${dupl_opts[@]}" "$BACKEND_URL"
2517 ;;
2518 'purgeincr')
2519 MAX_FULLS_WITH_INCRS=${ftpl_pars[0]:-$MAX_FULLS_WITH_INCRS}
2520 [ -z "$MAX_FULLS_WITH_INCRS" ] && error " Missing parameter <max_fulls_with_incrs>. Can be set in profile or as command line parameter."
2521
2522 duplify remove-all-inc-of-but-n-full "${MAX_FULLS_WITH_INCRS}" \
2523 -- "${dupl_opts[@]}" "$BACKEND_URL"
2524 ;;
2525 'restore')
2526 OUT_PATH="${ftpl_pars[0]}"; TIME="${ftpl_pars[1]:-now}";
2527 [ -z "$OUT_PATH" ] && error " Missing parameter target_path for restore.
2528
2529 Hint:
2530 Syntax is -> $ME <profile> restore <target_path> [<age>]"
2531
2532 duplify -- -t "$TIME" "${dupl_opts[@]}" "$BACKEND_URL" "$OUT_PATH"
2533 ;;
2534 'fetch')
2535 IN_PATH="${ftpl_pars[0]}"; OUT_PATH="${ftpl_pars[1]}";
2536 TIME="${ftpl_pars[2]:-now}";
2537 ( [ -z "$IN_PATH" ] || [ -z "$OUT_PATH" ] ) && error " Missing parameter <src_path> or <target_path> for fetch.
2538
2539 Hint:
2540 Syntax is -> $ME <profile> fetch <src_path> <target_path> [<age>]"
2541
2542 # duplicity 0.4.7 doesnt like cmd restore in combination with --file-to-restore
2543 duplify -- --restore-time "$TIME" "${dupl_opts[@]}" \
2544 --file-to-restore "$IN_PATH" "$BACKEND_URL" "$OUT_PATH"
2545 ;;
2546 'status')
2547 duplify collection-status -- "${dupl_opts[@]}" "$BACKEND_URL"
2548 ;;
2549 *)
2550 error " Unknown command '$cmd'.
2551
2552 Hint:
2553 Use '$ME usage' to get usage help."
2554 ;;
2555 esac
2556
2557 CMD_ERR=$?
2558 RUN_END=$(nsecs)
2559 RUNTIME=$(( $RUN_END - $RUN_START ))
2560
2561 # print message on error; set error code
2562 if [ "$CMD_ERR" -ne 0 ]; then
2563 error_print "$(datefull_from_nsecs $RUN_END) Task '$(echo $cmd|awk '$0=toupper($0)')' failed with exit code '$CMD_ERR'."
2564 FTPL_ERR=1
2565 fi
2566
2567 separator "Finished state $(error_to_string $CMD_ERR) at $(date_from_nsecs $RUN_END) - \
2568 Runtime $(printf "%02d:%02d:%02d.%03d" $((RUNTIME/1000000000/60/60)) $((RUNTIME/1000000000/60%60)) $((RUNTIME/1000000000%60)) $((RUNTIME/1000000%1000)) )"
2569
2570 done
2571
2572 exit ${FTPL_ERR}
