darkskysync package

Submodules

darkskysync.AbstractRemoteSyncAdapter module

Created on Sep 24, 2013

@author: J. Akeret

class darkskysync.AbstractRemoteSyncAdapter.AbstractRemoteSyncAdapter[source]

Bases: object

Abstract class for all synchronization adapters

loadFiles(files)[source]

Loads files from the remote location

darkskysync.DarkSkySync module

Created on Sep 24, 2013

@author: J. Akeret

class darkskysync.DarkSkySync.DarkSkySync(template=u'master', configFile=None, verbose=False)[source]

Bases: object

Helps to access and synchronize with a remote data storage. Features are:
  • lists the available file structure on the remote host
  • lists the file structure in the local cache
  • synchronizes the local cache with the remote file structure
  • allows for removing files from the local file system

It delegates the remote access to the according wrappers (currently either scp over ssh or rsync over ssh) and manages the local file structure

Parameters:
  • template – [optional] alternate name for the config template to use
  • configFile – [optional] alternate config file to use
  • verbose – [optional] flag to enable verbose mode
Raises:

ConfigurationError – error while reading the config file
DEFAULT_TEMPLATE = u'master'
REMOTE_SYNC_WRAPPER_TYPE_MAP = {u'local': <class 'darkskysync.LocalSyncAdapter.LocalSyncAdapter'>, u'rsync': <class 'darkskysync.RsyncWrapper.RsyncWrapper'>, u'ssh': <class 'darkskysync.SSHClientAdapter.SSHClientAdapter'>}
REMOTE_WRAPPER_TYPE_MAP = {u'rsync': <class 'darkskysync.SSHClientAdapter.SSHClientAdapter'>, u'ssh': <class 'darkskysync.SSHClientAdapter.SSHClientAdapter'>}
avail(path=None)[source]

Lists the available file structure on the remote host

Parameters:path – [optional] sub path to directory structure to check
Returns:list of file available on remote host
Raises:ConnectionError – remote host could not be accessed
list(path=None, recursive=False)[source]

Lists the available files in the local cache

Parameters:
  • path – [optional] sub path to directory structure to check
  • recursive – [optional] flag indicating if the cache should be browsed recursively
Returns:

list of files in the local cache
load(names, dry_run=False, force=False)[source]

Loads the given file names. Checks first if the requested files are available on the file system. If not, they will be downloaded from the remote host

Parameters:
  • names – a list of names which should be loaded
  • dry_run – [optional] flag indicating a dry run. No files will be downloaded
  • force – [optional] flag indicating that the files should be downloaded even if they already exist on the local file system
Retun:

list of files loaded from the remote host
Raises:
remove(files=None, allFiles=False)[source]

Removes files and filetrees from the local cache

Parameters:
  • files – [optional] list of files or directories to delete
  • allFiles – [optional] flag indicating that all files shoudl be removed from the cache
Raises:

darkskysync.DarkSkySyncCLI module

This is the DarkSkySync command line interface

Created on Sep 23, 2013

@author: J. Akeret

Usage:
DarkSkySync avail [<path>] [–config=<file>] [–template=<template>] [-v|–verbose] DarkSkySync list [<path>] [-r|–recursive] [–config=<file>] [–template=<template>] [-v|–verbose] DarkSkySync load <name>… [–dry_run] [-f|–force] [–config=<file>] [–template=<template>] [-v|–verbose] DarkSkySync remove (<name>…|–all) [–config=<file>] [–template=<template>] [-v|–verbose] DarkSkySync -h | –help DarkSkySync –version
Options
-h –help Show this screen. –version Show version. –config=<file> The configfile to use. –template=<template> The template to use [default:master] –all All files -v –verbose More output -f –force Force the download -r –recursive List the directories in a recursive way –dry_run Dry run - Not loading any files
class darkskysync.DarkSkySyncCLI.DarkSkySyncCLI[source]

Bases: object

Command line interface for the darkskysync

dispatch(dam, args)[source]

Dispatches the sub command to the given darkskysync instance

Parameters:
  • dam – the instance of the darkskysync to us
  • args – the arguments passed by the used
launch()[source]

Starts the darkskysync by parsing the args and the config

darkskysync.DarkSkySyncCLI.main()[source]

darkskysync.DataSource module

Created on Sep 23, 2013

@author: J. Akeret

class darkskysync.DataSource.DataSource(name, local, remote)[source]

Bases: object

Represents a configuration of a data source

darkskysync.DataSourceFactory module

Created on Sep 23, 2013

@author: J. Akeret

class darkskysync.DataSourceFactory.ConfigReader(configfile)[source]

Bases: object

Reads the configuration properties from a ini file.

datasource_section = u'datasource'
local_section = u'local'
login_section = u'login'
read_config()[source]

Reads the configuration properties from the ini file and links the section to comply with the datasource config dictionary format. :return: dictionary containing all configuration properties from the

ini file in compliance to the datasource config format
Raises:MultipleInvalid – not all sections present or broken links between secitons
remote_section = u'remote'
class darkskysync.DataSourceFactory.ConfigValidator(config)[source]

Bases: object

validate()[source]

Validates the given configuration self.config to comply. As well all types are converted to the expected format if possible.

class darkskysync.DataSourceFactory.DataSourceFactory(config)[source]

Bases: object

A factory creating DataSource configurations

DEFAULT_CONFIGURATION_FILE = u'/home/gitlab-runner/.darkskysync/config'
REMOTE_FACTORY_TYPE_MAP = {u'local': <class 'darkskysync.RemoteFactory.SSHRemoteLocationFactory'>, u'rsync': <class 'darkskysync.RemoteFactory.SSHRemoteLocationFactory'>, u'ssh': <class 'darkskysync.RemoteFactory.SSHRemoteLocationFactory'>}
createDataSource(template, name=None)[source]
Creates a data source by inspecting the configuration properties of the
given data source template.
Parameters:
  • template – name of the data source template
  • name – name of the data source. If not defined, the data source

will be named after the template.

Returns:DataSource instance
Raises:ConfigurationError – data source template not found in config
createLocal(template)[source]
createRemote(template)[source]
classmethod fromConfig(configfile)[source]

Helper method to initialize DataSourceFactory from an ini file. :param configfile: path to the ini file :return: configurator object

darkskysync.Exceptions module

Created on Sep 23, 2013

@author: J. Akeret

exception darkskysync.Exceptions.ConfigurationError[source]

Bases: exceptions.Exception

exception darkskysync.Exceptions.ConnectionError[source]

Bases: exceptions.Exception

exception darkskysync.Exceptions.FileNotFoundError[source]

Bases: exceptions.Exception

exception darkskysync.Exceptions.FileSystemError[source]

Bases: exceptions.Exception

exception darkskysync.Exceptions.IllegalArgumentException[source]

Bases: exceptions.Exception

darkskysync.LocalFileSystem module

Created on Sep 17, 2013

@author: J. Akeret

class darkskysync.LocalFileSystem.LocalFileSystem(name, filePath)[source]

Bases: object

Represents a configuration for a local file system

filePath = None
name = None

darkskysync.LocalSyncAdapter module

Created on Dec 2, 2015

author: jakeret

class darkskysync.LocalSyncAdapter.LocalSyncAdapter(dataSourceConfig, dry_run=False, force=False, verbose=True)[source]

Bases: darkskysync.AbstractRemoteSyncAdapter.AbstractRemoteSyncAdapter

An adapter implementation to the paramiko ssh client

Parameters:
  • dataSourceConfig – configuration of the data source
  • rsync_exec – name of the rsync command to use. Default is ‘rsync’
  • dry_run – If true no files are downloaded
  • force – If true the files in the local cache will be overwritten
  • verbose – If true additional information will be printed
LIST_CMD = u'ls'
NEWLINE = u'\n'
getRemoteFilesList(path=None)[source]
loadFiles(files)[source]

Returns the local file path for each requested file

Parameters:files – A single file name or path or a list of file names
Returns:a list of downloaded files

darkskysync.RemoteFactory module

Created on Sep 23, 2013

@author: J. Akeret

class darkskysync.RemoteFactory.AbstractRemoteLocationFactory[source]

Bases: object

Defines the contract for a remote location factory to proper function

create(name, conf)[source]

Creates a new instance with the given properties

class darkskysync.RemoteFactory.SSHRemoteLocationFactory[source]

Bases: darkskysync.RemoteFactory.AbstractRemoteLocationFactory

create(name, conf)[source]
createLogin(name, conf)[source]

darkskysync.RsyncWrapper module

Created on Sep 17, 2013

@author: J. Akeret

class darkskysync.RsyncWrapper.RsyncWrapper(dataSourceConfig, rsync_exec=u'rsync', dry_run=False, force=False, verbose=True)[source]

Bases: darkskysync.AbstractRemoteSyncAdapter.AbstractRemoteSyncAdapter

Wrapper for the rsync unix command.

Parameters:
  • dataSourceConfig – configuration of the data source
  • rsync_exec – name of the rsync command to use. Default is ‘rsync’
  • dry_run – If true no files are downloaded
  • force – If true the files in the local cache will be overwritten
  • verbose – If true additional information will be printed
FAILURE_KEYWORDS_EX_MAP = {u'Could not resolve hostname': ConnectionError(u'Name or service not known',), u'Permission denied': FileSystemError(u'Unable to write files. Reason: Permission denied',), u'rsync error': ConnectionError(u'An unexpected error occured',)}
loadFiles(files)[source]

Creates the rsync commands and executes them to download the files from the remote repository. :param files: A single file name or path or a list of file names

Returns:a list of downloaded files
prepareRsyncCommands(files)[source]

Creates the rsync commands for the given files :param files: A single file name or path or a list of file names

darkskysync.SSHClientAdapter module

Created on Sep 25, 2013

@author: J. Akeret

class darkskysync.SSHClientAdapter.SSHClientAdapter(dataSourceConfig, dry_run=False, force=False, verbose=True)[source]

Bases: darkskysync.AbstractRemoteSyncAdapter.AbstractRemoteSyncAdapter

An adapter implementation to the paramiko ssh client

Parameters:
  • dataSourceConfig – configuration of the data source
  • rsync_exec – name of the rsync command to use. Default is ‘rsync’
  • dry_run – If true no files are downloaded
  • force – If true the files in the local cache will be overwritten
  • verbose – If true additional information will be printed
LIST_CMD = u'ls'
NEWLINE = u'\n'
getRemoteFilesList(path=None)[source]

Lists the files available on the remote host

Parameters:path – Relative path to the resource of interest
loadFiles(files)[source]

Downloads the files form the remote host

Parameters:files – A single file name or path or a list of file names
Returns:a list of downloaded files

darkskysync.SSHRemoteFileSystemLocation module

Created on Sep 17, 2013

@author: J. Akeret

class darkskysync.SSHRemoteFileSystemLocation.SSHRemoteFileSystemLocation(name, type, host, port, filePath, login)[source]

Bases: object

Represents the configuration for the remote file system informations

darkskysync.SSHWrapper module

Created on Sep 24, 2013

@author: J. Akeret

class darkskysync.SSHWrapper.SSHWrapper(dataSourceConfig, sshExec=u'ssh')[source]

Bases: object

Wrapps the ssh unix command

Parameters:
  • dataSourcConfig – configuration of the data source
  • sshExec – name of the ssh command to use. Default is ‘ssh’
LIST_CMD = u'ls'
prepareSshCmds()[source]

Prepares the ssh command in order to use the configured known hosts and private key files

prepareSshFileCmds()[source]

Prepares the commands for the file transfer

darkskysync.SshLoginInfo module

Created on Sep 17, 2013

@author: J. Akeret

class darkskysync.SshLoginInfo.SshLoginInfo(name, username, user_key_private=u'~/.ssh/id_rsa', known_hosts=u'~/.ssh/known_hosts')[source]

Bases: object

Represents the configuration for the remote login informations

known_hosts = None
name = None
user_key_private = None
username = None

Module contents

darkskysync.avail(path=None)[source]

Lists the available file structure on the remote host

Parameters:path – [optional] sub path to directory structure to check
Returns:list of file available on remote host
Raises:ConnectionError – remote host could not be accessed
darkskysync.list(path=None, recursive=False)[source]

Lists the available files in the local cache

Parameters:
  • path – [optional] sub path to directory structure to check
  • recursive – [optional] flag indicating if the cache should be browsed recursively
Returns:

list of files in the local cache
darkskysync.load(names, dry_run=False, force=False)[source]

Loads the given file names. Checks first if the requested files are available on the file system. If not, they will be downloaded from the remote host

Parameters:
  • names – a list of names which should be loaded
  • dry_run – [optional] flag indicating a dry run. No files will be downloaded
  • force – [optional] flag indicating that the files should be downloaded even if they already exist on the local file system
Retun:

list of files loaded from the remote host
Raises:
darkskysync.remove(files=None, allFiles=False)[source]

Removes files and filetrees from the local cache

Parameters:
  • files – [optional] list of files or directories to delete
  • allFiles – [optional] flag indicating that all files shoudl be removed from the cache
Raises: