toxiccore package

Submodules

toxiccore.build_config module

class toxiccore.build_config.APTPluginConfig(conf)

Bases: BasePluginConfig

get_config()

class toxiccore.build_config.BasePluginConfig(conf)

Bases: object

Plugin configs are meant to be used along with the LanguageConfig. Based in the toxicbuild.yml config we can create the build plugins configuration for a builder.

Note

The build plugins currently live in the slave but that is a very silly idea. In a future relase that’s gonna change.

class toxiccore.build_config.LanguageConfig(conf)

Bases: object

An abstraction to create builders based in a language config in a toxicbuild.yml config file.

DEFAULT_OS = 'debian'

LANGUAGE_PLUGINS = {'python': <class 'toxiccore.build_config.PythonPluginConfig'>}

SYSTEM_PACKAGES_PLUGINS = {'debian': <class 'toxiccore.build_config.APTPluginConfig'>, 'ubuntu': <class 'toxiccore.build_config.APTPluginConfig'>}

property builders

class toxiccore.build_config.LanguagePluginConfig(lang_ver, conf)

Bases: BasePluginConfig

class toxiccore.build_config.PythonPluginConfig(lang_ver, conf)

Bases: LanguagePluginConfig

get_config()

async toxiccore.build_config.get_config(workdir, config_type, config_filename)

Returns the build configuration for a given repository.

Parameters:

• workdir – The directory where config file is located.

• config_type – Which type of configuration we are using, ‘py’ or ‘yaml’.

• config_filename – The filename of the config file.

toxiccore.build_config.get_toxicbuildconf(directory)

Returns the toxicbuild.conf module.

Parameters:

directory – Directory to look for toxicbuild.conf

async toxiccore.build_config.get_toxicbuildconf_yaml(directory, filename='toxicbuild.yml')

Returns the python objet representing the yaml build configuration.

Parameters:

• directory – The path of the directory containing the config file.

• filename – The actual name of the config file.

toxiccore.build_config.list_builders_from_config(config, branch=None, slave=None, config_type='yml')

Lists builders from a build config

Parameters:

• config – The build configuration.

• branch – The branch for which builders are being listed.

• slave – The slave for which builders are being listed.

toxiccore.build_config.load_config(config_type, config)

Loads a config based in a string with the config content.

Parameters:

• config_type – Which type of configuration we are using, ‘py’ or ‘yaml’.

• config – A string with the build config

toxiccore.build_config.load_yaml_conf(config)

Loads a configuration using yaml.load

Parameters:

conf – A string - usually the contents of a file - containing a yaml config.

toxiccore.client module

This module implements a base client for basic tcp communication, reading and writing json data.

Usage:

host = 'somehost.net'
port = 1234
async with BaseToxicClient(host, port):
    await client.write({'hello': 'world'})
    json_response = await client.get_response()

class toxiccore.client.BaseToxicClient(host, port, use_ssl=False, validate_cert=True, **ssl_kw)

Bases: LoggerMixin

Base client for communication with toxicbuild servers.

async READ_STREAM_FN(timeout=None)

Reads the input stream. First reads the bytes until the first “n”. These first bytes are the length of the full message.

Parameters:

• reader – An instance of asyncio.StreamReader

• timeout – Timeout for the operation. If None there is no timeout

async WRITE_STREAM_FN(data, timeout=None)

Writes data to output. Encodes data to utf-8 and prepend the lenth of the data before sending it.

Parameters:

• writer – An instance of asyncio.StreamWriter

• data – String data to be sent.

• timeout – Timeout for the write operation. If None there is no timeout

async connect()

Connects to the server.

Note

This is called by the asynchronous context manager (aka async with)

disconnect()

Disconnects from the server

async get_response(timeout=None)

Reads data from the server and raises and exception in case of error

is_connected()

async read(timeout=None)

Reads data from the server. Expects a json.

param timeout: Timeout for the read operation. If None there is

no timeout

async request2server(action, body, token, timeout=None)

Performs a request to a toxicbuild server and server returns the response.

Parameters:

• action – The action to perform on the server.

• body – The body for the action.

• token – An authentication token.

async write(data, timeout=None)

Writes data to the server.

Parameters:

• data – Data to be sent to the server. Will be converted to json and enconded using utf-8.

• timeout – Timeout for the write operation. If None there is no timeout.

toxiccore.cmd module

class toxiccore.cmd.ToxicProgram(*args, **kwargs)

Bases: Program

Transforms a python function into a command line program. Extends mando’s Program to execute _generate_command() only when execute() is called to enable toxicbuild command hacks.

execute(*args, **kwargs)

Parse the arguments and execute the resulting command.

Parameters:

args – The arguments to parse.

toxiccore.cmd.execute(*args, **kwargs)

Parse the arguments and execute the resulting command.

Parameters:

args – The arguments to parse.

toxiccore.conf module

class toxiccore.conf.Settings(envvar, default_filename)

Bases: object

Simple interface to a settings file.

toxiccore.exceptions module

exception toxiccore.exceptions.BadJsonData

Bases: Exception

exception toxiccore.exceptions.ConfigError

Bases: Exception

exception toxiccore.exceptions.ExecCmdError

Bases: Exception

exception toxiccore.exceptions.ImpossibillityError

Bases: Exception

exception toxiccore.exceptions.PluginNotFound

Bases: Exception

exception toxiccore.exceptions.ToxicClientException

Bases: Exception

exception toxiccore.exceptions.VCSError

Bases: Exception

toxiccore.mail module

class toxiccore.mail.MailSender(recipients, **smtp_settings)

Bases: LoggerMixin

Simple mail sender.

To send an email, use the context manager:

recipients = ['me@mail.com', 'other@mail.com']
smtp_settings = {'host': 'some.host', 'port': 123,
                 'mail_from': 'me@myplace.net',
                 'username': 'me', 'password': '1234',
                 'validate_certs': True, 'starttls': False}

async with MailSender(recipients, **smtp_settigs) as sender:
    await sender.send('Subject', 'This is the message body')

async connect()

Connects to a smtp server.

async disconnect()

Closes the connection to the smpt server

async send(subject, message)

Send an email message.

Parameters:

• subject – Email subject.

• message – Email body

exception toxiccore.mail.MailSenderNotConnected

Bases: Exception

toxiccore.plugins module

class toxiccore.plugins.Plugin(*args, **kwargs)

Bases: object

This is a base plugin. Plugins may implement aditional behavior to your builds.

classmethod get_plugin(name)

Returns a Plugin subclass based on its name.

Parameters:

name – Plugin’s name.

classmethod list_plugins(plugin_type=None)

Returns a list of Plugin subclasses.

Parameters:

plugin_type – the plugin’s type.

name = 'BaseCorePlugin'

no_list = False

class toxiccore.plugins.PluginMeta(name, bases, attrs)

Bases: type

toxiccore.protocol module

class toxiccore.protocol.BaseToxicProtocol(loop, connection_lost_cb=None)

Bases: StreamReaderProtocol, LoggerMixin

Base protocol for toxicbulid servers. To create your own protocol extend this class and implement the client_connected method.

Example:

class MyServer(BaseToxicProtocol):

    async def client_connected(self):
        assert self.action
         r = await fn(**self.data)
         # code == 0 means everything ok.
         # code > 0 means something went wrong
         self.send_response(code=0, body={'some': 'thing'})

async check_data()

Checks if the data is valid, it means, checks if has some data, checks if it is a valid json and checks if it has a action key

async client_connected()

Coroutine that handles connections. You must implement this in your sub-classes. When this method is called, self.data, containing a dictionary with the data passed in the request and self.action, a string indicating which action to take are already available.

close_connection()

Closes the connection with the client

connection_lost(exc)

Called once, when the connection is lost.

Parameters:

exc – The exception, if some.

connection_made(transport)

Called once, when the client connects.

Parameters:

transport – transport for asyncio.StreamReader and asyncio.StreamWriter.

encrypted_token = None

async get_json_data(timeout=None)

Returns the json sent by the client. :param timeout: Timeout for the operation. If None there is no timeout

async get_raw_data(timeout=None)

Returns the raw data sent by the client :param timeout: Timeout for the operation. If None there is no timeout

async send_response(code, body, timeout=None)

Send a response to client formated by the (unknown) toxicbuild remote build specs.

Parameters:

• code – code for this message. code == 0 is success and code > 0 is error.

• body – response body. It has to be a serializable object.

• timeout – Timeout for the operation. If None there is no timeout

toxiccore.requests module

This module implements a simple asynchronous interface for http requests.

Usage:

from . import requests
response = await requests.get('http://google.com/')
print(response.text)

class toxiccore.requests.Response(status, text, headers=None)

Bases: object

Encapsulates a response from a http request

json()

Loads the json in the response text.

async toxiccore.requests.delete(url, **kwargs)

Performs a http DELETE request

Parameters:

• url – Request’s url.

• kwargs – Args passed to requests._request().

async toxiccore.requests.get(url, **kwargs)

Performs a http GET request

Parameters:

• url – Request’s url.

• kwargs – Args passed to requests._request().

async toxiccore.requests.post(url, **kwargs)

Performs a http POST request

Parameters:

• url – Request’s url.

• kwargs – Args passed to requests._request().

async toxiccore.requests.put(url, **kwargs)

Performs a http PUT request

Parameters:

• url – Request’s url.

• kwargs – Args passed to requests._request().

toxiccore.server module

class toxiccore.server.ToxicServer(addr, port, loop=None, use_ssl=False, **ssl_kw)

Bases: LoggerMixin

PROTOCOL_CLS

alias of BaseToxicProtocol

get_protocol_instance()

Returns an instance of BuildServerProtocol.

async shutdown()

Overwrite this to handle the shutdown of your server

start()

Starts the build server.

sync_shutdown()

toxiccore.shell module

async toxiccore.shell.exec_cmd(cmd, cwd, timeout=3600, out_fn=None, **envvars)

Executes a shell command. Raises with the command output if return code > 0.

Parameters:

• cmd – command to run.

• cwd – Directory to execute the command.

• timeout – How long we should wait some output. Default is 3600.

• out_fn – A coroutine that receives each line of the step output. The coroutine signature must be in the form: mycoro(line_index, line).

• envvars – Environment variables to be used in the command.

toxiccore.socks module

async toxiccore.socks.read_stream(reader, timeout=None)

Reads the input stream. First reads the bytes until the first “n”. These first bytes are the length of the full message.

Parameters:

• reader – An instance of asyncio.StreamReader

• timeout – Timeout for the operation. If None there is no timeout

async toxiccore.socks.write_stream(writer, data, timeout=None)

Writes data to output. Encodes data to utf-8 and prepend the lenth of the data before sending it.

Parameters:

• writer – An instance of asyncio.StreamWriter

• data – String data to be sent.

• timeout – Timeout for the write operation. If None there is no timeout

toxiccore.utils module

class toxiccore.utils.LoggerMixin

Bases: object

A simple mixin to use log on a class.

log(msg, level='info')

Appends the class name before the log message.

classmethod log_cls(msg, level='info')

class toxiccore.utils.MatchKeysDict

Bases: dict

A dictionary that returns the values matching the keys using utils.match_string().

>>> d = MatchKeysDict()
>>> d['k*'] = 1
>>> d['key']
1
>>> k['keyboard']
1

get(key, default=None)

Return the value for key if key is in the dictionary, else default.

class toxiccore.utils.MonkeyPatcher

Bases: object

patch_item(obj, attr, newitem, undo=True)

Sets attr in obj with newitem. If not undo the item will continue patched after leaving the context manager

class toxiccore.utils.SourceSuffixesPatcher

Bases: MonkeyPatcher

We must to path SOURCE_SUFFIXES in the python importlib so we can import source code files with extension other than .py, in this case, namely .conf

SOURCE_SUFFIXES = ['.py', '.conf']

patch_source_suffixes()

Patches the SOURCE_SUFFIXES constant in the module importlib._bootstrap_external.

toxiccore.utils.bcrypt_string(src_string, salt=None)

class toxiccore.utils.changedir(path)

Bases: object

toxiccore.utils.compare_bcrypt_string(original, encrypted)

Compares if a un-encrypted string matches an encrypted one.

Parameters:

• original – An un-encrypted string.

• encrypted – An bcrypt encrypted string.

toxiccore.utils.create_random_string(length)

toxiccore.utils.create_validation_string(secret)

Creates a random string that can be used to validate against it. The algorithm is as follows:

Given a secret, a random string is generated, then <secret>-<random-str> are encrypted using bcrypt. Finally <encrypted-str>:<random-str> are base64 encoded.

toxiccore.utils.daemonize(call, cargs, ckwargs, stdout, stderr, workdir, pidfile)

Run a callable as a daemon

Parameters:

• call – a callable.

• cargs – args to call.

• ckwargs – kwargs to call.

• stdout – daemon’s stdout.

• stderr – daemon’s stderr.

• workdir – daemon’s workdir

• pidfile – pidfile’s path

toxiccore.utils.datetime2string(dt, dtformat='%w %m %d %H:%M:%S %Y %z')

Transforms a datetime object into a formated string.

Parameters:

• dt – The datetime object.

• dtformat – The format to use.

async toxiccore.utils.exec_cmd(cmd, cwd, timeout=3600, out_fn=None, **envvars)

Executes a shell command. Raises with the command output if return code > 0.

Parameters:

• cmd – command to run.

• cwd – Directory to execute the command.

• timeout – How long we should wait some output. Default is 3600.

• out_fn – A coroutine that receives each line of the step output. The coroutine signature must be in the form: mycoro(line_index, line).

• envvars – Environment variables to be used in the command.

DEPRECATED: Use toxiccore.shell.exec_cmd()

toxiccore.utils.format_timedelta(td)

Format a timedelta object to a human-friendly string.

Parameters:

dt – A timedelta object.

toxiccore.utils.get_envvars(envvars, use_local_envvars=True)

Returns environment variables to be used in shell. Does the interpolation of values using the current values from the envvar and the values passed as parameters.

toxiccore.utils.inherit_docs(cls)

Inherit docstrings from base classes’ methods. Can be used as a decorator

Parameters:

cls – Class that will inherit docstrings from its parents.

toxiccore.utils.interpolate_dict_values(to_return, valued, base)

Interpolates the values of valued with values of base.

Parameters:

• to_return – A dictionary that will be updated with interpolated values.

• valued – A dict with values needing interpolation.

• base – A dict with the base values to use in interpolation.

toxiccore.utils.load_module_from_content(module_content)

Loads a module from a string that is the contents of the module.

Parameters:

module_content – A string with the module content.

toxiccore.utils.load_module_from_file(filename)

Load a module from a source file :param filename: full path for file to be loaded.

toxiccore.utils.localtime2utc(localdatetime)

Transforms a local datetime object into a datetime object in utc time.

Parameters:

localdatetime – A datetime object.

toxiccore.utils.log(msg, level='info')

toxiccore.utils.match_string(smatch, filters)

Checks if a string match agains a list of filters containing wildcards.

Parameters:

• smatch – String to test against the filters

• filters – Filter to match a string.

toxiccore.utils.now()

Returns the localtime with timezone info.

async toxiccore.utils.read_file(filename)

Reads the contents of a file asynchronously.

Parameters:

filename – The path of the file.

async toxiccore.utils.read_stream(reader, timeout=None)

Reads the input stream. First reads the bytes until the first “n”. These first bytes are the length of the full message.

Parameters:

• reader – An instance of asyncio.StreamReader

• timeout – Timeout for the operation. If None there is no timeout

DEPRECATED: Use toxiccore.socks.read_stream()

async toxiccore.utils.run_in_thread(fn, *args, **kwargs)

Runs a callable in a background thread.

Parameters:

• fn – A callable to be executed in a thread.

• args – Positional arguments to fn

• kwargs – Named arguments to fn

Usage

r = await run_in_thread(call, 1, bla='a')

toxiccore.utils.set_loglevel(loglevel)

toxiccore.utils.set_tzinfo(dt, tzoff)

Sets a timezone info to a datetime object.

Parameters:

dt – A datetime object.

Para tzoff:

The timezone offset from utc in seconds

toxiccore.utils.string2datetime(dtstr, dtformat='%w %m %d %H:%M:%S %Y %z')

Transforns a string into a datetime object acording to dtformat.

Parameters:

• dtstr – The string containing the formated date.

• dtformat – The format of the formated date.

toxiccore.utils.utc2localtime(utcdatetime)

Transforms a utc datetime object into a datetime object in local time.

Parameters:

utcdatetime – A datetime object

toxiccore.utils.validate_string(b64_str, secret)

Validates a string created with create_validation_string().

Given a base64 string the validation is as follows:

First decodes the base64 string in <encrypted-string>:<random-str> then bcrypt-compare <secret>-<random-str> with <encrypted-string>

async toxiccore.utils.write_stream(writer, data, timeout=None)

Writes data to output. Encodes data to utf-8 and prepend the lenth of the data before sending it.

Parameters:

• writer – An instance of asyncio.StreamWriter

• data – String data to be sent.

• timeout – Timeout for the write operation. If None there is no timeout

DEPRECATED: Use toxiccore.socks.write_stream()

toxiccore.vcs module

class toxiccore.vcs.Git(workdir)

Bases: VCS

An interface to git version control system

async add_remote(remote_url, remote_name)

Adds a new remote to the repository.

Parameters:

• remote_url – The url of the remote repository.

• remote_name – The name of the remote.

async branch_exists(branch_name)

Checks if a local branch exists.

Parameters:

branch_name – The name of the branch to check.

async checkout(named_tree)

Checkout to named_tree :param named_tree: A commit, branch, tag…

async clone(url)

Clones a repository into self.workdir :param url: repository url

async create_local_branch(branch_name, base_name)

Creates a branch new in the local repository

Parameters:

• branch_name – The name for the new branch

• base_name – The name of the base branch.

date_format = '%a %b %d %H:%M:%S %Y'

async delete_local_branch(branch_name)

Deletes a local branch.

Parameters:

branch_name – The name of the branch to be deleted.

async fetch()

Fetch changes from remote repository

async get_local_revisions(since=None, branches=None)

Returns the newer revisions since for branches in the local repository

Parameters:

• since – dictionary in the format: {branch_name: since_date}. since is a datetime object.

• branches – A list of branches to look for new revisions. If branches is None all remote branches will be used. You can use wildcards in branches to filter the local branches.

async get_remote(remote_name='origin')

Returns the remote url used in the repo.

Parameters:

remote_name – The name of the remote url to change.

async get_remote_branches()

Returns a list of the remote branches available.

async get_revisions(since=None, branches=None)

Returns the newer revisions since for branches from the default remote repository.

Parameters:

• since – dictionary in the format: {branch_name: since_date}. since is a datetime object.

• branches – A list of branches to look for new revisions. If branches is None all remote branches will be used. You can use wildcards in branches to filter the remote branches.

async get_revisions_for_branch(branch, since=None)

Returns the revisions for branch since since. If since is None, all revisions will be returned.

Parameters:

• branch – branch name

• since – datetime

async has_changes()

Informs if there are new revisions in the repository

async import_external_branch(external_url, external_name, external_branch, into)

Imports a branch from an external (not the origin one) repository into a local branch.

Parameters:

• external_url – The url of the external repository.

• external_name – Name to idenfity the remote url.

• external_branch – The name of the branch in the external repo.

• into – The name of the local branch.

async pull(branch_name, remote_name='origin')

Pull changes from branch_name on remote repo.

Parameters:

• branch_name – A branch name, like ‘master’.

• remote_name – The remote repository to push from.

async rm_remote(remote_name)

async set_remote(url, remote_name='origin')

Sets the remote url of the repository.

Parameters:

• url – The new remote url.

• remote_name – The name of the remote url to change.

async try_set_remote(url, remote_name='origin')

Sets the remote url if the remote is not equal as url.

Parameters:

• url – The new url for the remote.

• remote_name – The name of the remote url to change.

async update_submodule()

vcsbin = 'git'

class toxiccore.vcs.VCS(workdir)

Bases: LoggerMixin

Generic inteface to a vcs (clone, fetch, get revisions etc…).

abstract async add_remote(remote_url, remote_name)

Adds a new remote to the repository.

Parameters:

• remote_url – The url of the remote repository.

• remote_name – The name of the remote.

async classmethod branch_exists(branch_name)

Checks if a local branch exists.

Parameters:

branch_name – The name of the branch to check.

abstract async checkout(named_tree)

Checkout to named_tree :param named_tree: A commit, branch, tag…

abstract async clone(url)

Clones a repository into self.workdir :param url: repository url

abstract async create_local_branch(branch_name, base_name)

Creates a branch new in the local repository

Parameters:

• branch_name – The name for the new branch

• base_name – The name of the base branch.

abstract async delete_local_branch(branch_name)

Deletes a local branch.

Parameters:

branch_name – The name of the branch to be deleted.

async exec_cmd(cmd, cwd=None)

Executes a shell command. If cwd is None self.workdir will be used.

Parameters:

cwd – Directory where the command will be executed.

abstract async fetch()

Fetch changes from remote repository

abstract async get_local_revisions(since=None, branches=None)

Returns the newer revisions since for branches in the local repository

Parameters:

• since – dictionary in the format: {branch_name: since_date}. since is a datetime object.

• branches – A list of branches to look for new revisions. If branches is None all remote branches will be used. You can use wildcards in branches to filter the local branches.

abstract async get_remote(remote_name)

Returns the remote url used in the repo.

Parameters:

remote_name – The name of the remote url to change.

abstract async get_remote_branches()

Returns a list of the remote branches available.

abstract async get_revisions(since=None, branches=None)

Returns the newer revisions since for branches from the default remote repository.

Parameters:

• since – dictionary in the format: {branch_name: since_date}. since is a datetime object.

• branches – A list of branches to look for new revisions. If branches is None all remote branches will be used. You can use wildcards in branches to filter the remote branches.

abstract async get_revisions_for_branch(branch, since=None)

Returns the revisions for branch since since. If since is None, all revisions will be returned.

Parameters:

• branch – branch name

• since – datetime

abstract async has_changes()

Informs if there are new revisions in the repository

abstract async import_external_branch(external_url, external_name, external_branch, into)

Imports a branch from an external (not the origin one) repository into a local branch.

Parameters:

• external_url – The url of the external repository.

• external_name – Name to idenfity the remote url.

• external_branch – The name of the branch in the external repo.

• into – The name of the local branch.

abstract async pull(branch_name, remote_name='origin')

Pull changes from branch_name on remote repo.

Parameters:

• branch_name – A branch name, like ‘master’.

• remote_name – The remote repository to push from.

abstract async set_remote(url, remote_name)

Sets the remote url of the repository.

Parameters:

• url – The new remote url.

• remote_name – The name of the remote url to change.

abstract async try_set_remote(url, remote_name)

Sets the remote url if the remote is not equal as url.

Parameters:

• url – The new url for the remote.

• remote_name – The name of the remote url to change.

vcsbin = None

workdir_exists()

Informs if the workdir for this vcs exists

toxiccore.vcs.get_vcs(vcs_type)

Retuns a subclass of vcs.VCS for vcs_type

Module contents