% Developer guide
Beaker consists of several different components. The largest and most critical are a set of web services that manage an inventory of hardware systems distributed across multiple labs and take care of provisioning systems appropriately and dispatching jobs to them.
This is the software that is developed in the main Beaker git repository (click link to browse). The bulk of this developer guide focuses on this component.
However, http://git.beaker-project.org plays host to a few other components which are also considered part of the wider "Beaker project":
- beaker-project.org: The source for the project web site (including this developer's guide)
- beah: The test harness used to communicate between running tests and the Beaker infrastructure. Other test harnesses (such as autotest or STAF) are not yet officially supported. Unlike the Beaker web services (which are only officially supported on the platforms described below), the test harness must run on all operating systems supported for testing.
- rhts: The code in this repo isn't part of Beaker as such, it's a collection of utilities designed to help with writing and running Beaker test cases.
Start by cloning Beaker's git repository:
git clone git://git.beaker-project.org/beaker
For the purposes of development, Beaker should be run on the develop branch:
git checkout develop
This branch will become the next Beaker release. If you want to test against
the latest released version, you can use the master branch. For older
releases, use the relevant git tag.
Cloning the other repos is similar to the above, but they all develop directly on master (as they don't have a concept of "hot fix" releases that only include bug fixes - any release may include a combination of both new features and bug fixes)
Beaker currently only supports MySQL with InnoDB as the database backend (it does use SQL Alchemy internally though, so porting it to an alternate backend shouldn't be too difficult). On RHEL and Fedora systems, MySQL can easily be installed with:
# yum install mysql
For Beaker development, the following settings should be added to
/etc/my.cnf in the [mysqld] section before starting the MySQL daemon:
default-storage-engine=INNODB
max_allowed_packet=50M
character-set-server=utf8
Once these settings are in place, start the database daemon:
# service mysqld start
Before running the development server for the first time, you must create and
populate its database (your working directory should be the Server
subdirectory of your local clone of the main beaker project):
mysql -uroot <<"EOF"
CREATE DATABASE beaker;
GRANT ALL ON beaker.* TO 'beaker'@'localhost' IDENTIFIED BY 'beaker';
EOF
PYTHONPATH=../Common:. python bkr/server/tools/init.py \
--user=admin \
--password=adminpassword \
--email=me@example.com
By default this uses the beaker database on localhost. This can be changed
by editing dev.cfg and updating the above configuration commands
appropriately.
You can then start a development server using the start-server.py script,
with PYTHONPATH adjusted for the git checkout:
cd Server/
PYTHONPATH=../Common:. ./start-server.py
The Beaker team uses RHEL 6 for development, testing, and deployment, therefore it is recommended to use RHEL 6 when writing your patch. Beaker should also work on Fedora 16 or higher, although this configuration is not well tested.
If you want to set up a complete Beaker testing environment (including a lab controller) with the ability to provision systems and run jobs, refer to the Beaker in a box quick start guide, or the more detailed Installation guide.
Running Lab Controller processes in a development environment is currently not well tested.
Please see the README file in the root directory of Beaker's source tree for a detailed description of its layout.
The lab controller is the intermediary point of communication between the lab machines, and the beaker server. It provides entry points for the following processes:
- beaker-proxy
- beaker-watchdog
- beaker-transfer
- beaker-provision
These processes authenticate themselves using the credentials found in
/etc/beaker/labcontroller.conf.
beaker-proxy is a process that binds to an open unprivileged port on the lab controller. It provides an xmlrpc server that more or less forwards calls made to it (from lab machines) onto the beaker server.
Here is a simple example of a typical controller from bkr.labcontroller.proxy:
def task_info(self,
qtask_id):
""" accepts qualified task_id J:213 RS:1234 R:312 T:1234 etc.. Returns dict with status """
logger.debug("task_info %s", qtask_id)
return self.hub.taskactions.task_info(qtask_id)
This is quite straightforward. An xmlrpc call would be made to the the lab
controller onto port 8000 (the default for beaker-proxy), with the method
task_info and the qtask id. The task_info() method then calls and
returns bkr.server.taskactions.task_info. The hub variable is a HubProxy
class from kobo.client. HubProxy in turn is merely a thin wrapper
over xmlrpclib.ServerProxy that adds session based authentication management.
When provisioning a system in beaker, Beaker transfers various log files to the lab controller as a matter of course. Even more log files are created when running an actual recipe. By default, these logs are uploaded from the test machine to the lab controller.
Using the CACHE and ARCHIVE_* variables in labcontroller.conf, you can
specify a remote server where the log files can be moved to. Once configured,
beaker-transfer is responsible for moving these files from the lab controller
to the remote server.
To ensure that recipes do not run on for longer than what they are expected to, Beaker uses a watchdog process. This process keeps track of running recipes on all test systems attached to a particular lab controller. If a recipe is running for longer than what the server has specified it can run for, the watchdog aborts the recipe.
Here is a breakdown of how the process works:
- Recipe progresses up to the Scheduled stage, and a watchdog is created for the recipe.
- Recipe goes into
Waitingstage and (amongst others things) a reboot command is sent to the lab controller. - System is rebooted and then an RPC is made to the server to start the first task, and create a kill time for the watchdog.
- For each successive task that is run in the recipe, the harness extends the
kill time of the watchdog by the expected run time specified
by the
TestTimevalue in testinfo.desc. - If a task runs beyond it's kill time, the harness will first try and abort the job and continue onto the next test. If the system crashes or panics and the harness isn't able to recover then beaker-watchdog will abort the recipe (and any remaining tasks). The time between the harness watchdog (also known as local watchdog) and beaker-watchdog (also known as external watchdog) is ten minutes.
It is the responsibility of beaker-provision to handle the provisioning of a system. Its main duties are creating boot loader images and power cycling systems.
Here is a breakdown of how this process works:
- Beaker server generates and serves a kickstart file.
- Beaker server sends a command to beaker-provision with the path to the distro tree, kernel options, and a link to the kickstart file. It also sends a command to power cycle the system.
- From these details, beaker-provision generates the relevant boot loader configuration file, and then power cycles the test system.
- The system boots and starts its install.
The main focus of the beaker server is to maintain a distro and system inventory, run tasks against this inventory, and display the results of those tasks. Any interaction with this inventory must be via the server. For further details of the design of Beaker's services, please see the relevant docs.
Beaker is developed to run in Red Hat Enterprise Linux 6 (RHEL6). Versions previous to 0.7 were designed for RHEL5. Although it may be technically possible to run Beaker on other distributions, package dependencies and other issues may ensue.
The Beaker server is a web application that is built upon Turbogears (TG) 1.x. Here is a quick breakdown of TG (and it's relevant sub frameworks) and how Beaker utilizes them.
- TurboGears 1.x
TurboGears (TG) is a “front-to_back” web meta-framework. For more information about TG, please see the TurboGears website.- Core TG
These are modules which are implemented within TG itself.\- identity
does session based authentication, is used to control access to resources. - widgets
provides pre built templates and display functionality. Templates are written in Kid. Beaker's own custom widgets are also built upon TG widgets.
- identity
- CherryPy 2
Provides resource routing, handling of request and response objects. CherryPy 2 is no longer under active development. - Kid
Provides the templating language for hand written templates, as well as TG widgets. Kid is not longer under active development. - SQLAlchemy
An ORM database interface. Used exclusively for all access to Beaker's database. Note that Beaker uses some TG database modules, but these are thin wrappers over SQLAlchemy. Beaker currently uses version 6.8. - JQuery/MochiKit
MochiKit is bundled with TG, however JQuery is heavily used alongside it.
- Core TG
As a result of being built on TG, Beaker is an MVC inspired application. Whilst it mostly follows TG conventions, Beaker does sometimes go outside of these when it's appropriate (and advantageous) to do so.
The bkr.server.model module primarily consists of Object Relational Mapped
(ORM) classes. Fundamentally, these are user defined python classes associated
to database tables, the objects of which are mapped to rows in the related table.
These are not declaratively defined, but instead use
'Classical Mapping'.
Some basic guidelines to follow when modifying model:
- Definitions of Tables, ORM classes, and calls to mapper() are segregated into three distinct sections. Tables are defined above ORM classes, and ORM classes above mapper functions. If possible define related Tables in the vicinity of each other, and likewise for ORM classes and mappers.
- Commonly used queries should be contained within bound methods of the respective classes.
- Enumerated types should be defined as type DeclEnum and not be described in a database schema. This helps avoid over normalization, cuts down on unnecessary calls to the database, and reduces the likelihood of complex joins that confuse the query optimizer. This only applies though if it's an enumeration that is static.
- When writing queries, use ORM attributes over 'SQL Expression Language' whenever possible, and never use 'Text'.
- Write efficient queries. Do what you can to write the most reasonably efficient query. For various reasons, Beaker has few options of removing its historical data. Thus query speed and data-set size can only increase over time. As Beaker's UI relies heavily on database calls, writing inefficient queries can quickly become a bottleneck and create a marked reduction in usability.
- Beyond the basic relationship mapping, relationships should be defined keeping performance in mind. The sqlalchemy documentation provides some good ideas.
- Remember to define relevant cascade options.
A controller is called when a HTTP request is made. The URL is translated to
a particular controller. CherryPy is responsible for handling this method
look-up. For example, a call to http://beaker.example.com/tasks/executed
will call the bkr.server.tasks.executed method.
Generally speaking, Beaker controllers are grouped into a single module for
either one of two purposes. Either because the controller provides various
modifies and accessors for a single ORM class (e.g the bkr.server.system
module contains various accessor and modifier methods for the System class),
or for the purpose of supporting a single page view and any associated actions
(e.g the bkr.server.preferences module contains all of the views and actions
needed for viewing and updating users preferences).
Sometimes a mix of these two can be found, and this is also fine
(i.e bkr.server.tasks contains controllers for displaying and searching on
task details, as well as methods designed to be called remotely to provide
details of Task objects).
Both Kid templates and TG widgets are used to support the 'View' of MVC. Beaker uses TG widgets to provide re-usability of commonly rendered page elements. A widget encapsulates the template to be rendered, as well as any javascript and CSS files that are needed by that template. Generally speaking, creating a widget is preferable to using a controller + template due to the re-usability of a widget. However there is no hard and fast rule in regards to this.
As well as standard widgets being provided by TG, Beaker also implements many
of its own widgets in the bkr.server.widgets module.
Templates are used in one of two ways; by specifying a template in an 'expose' decorator; by setting the template variable in a widget, and then calling that widget's 'display' method. Examples of both will be shown in the patch walk-through.
The beaker-client package provides shell commands that makes varied calls
to the server. The format of the calls are bkr <cmd> <options>, where <cmd>
corresponds to a module in the bkr/client/commands directory. The modules
of the corresponding code is a normalized version of the same name as the
command, but with the prefix cmd_. For example, bkr job-list will call the
run() method of the bkr.client.commands.cmd_job_list module.
This functionality is provided by the kobo.client.ClientCommand class, of
which all Beaker commands inherit (indirectly or directly). This class also
provides the authentication with the Beaker server via the same kobo classes as
the lab controller.
To get a better sense of how the different modules come together, let's look at parts of a real patch that has been applied to Beaker. The bug is an RFE to remove tasks from the task library.
Let's look at what we have to do in model.py first.
diff --git a/Server/bkr/server/model.py b/Server/bkr/server/model.py
index bd2995d..ae8de19 100644
--- a/Server/bkr/server/model.py
+++ b/Server/bkr/server/model.py
@@ -1002,7 +1002,7 @@
ForeignKey('tg_user.user_id')),
Column('version', Unicode(256)),
Column('license', Unicode(256)),
- Column('valid', Boolean),
+ Column('valid', Boolean, default=True),
mysql_engine='InnoDB',
)
We need to change our Table instance so that newly created tasks are valid by default. This table's schema is created from the 'beaker-init' command, which is only run when setting up a new beaker environment, or when there are new tables to create. We will need to update the current schema in the DB manually, but we will look at that later.
@@ -5845,9 +5845,23 @@ class Task(MappedObject):
"""
Tasks that are available to schedule
"""
+ @property
+ def task_dir(self):
+ return get("basepath.rpms", "/var/www/beaker/rpms")
+
@classmethod
- def by_name(cls, name):
- return cls.query.filter_by(name=name).one()
+ def by_name(cls, name, valid=None):
+ query = cls.query.filter(Task.name==name)
+ if valid:
+ query = query.filter(Task.valid==bool(valid))
+ return query.one()
+
+ @classmethod
+ def by_id(cls, id, valid=None):
+ query = cls.query.filter(Task.id==id)
+ if valid:
+ query = query.filter(Task.valid==bool(valid))
+ return query.one()
by_name() and by_id() are commonly implemented methods for querying the
database. These also need to be updated so we can specify whether we want valid
or invalid tasks to be returned.
@@ -5919,6 +5933,17 @@ def elapsed_time(self, suffixes=[' year',' week',' day',' hour',' minute',' seco
return separator.join(time)
+ def disable(self):
+ """
+ Disable task so it can't be used.
+ """
+ for rpm in [self.oldrpm, self.rpm]:
+ rpm_path = "%s/%s" % (self.task_dir, rpm)
+ if os.path.exists(rpm_path):
+ os.unlink(rpm_path)
+ self.valid=False
+ return
+
A cohesive method that will handle the details of disabling a task is
desirable. This method acts directly on the Task object and will be called
by a controller.
@@ -1469,7 +1469,33 @@ def display(self, task, **params):
return super(RecipeActionWidget,self).display(task, **params)
-class JobActionWidget(TaskActionWidget):
+class TaskActionWidget(RPC):
+ template = 'bkr.server.templates.task_action'
+ params = ['redirect_to']
+ action = url('/tasks/disable_from_ui')
+ javascript = [LocalJSLink('bkr', '/static/javascript/task_disable.js')]
+
+ def __init__(self, *args, **kw):
+ super(TaskActionWidget, self).__init__(*args, **kw)
+
+ def display(self, task, action=None, **params):
+ id = task.id
+ task_details={'id': 'disable_%s' % id,
+ 't_id' : id}
+ params['task_details'] = task_details
+ if action:
+ params['action'] = action
+ return super(TaskActionWidget, self).display(task, **params)
+
+ def update_params(self, d):
+ super(TaskActionWidget, self).update_params(d)
+ d['task_details']['onclick'] = "TaskDisable('%s',%s, %s)" % (
+ d.get('action'),
+ jsonify.encode({'t_id': d['task_details'].get('t_id')}),
+ jsonify.encode(self.get_options(d)),
+ )
+
A widget enables us to encapsulate code that renders a template. Typically, various arguments are passed to the widget's display method. This data is often used to determine what will actually be displayed.
Widgets should be stateless. Data that is to be rendered by the widget should
be passed to the display() method, not to the widget's constructor.
Note that we've also added a new javascript file, task_disable.js. If this
widget is to be returned in a controller, this javascript source file would
be linked into the DOM for us.
Although not always necessary, widgets are a good way to harness code re-usability.
diff --git a/Server/bkr/server/templates/task_action.kid b/Server/bkr/server/templates/task_action.kid
new file mode 100644
index 0000000..5c0ea33
--- /dev/null
+++ b/Server/bkr/server/templates/task_action.kid
@@ -0,0 +1,7 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xmlns:py="http://purl.org/kid/ns#">
+<div>
+<a py:if="'admin' in tg.identity.groups" class='list' style='cursor:pointer;color: #22437f;' py:attrs='task_details'>Disable</a><br/>
+</div>
+
+</html>
This is the template that the TaskActionWidget renders. Notice how variables
such as task_details are passed from the widget to the template, via the
params arg.
diff --git a/Server/bkr/server/tasks.py b/Server/bkr/server/tasks.py
index 2fe447b..21b1afd 100644
--- a/Server/bkr/server/tasks.py
+++ b/Server/bkr/server/tasks.py
@@ -25,6 +25,7 @@
from bkr.server.widgets import TasksWidget
from bkr.server.widgets import TaskSearchForm
from bkr.server.widgets import SearchBar
+from bkr.server.widgets import TaskActionWidget
from bkr.server.xmlrpccontroller import RPCRoot
from bkr.server.helpers import make_link
from bkr.server import testinfo
@@ -50,9 +51,10 @@ class Tasks(RPCRoot):
# For XMLRPC methods in this class.
exposed = True
+ task_list_action_widget = TaskActionWidget()
tasks.py is our controller module for actions on the '/tasks' page. We
import our widget here and instantiate it.
@@ -315,6 +354,7 @@ def index(self, *args, **kw):
widgets.PaginateDataGrid.Column(name='name', getter=lambda x: make_link("./%s" % x.id, x.name), title='Name', options=
widgets.PaginateDataGrid.Column(name='description', getter=lambda x:x.description, title='Description', options=dict(s
widgets.PaginateDataGrid.Column(name='version', getter=lambda x:x.version, title='Version', options=dict(sortable=True
+ widgets.PaginateDataGrid.Column(name='action', getter=lambda x: self.task_list_action_widget.display(task=x, type_='t
])
Although perhaps not a typical example, here the widget's display() method
is called from a lambda function and will be rendered inside the task grid.
More often the widget is returned from a controller and its display() method
called in the template.
diff --git a/SchemaUpgrades/upgrade_0.6.11.txt b/SchemaUpgrades/upgrade_0.6.11.txt
new file mode 100644
index 0000000..c9f565e
--- /dev/null
+++ b/SchemaUpgrades/upgrade_0.6.11.txt
@@ -0,0 +1,8 @@
+Make task.valid default True
+---------------------------------
+UPDATE task set valid = True;
+ALTER TABLE task MODIFY valid TINYINT DEFAULT 1;
+
+To roll back:
+ Nothing is needed, it doesn't hurt to leave the task valid=true.
+
As previously mentioned, we will need to manually update the database to
reflect the new schema changes introduced. We do this in the SchemaUpgrades
directory, in a file named upgrade_<version>.txt. Despite the name of the parent
directory, this is the place to put any manual upgrades that are needed as part
of an upgrade. Note that roll back code should also be included if it makes
sense to do so.
Start by creating a local git branch where you will commit your work:
git checkout -b myfeature develop
Beaker has a large and thorough suite of integration tests, including many Selenium/WebDriver browser tests. You should test your patch by running the test suite either locally or in Beaker, or both.
In order to run the test suite locally, you must create the additional test database in your local MySQL instance:
mysql -uroot <<"EOF"
CREATE DATABASE beaker_test;
GRANT ALL ON beaker_test.* TO 'beaker'@'localhost' IDENTIFIED BY 'beaker';
EOF
In addition, the necessary Selenium support is not yet available as an RPM,
so it is necessary to download
this jar file
and save it as /usr/local/share/selenium/selenium-server-standalone-2.21.0.jar.
(Yes, it must be that exact version at that exact path. The longer term
fix to make this step cleaner is to provide an appropriate RPM in the Beaker
repos)
While working on your patch, you would run the unit test for your new feature/fix:
cd IntegrationTests/
./run-tests.sh -sv bkr.inttest.path_to_my_new_test
Once the new or updated test is passing, you should also run the whole suite to check for unintended side effects:
./run-tests.sh
The run-tests.sh script is a thin wrapper around
nosetests which sets up PYTHONPATH for
running from a git checkout.
Once you've verified the patch passes the test suite, commit it to your local branch:
git commit
You can also run the Beaker test suite in Beaker (assuming you have access to a
working Beaker instance) using the /distribution/beaker/dogfood task. If your
Beaker instance doesn't already have a copy of this task, you can build it from
Beaker's source under the Tasks subdirectory. You can base your job on this
sample dogfood job XML.
You can use tito to build Beaker RPMs for testing:
tito build --test --rpm
or if you have Mock or Koji suitably configured, tito can generate an SRPM and you can build from that:
tito build --test --srpm --dist .el6
The --test option to tito build will build from the HEAD commit in git, so
make sure you have committed your changes to your local branch.
The Beaker project uses the Gerrit code review tool to manage patches. All patches are reviewed on Beaker's Gerrit installation before being merged:
New users can sign in using any OpenID account (preferably your Fedora OpenID; avoid using Google Accounts). Be sure to configure your e-mail addresses and SSH keys in Gerrit after creating your account. Your SSH key is needed to authenticate you when pushing patches using git. The e-mail address in your git commits must also match one of the e-mail addresses you have registered in Gerrit.
For convenience you can add the Gerrit server as a remote to your .git/config:
[remote "gerrit"]
url = git+ssh://gerrit.beaker-project.org:29418/beaker
Once you're happy with the change and the test you have written for it, push your local branch to Gerrit for review:
git push gerrit myfeature:refs/for/develop
A new "change" in Gerrit will be created from your commit. Beaker developers can then review and merge it as appropriate. See the Gerrit documentation for more info.
If your patch fixes a bug, be sure to include a reference to the Bugzilla number as a footer line like "Bug: 123456" in the commit message (example).
To update the patch on an existing change, you can use git commit --amend.
You must ensure that the correct Change-Id footer appears in your amended
commit message. Refer to the Gerrit
Change-Id
documentation for more details.
To avoid forgetting the Change-Id footer and accidentally creating a new review instead of updating an existing one, it's useful to install this hook which automatically adds an appropriate "Change-Id" entry to the commit message when a patch is first committed locally:
scp -p -P 29418 gerrit.beaker-project.org:hooks/commit-msg .git/hooks/