OpenStack Infra: How to deploy Zuul

This is the second post on Zuul, which focuses on deploying it and its services. To learn what is Zuul and how it works, I recommend to read the previous post.

Methods of deployment

I’ll introduce two ways to deploy Zuul. Both are basically the same, but in one you will have this nice Ansible role which you can execute really quickly, while using the other way will require you to execute each command manually.

For a quick deployment, use the Ansible way. To learn how exactly the deployment is done, step by step, use the manual way.

Note: This assumes you are deploying Zuul on RHEL/CentOS 7 server.

How to deploy Zuul

Pre deployment – Gerrit & Zuul connection

To have working Zuul server, you need to have access to Gerrit with the user you plan to specify in zuul.conf (let’s assume it’s named zuul, although it can be any other user).

To make sure you have the required access, run the following:

ssh -p 29418 <zuul_user>@<gerrit_server> gerrit stream-events

The short & automated way

First, clone ‘ansible-zuul’ project. I created this project in order to provide easy and automated way to deploy Zuul

git clone https://github.com/bregman-arie/ansible-zuul.git

Next, create the deploy.yml playbook (or copy it from ansible-zuul/playbooks/deploy.yml):

---
- hosts: zuul
  vars:
    gerrit_server: 'https://my_gerrit.com'
    status_page: yes 
  roles:
    - role: ansible-zuul

gerrit_server is the url of your Gerrit server and status_page is set to include the web page setup or not. By default, when you do ‘pip install zuul’, it will not setup the web page for you (No idea why…).

Now edit /etc/ansible/hosts and make sure the target server (where you want to deploy Zuul) is mentioned there

vi /etc/ansible/hosts

[zuul]
192.168.4.4

Now run the playbook (it may take couple of minutes):

ansible-playbook deploy.yml -vv

Enjoy, you have Zuul 🙂

You can now proceed to adding your first pipeline and project.

The long & manual way

Wow, you actually chose the long way? nice. Let’s not waste time.

packages and user

Connect to the server and install the following packages:

sudo yum install -y python-lockfile python-paramiko python-daemon logrotate python-webob python-paste python-jenkins-job-builder python-zmq logrotate

If you don’t have those packages available, try to install EPEL repo first OR switch to pip installation.

sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Add the user ‘zuul’ (make sure it also creates the ‘zuul’ group)

sudo useradd zuul
project installation

Clone the Zuul repository

sudo git clone https://git.openstack.org/openstack-infra/zuul /opt/zuul

Install Zuul

sudo pip install /opt/zuul
Services

Now let’s create the services file. You might be using init or systemd, so I covered them both.

Let’s start with systemd. First service would be zuul-server

> vi /etc/systemd/system/zuul-server.service

[Unit]
Description=Zuul Server Service
After=syslog.target network.target

[Service]
Type=simple
User=zuul
Group=zuul
EnvironmentFile=-/etc/sysconfig/zuul
ExecStart=/usr/bin/zuul-server -d
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Next, zuul-merger

> vi /etc/systemd/system/zuul-merger.service

[Unit]
Description=Zuul Merger Service
After=syslog.target network.target

[Service]
Type=simple
User=zuul
Group=zuul
EnvironmentFile=-/etc/sysconfig/zuul
ExecStart=/usr/bin/zuul-merger -d

[Install]
WantedBy=multi-user.target

Next, zuul-launcher

[Unit]
Description=Zuul Launcher Service
After=syslog.target network.target

[Service]
Type=simple
User=zuul
Group=zuul
EnvironmentFile=-/etc/sysconfig/zuul
ExecStart=/usr/bin/zuul-launcher -d

[Install]
WantedBy=multi-user.target

Now the init (remember, you don’t need it, if you setup the systemd files).

I’ll put here only one of the files, since it’s quite long and basically it’s the same file, so simply switch from any ‘zuul-server’ to ‘zuul-merger’ and ‘zuul-launcher’ when you copy paste the files.

Zuul-server init file will look like this:

vi /etc/init.d/zuul-server

#! /bin/sh
### BEGIN INIT INFO
# Provides:          zuul
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Zuul
# Description:       Trunk gating system
### END INIT INFO

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Zuul"
NAME=zuul-server
DAEMON=/usr/local/bin/zuul-server
PIDFILE=/var/run/$NAME/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
USER=zuul

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
    # Return
    #   0 if daemon has been started
    #   1 if daemon was already running
    #   2 if daemon could not be started
    #   3 if pid file exits already

    mkdir -p /var/run/$NAME
    chown $USER /var/run/$NAME
    if [ -f $PIDFILE ]; then
        return 3
    fi
    start-stop-daemon 
       --start --quiet --pidfile $PIDFILE -c $USER 
        --exec $DAEMON --test > /dev/null || return 1
    start-stop-daemon 
        --start --quiet --pidfile $PIDFILE -c $USER 
        --exec $DAEMON -- $DAEMON_ARGS || return 2
    # Add code here, if necessary, that waits for the process to be ready
    # to handle requests from services started subsequently which depend
    # on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
    # Return
    #   0 if daemon has been stopped
    #   1 if daemon was already stopped
    #   2 if daemon could not be stopped
    #   other if a failure occurred
    start-stop-daemon --stop --signal 9 --pidfile $PIDFILE
    RETVAL="$?"
    [ "$RETVAL" = 2 ] && return 2
    rm -f /var/run/$NAME/*
    return "$RETVAL"
}

#
# Function that stops the daemon/service
#
do_graceful_stop()
{
    PID=`cat $PIDFILE`
    kill -USR1 $PID

    # wait until really stopped
    if [ -n "${PID:-}" ]; then
        i=0
        while kill -0 "${PID:-}" 2> /dev/null;  do
            if [ $i -eq '0' ]; then
                echo -n " ... waiting "
            else
                echo -n "."
            fi
            i=$(($i+1))
            sleep 1
        done
    fi

    rm -f /var/run/$NAME/*
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
    #
    # If the daemon can reload its configuration without
    # restarting (for example, when it is sent a SIGHUP),
    # then implement that here.
    #
    start-stop-daemon 
        --stop --signal 1 --quiet --pidfile $PIDFILE --name $DAEMON
    return 0
}

case "$1" in
    start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
        do_start
        case "$?" in
            0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
            2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
            3) echo "Pidfile at $PIDFILE already exists, run service zuul stop to clean up." ;;
        esac
        ;;
    stop)
        [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
        do_stop
        case "$?" in
            0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
            2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
    status)
        status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
        ;;
    reload|force-reload)
        #
        # If do_reload() is not implemented then leave this commented out
        # and leave 'force-reload' as an alias for 'restart'.
        #
        log_daemon_msg "Reloading $DESC" "$NAME"
        do_reload
        log_end_msg $?
        ;;
    restart)
        #
        # If the "reload" option is implemented then remove the
        # 'force-reload' alias
        #
        log_daemon_msg "Restarting $DESC" "$NAME"
        do_graceful_stop
        do_start
        ;;
    *)
        echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}" >&2
        exit 3
        ;;
esac

Yeah, that was painful, but let’s keep going.

Make sure you have permissions to execute it

sudo chmod +x /etc/init.d/zuul-server

Next, create this list of directories

mkdir -p /etc/zuul/config
mkdir -p /var/log/zuul
mkdir -p /var/run/zuul
mkdir -p /var/run/zuul-merger
mkdir -p /var/lib/zuul/git
mkdir -p /var/lib/zuul/www/lib
Configuration files

Let’s start with the main configuration file (‘zuul.conf’).

Before you edit it, note that this file is very important and requires you to change some values according to your setup.

For example, you may have dedicated Gearman server, so you should not put “start = true” like in the following example. You should also point the Gerrit server option to your Gerrit and put the ssh key in place according to sshkey option.

vi /etc/zuul/zuul.conf

[gearman_server]
listen_address = 0.0.0.0
log_config = /etc/zuul/gearman-logging.conf
start = true

[gearman]
server = 127.0.0.1
port = 4730

[zuul]
layout_config = /etc/zuul/config/layout.yaml
log_config = /etc/zuul/server-logging.conf
pidfile = /var/run/zuul-server/zuul-server.pid
state_dir = /var/lib/zuul

[gerrit]
port = '29418'
server = <my_gerrit_IP_address>
sshkey = '/home/zuul/.ssh/id_rsa'
user = zuul

[launcher]
jenkins_jobs=/var/lib/zuul
log_config = /etc/zuul/launcher-logging.conf
workspace_root=/var/lib/zuul/workspace

[merger]
git_dir = /var/lib/zuul/git
log_config = /etc/zuul/merger-logging.conf
pidfile = /var/run/zuul-merger/zuul-merger.pid
zuul_url = 127.0.0.1

Good,  now let’s setup the logging configration files. Again, I’ll put one example for them all (launcher, gearman, merger and server), so you should copy & paste & switch values.

vi /etc/zuul/server-logging.conf

[loggers]
keys=root,zuul,gerrit,gear

[handlers]
keys=console,debug,normal

[formatters]
keys=simple

[logger_root]
level=WARNING
handlers=console

[logger_zuul]
level=DEBUG
handlers=debug,normal
qualname=zuul

[logger_gerrit]
level=INFO
handlers=debug,normal
qualname=gerrit

[logger_gear]
level=WARNING
handlers=debug,normal
qualname=gear

[handler_console]
level=WARNING
class=StreamHandler
formatter=simple
args=(sys.stdout,)

[handler_debug]
level=DEBUG
class=logging.handlers.WatchedFileHandler
formatter=simple
args=('/var/log/zuul/server-debug.log',)

[handler_normal]
level=INFO
class=logging.handlers.WatchedFileHandler
formatter=simple
args=('/var/log/zuul/server.log',)

[formatter_simple]
format=%(asctime)s %(levelname)s %(name)s: %(message)s
datefmt=

Stay strong, you are almost there.

Now the exciting part – let’s start the services

sudo systemctl start zuul-server
sudo systemctl start zuul-merger
sudo systemctl start zuul-launcher

Done! I’m proud of you!

There are other stuff you might need to configure/enable, but if you reached this step, it’s a piece of cake for you.

Connect Jenkins to Zuul

Or more accurately, connect Jenkins to Gearman.

If you chose to work with Jenkins, go to your Jenkins server and click in the left menu on “Manage Jenkins”. Next, click on “Configure System”.

gearman-jenkinsLook for Gearman configuration. You should see to blank fields for ‘Gearman Server Host’ and “Gearman Server Port”.

In “Gearman Server Host” put the IP address of your Gearman server or Zuul if you configured Zuul to spawn Gearman.

In the port you should put the number that match your zuul.conf configuration (in my setup it’s 4730).

Don’t  forget to check the “Enable Gearman” checkbox. It is also recommended to push on the “Test Connection” button to verify that Jenkins is able to connect the Gearman server.

Zuul web status page

Zuul includes nice status page to display all the pipelines and the active builds. Status source code

Unfortuantly, when you deploy Zuul, it won’t automatically install the status page. You can either use ‘ansible-zuul’ to deploy it (add the variable ‘status_page:yes’ in the main playbook).

Or you can follow the next manual steps.

Start by installing httpd

sudo yum -y install httpd

Next, you need to set selinux bool to permit httpd to make network connections

setsebool httpd_can_network_connect 1

There are several dependencies required to run the status page. There is a script to fetch and install them.

You need to install them in your zuul httpd directory (by default it should be /var/lib/zuul/www). To do that, simply put the script there and run it

cp /opt/zuul/etc/status/fetch-dependencies.sh /var/lib/zuul/www
/var/lib/zuul/www/fetch-dependencies.sh

Next you need set up all the html and javascript files in Zuul httpd directory. Those files should be taken from Zuul installation path (in my setup it’s ‘/opt/zuul/etc/status/public_html’)

cp -r /opt/zuul/etc/status/public_html/* /var/lib/zuul/www

Now setup zuul vhost section in httpd.conf.  Insert the following section in ‘/etc/httpd/conf/httpd.conf’ and make sure to change the variables to match your environment!

<VirtualHost *:80>
  ServerName zuul.localdomain
  ServerAdmin root
  DocumentRoot /var/lib/zuul/www

  <Directory /var/lib/zuul/www>
    Require all granted
    Allow from all
  </Directory>
  <Directory /usr/lib/git-core>
    Require all granted
    Allow from all
  </Directory>

  ErrorLog /var/log/httpd/zuul-error.log

  LogLevel warn

  CustomLog /var/log/httpd/zuul-access.log combined

  RewriteEngine on
  RewriteRule ^/status.json$ http://127.0.0.1:8001/status.json [P]
  RewriteRule ^/status/(.*) http://127.0.0.1:8001/status/$1 [P]

  AddOutputFilterByType DEFLATE application/json

  SetEnv GIT_PROJECT_ROOT /var/lib/zuul/git/
  SetEnv GIT_HTTP_EXPORT_ALL

  AliasMatch ^/p/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$ /var/lib/zuul/git/$1
  AliasMatch ^/p/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /var/lib/zuul/git/$1
  ScriptAlias /p/ /usr/lib/git-core/git-http-backend/

  <IfModule mod_cache.c>
    CacheDefaultExpire 5
    <IfModule mod_mem_cache.c>
      CacheEnable mem /status.json
      # 12MByte total cache size.
      MCacheSize 12288
      MCacheMaxObjectCount 10
      MCacheMinObjectSize 1
      # 8MByte max size per cache entry
      MCacheMaxObjectSize 8388608
      MCacheMaxStreamingBuffer 8388608
    </IfModule>
    <IfModule mod_cache_disk.c>
      CacheEnable disk /status.json
      CacheRoot /var/cache/apache2/mod_cache_disk
    </IfModule>
  </IfModule>
</VirtualHost>

It’s important to change lines 2-4 to match your environment profile.

That’s it, the only thing left for you to do, is to start httpd

sudo systemctl start httpd
sudo systemctl enable httpd

You can now try to access Zuul web status page in your browser: http://<zuul_server&gt;

Verify Zuul is installed properly

First, let’s check Zuul is in the processes table

ps -ef

zuul     12585     1  0 08:48 ?        00:00:01 /usr/bin/python /usr/bin/zuul-server -d
zuul     12590 12585  0 08:48 ?        00:00:00 /usr/bin/python /usr/bin/zuul-server -d

You can also check the service is running

sudo systemctl status zuul-server

● zuul-server.service - Zuul Server Service
 Loaded: loaded (/etc/systemd/system/zuul-server.service; enabled; vendor preset: disabled)
 Active: active (running) since Mon 2016-08-22 08:48:24 IDT; 10min ago
 Main PID: 12585 (zuul-server)
 CGroup: /system.slice/zuul-server.service
 ├─12585 /usr/bin/python /usr/bin/zuul-server -d
 └─12590 /usr/bin/python /usr/bin/zuul-server -d

Add your first job

Best way to check you have Zuul started and running properly, is to add your first job.

Edit layout.yaml (You can check where it’s configured in zuul.conf under [zuul] section. In my setup it’s at /etc/zuul/config/layout.yaml).

pipelines:
  - name: periodic
    source: gerrit
    manager: IndependentPipelineManager
    trigger:
      timer:
        - time: '0 * * * *'

projects:
  - name: my_project
    periodic:
      - my_project-periodic

Note: I would set the time to something close to your current time, so you can see the job is actually starts running.

Good, now reload zuul-server

sudo systemctl reload zuul-server

Now you can see in the server.log that your job is in ‘periodic’ pipeline.

less /var/log/zuul/server.log

2016-08-22 09:04:15,003 INFO zuul.IndependentPipelineManager: Configured Pipeline Manager periodic
2016-08-22 09:04:15,003 INFO zuul.IndependentPipelineManager:   Source: <zuul.source.gerrit.GerritSource object at 0x7f6d941f1e10>
2016-08-22 09:04:15,003 INFO zuul.IndependentPipelineManager:   Requirements:
2016-08-22 09:04:15,003 INFO zuul.IndependentPipelineManager:   Events:
2016-08-22 09:04:15,003 INFO zuul.IndependentPipelineManager:     <EventFilter types: timer ignore_deletes: True timespecs: 0 * * * *>
2016-08-22 09:04:15,004 INFO zuul.IndependentPipelineManager:   Projects:
2016-08-22 09:04:15,004 INFO zuul.IndependentPipelineManager:     my_project
2016-08-22 09:04:15,004 INFO zuul.IndependentPipelineManager:       <Job my_project-periodic-build>

Common Zuul Failures

Authentication failed

2016-08-18 15:31:14,348 ERROR gerrit.GerritWatcher: Could not connect to my_gerrit.com
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 174, in _run
    key_filename=self.keyfile)
  File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 380, in connect
    look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)
  File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 597, in _auth
    raise saved_exception
AuthenticationException: Authentication failed.

It means Zuul couldn’t connect to the Gerrit server. There are variety of reasons as to why you might see it – connectivity issues, bad key, firewall, etc.

In your zuul.conf file, you specified under ‘[gerrit]’ section, the variables ‘port’, ‘server’, ‘sshkey’ and ‘user’. Use the value of these variables to test the connection.

ssh -i /home/zuul/id_rsa zuul@my_gerrit.com

You should see something similar to the following output (depends on your Gerrit server)

****    Welcome to Gerrit Code Review    ****

  Hi Zuul, you have successfully connected over SSH.

status.json: service is unavailable

If, when accessing the Zuul web status page you see “status.json: service is unavailable”, then you are in the right place.

To confirm you have an issue, you can also have a look in your zuul httpd log (in my environment it’s /var/log/httpd/zuul-error.log)

sudo tail /var/log/httpd/zuul-error.log
HTTP: failed to make connection to backend: 127.0.0.1

It means httpd can’t make network connections and this usually because SElinux isn’t configured properly.

Try to permit it with setsebool and restart httpd

sudo setsebool httpd_can_network_connect 1
sudo systemctl restart httpd

Gearman: connection refused

If when trying to test the connection between Jenkins and Zuul you receive ‘test failed’ or ‘unable to connect’ then you need to change your Zuul configuration.

This is usually happens when you have under [gearman_server] section the ip address “127.0.0.1” instead of “0.0.0.0” as the value of ‘listen_address”

You should have similar section in your configuration:

[gearman_server]
listen_address = 0.0.0.0
log_config = /etc/zuul/gearman-logging.conf
start = true

Update: the user wangxf shared with us some of the issues he found and solved. Adding them here (thanks wangxf!):

“Host key verification failed. fatal: Could not read from remote repository”

If you get the following failure

stderr: ‘Host key verification failed.
fatal: Could not read from remote repository.
return repo
UnboundLocalError: local variable ‘repo’ referenced before assignment

You can fix them with the following steps:

cp id_rsa, id_ras.pub and known_hosts to your ssh folder of zuul.
chown -R zuul:zuul id_rsa id_rsa.pub know_hosts

“Fail to run git merge -s resolve FETCH_HEAD”

If the following command fails:  git merge -s resolve FETCH_HEAD to set your account’s default identity.

Add the following values in merger section:

zuul_url = 127.0.0.1
git_user_email=replaceme@domain
git_user_name=replaceme

/usr/lib/python2.7/site-packages/zuul/connection/gerrit.py raises URLError

If you get the following exception:

File “/usr/lib/python2.7/site-packages/zuul/connection/gerrit.py”, line 391, in getInfoRefs
raise URLError(err)

Add the following value under gerrit section

baseurl=http://<your_gerrit>

19 thoughts on “OpenStack Infra: How to deploy Zuul

  1. First of all, I’ve been reading your posts about zuul and they’re awesome, I’ve using this in particular for a solution to integrate several tools in a CI/CD solution. I noticed a couple of steps that are missing in this post.

    The first one is related with the script that starts the daemon in ubuntu /etc/init.d/zuul-server,

    1. The missing ‘#’ in the first line
    2. The closing bracket(}) in do_graceful_stop() function

    And the last one is related with the permissions on that file:

    # chmod +x /etc/init.d/zuul-server

    Thanks again for sharing this information

    Like

      1. I share with some problem and how to fixed it.
        issue:
        stderr: ‘Host key verification failed.
        fatal: Could not read from remote repository.
        return repo
        UnboundLocalError: local variable ‘repo’ referenced before assignment
        fix:
        1. cp id_rsa id_ras.pub known_hosts to your ssh key folder for zuul.
        2. chown -R zuul:zuul id_rsa
        3. chown -R zuul:zuul id_rsa.pub
        4. chown -R zuul:zuul known_hosts

        Like

      2. issue:
        cmdline: git merge -s resolve FETCH_HEAD
        to set your account’s default identity.
        Omit –global to set the identity only in this repository.
        fix:
        add two values under merger section, like this:

        zuul_url = 127.0.0.1
        git_user_email=replaceme@awcloud.com
        git_user_name=replaceme

        Like

      3. issue:
        File “/usr/lib/python2.7/site-packages/zuul/connection/gerrit.py”, line 391, in getInfoRefs
        raise URLError(err)
        URLError:

        fix:
        add baseurl under gerrit section:
        ….
        baseurl=http://gerrit.corp.awcloud.com

        Like

    1. How did you install it? using ansible-zuul or the manual steps here? if ansible-zuul, try to run this: “setsebool httpd_can_network_connect 1”

      Check also if there is any useful information in /var/log/httpd

      Like

      1. I using manual steps
        /var/log/httpd/error.log:

        attempt to make remote request from mod_rewrite without proxy enabled: proxy:http://127.0.0.1:8001/status.json

        Like

      2. Tell me please. Zuul status displays tests in the last 5 minutes, then they disappear.
        How to make that all history was displayed?

        Like

      3. I don’t think you want to display all the history, because it’s a CI system and you’ll end up with a lot of changes in your pipelines, even in a small environment.

        If still want to change it, you can do it in status/public_html/jquery.zuul.js

        Like

  2. Hello trying to install ODL Boron SR3 ML2 plugin with Neutron Networking for Media Scale testing , Video Processing .. unable to find a good reference .. little help

    Cheers,

    V.

    Like

  3. I used ansible depoly zuul system.

    ERROR! the role ‘ansible-zuul’ was not found in /tmp/ansible-zuul/roles:/root/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/tmp/ansible-zuul

    The error appears to have been in ‘/tmp/ansible-zuul/deploy.yml’: line 8, column 7, but may
    be elsewhere in the file depending on the exact syntax problem.

    The offending line appears to be:

    roles:
    – role: ansible-zuul
    ^ here

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s