Friday, January 16, 2015

Setting up a Continuous Integration Environment with Jenkins and Docker PART 4 Private Docker Registry

In Part 4 of this series I will be walking through the steps that were needed to set up my own Private Docker Registry on a CentOS 6 Linux server.

Step 1) Install Prerequisites  
The Docker registry is written in python, thus we need to install python development utilities and some libraries.
> yum install python-pip phython-devel libevent libevent-devel pyliblzma jjpython-gunicorn
...lots of output ...
Is this ok [y/N]: y

=== may not be needed
yum install mod_wsgi.x86_64
yum install  python-wsgiproxy.noarch
yum install python-moksha-wsgi.noarch python-wsgi-jsonrpc.noarch

Step 2) Install and Configure Docker Registry
>pip install docker-registry
...lots of output ...
Cleaning up...

By default Docker saves its data under the /tmp directory.  I will create a permanent location, then will need to configure Docker to to use the new location.

> mkdir /var/docker-registry

Locate the file config_sample.yml. On my system it is located in /usr/lib/python2.6/site-packages/config. Now copy the file to config.yml

> cd /usr/lib/python2.6/site-packages/config
> cp config_sample.yml config.yml
Edit config.yml, then replace any reference to /tmp with /var/docker-registry

Change this:

 sqlalchemy_index_database: _env:SQLALCHEMY_INDEX_DATABASE:sqlite:////tmp/docker-registry.db

To this:

 sqlalchemy_index_database: _env:SQLALCHEMY_INDEX_DATABASE:sqlite:////var/docker-registry/docker-registry.db

Change this:
local: &local
    <<: *common
    storage: local

    storage_path: _env:STORAGE_PATH:/tmp/registry

To this:

local: &local
    <<: *common
    storage: local

    storage_path: _env:STORAGE_PATH:/var/docker-registry/registry

Change this:

glance: &glance
    <<: *common
    storage: glance
    storage_alternate: _env:GLANCE_STORAGE_ALTERNATE:file

    storage_path: _env:STORAGE_PATH:/tmp/registry

To this:

glance: &glance
    <<: *common
    storage: glance
    storage_alternate: _env:GLANCE_STORAGE_ALTERNATE:file

    storage_path: _env:STORAGE_PATH:/var/docker-registry/registry

If you want to do something more complex like using AWS S3 buckets, Google Cloud Storage, or Openstack you can configure it in this file.

Now that the config is in place let's try to start the server.

> gunicorn --access-logfile - --debug -k gevent -b -w 1 docker_registry.wsgi:application
07/Jan/2015:16:52:32 +0000 WARNING: Cache storage disabled!
07/Jan/2015:16:52:32 +0000 WARNING: LRU cache disabled!
07/Jan/2015:16:52:32 +0000 DEBUG: Will return docker-registry.drivers.file.Storage

Your output should be similar.
Go ahead and hit CTRL-C to kill the process.

STEP 3) Docker Registry as a Service on start up

First check to see if the Docker Registry is configured as a service
> chkconfig --list | grep docker
docker-registry 0:off   1:off   2:on    3:on    4:on    5:on    6:off

Next start the Docker Registry.  I use restart here just so you can see the output.

> service docker-registry restart
Stopping docker-registry: [FAILED]
Starting docker-registry: OK

Awesome! it looks like it is up and running, lets verify that docker-registry is up.

> ps -ef | grep docker
root      5445     1  0 Jan07 pts/0    00:00:00 /usr/bin/python /usr/bin/gunicorn --access-logfile - --debug --max-requests 100 --graceful-timeout 3600 -t 3600 -k gevent -b -w 4 docker_registry.wsgi:application
root      5451  5445  0 Jan07 pts/0    00:00:23 /usr/bin/python /usr/bin/gunicorn --access-logfile - --debug --max-requests 100 --graceful-timeout 3600 -t 3600 -k gevent -b -w 4 docker_registry.wsgi:application
root      5452  5445  0 Jan07 pts/0    00:00:25 /usr/bin/python /usr/bin/gunicorn --access-logfile - --debug --max-requests 100 --graceful-timeout 3600 -t 3600 -k gevent -b -w 4 docker_registry.wsgi:application
root      5455  5445  0 Jan07 pts/0    00:00:25 /usr/bin/python /usr/bin/gunicorn --access-logfile - --debug --max-requests 100 --graceful-timeout 3600 -t 3600 -k gevent -b -w 4 docker_registry.wsgi:application
root      5458  5445  0 Jan07 pts/0    00:00:25 /usr/bin/python /usr/bin/gunicorn --access-logfile - --debug --max-requests 100 --graceful-timeout 3600 -t 3600 -k gevent -b -w 4 docker_registry.wsgi:application
root      6211 20583  0 13:46 pts/0    00:00:00 grep docker

STEP 4) Adding Authentication

Borrowing heavily from the References linked to at the bottom of this post I will be implementing Nginx (pronounced Engine X) as a front end for access to the Docker Registry. By default the Docker Registry listens on port 5000. In many labs and environments port 5000 is not available through internal firewalls and certainly not on the open internet. Thus we need a method for forwarding the requests to Docker.  Nginx is a fairly new opensource web server for http, https and other protocols, it can be used as a load balancer and reverse proxy server.  It has features that allow it to handle a higher volume of requests than Apache, by handling requests differently. There are many discussions and blog posts on the web about Nginx vs Apache. I recommend reading a couple. 

Lets verify that htpasswd is installed, this is an Apache tool that can be used to generate encrypted passwords.
> yum provides \*bin/htpasswd
Loaded plugins: fastestmirror
httpd-tools-2.2.15-39.el6.centos.x86_64 : Tools for use with the Apache HTTP Server
Repo        : base
Matched from:
Filename    : /usr/bin/htpasswd

Now we know what httpd package has htpasswd lets try to install it. 
> yum install httpd-tools
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base:
 * epel:
 * extras:
 * updates:
Package httpd-tools-2.2.15-39.el6.centos.x86_64 already installed and latest version
Nothing to do

It has already been installed, if it wasn't installed this would have installed it.
Install Nginx

yum install nginx
...lots of output...
Is this ok [y/N]:y

Create a Docker user, create a password for the user when prompted.
After this step we will have a password file with our users, and a Docker registry available.

> htpasswd -c /etc/nginx/docker-registry.htpasswd pete
New password:
Re-type new password:
Adding password for user pete
To add additional users, rerun the above command without the -c flag. The -c flag creates the password file. If you reuse -c you will overwrite the file.

Next I need to tell Nginx to use the created authentication file, and forward requests to the Docker registry. 
Create the file: /etc/nginx/conf.d/docker-registry.conf
then add the following content, and edit as appropriate for your environment.

# For versions of Nginx > 1.3.9 that include chunked transfer encoding support
# Replace with appropriate values where necessary

upstream docker-registry {
 server localhost:5000;

server {
 listen 8080;

 # ssl on;
 # ssl_certificate /etc/nginx/ssl/<servername>.crt;
 # ssl_certificate_key /etc/nginx/ssl/<servername>.key;

 proxy_set_header Host       $http_host;   # required for Docker client sake
 proxy_set_header X-Real-IP  $remote_addr; # pass on real client IP

 client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads

 # required to avoid HTTP 411: see Issue #1486 (
 chunked_transfer_encoding on;

 location / {
     # let Nginx know about our auth file
     auth_basic              "Restricted";
     auth_basic_user_file    docker-registry.htpasswd;

     proxy_pass http://docker-registry;
 location /_ping {
     auth_basic off;
     proxy_pass http://docker-registry;
 location /v1/_ping {
     auth_basic off;
     proxy_pass http://docker-registry;


Restart Nginx to activate the virtual host, and test the connections to Docker and Nginx
> service nginx restart
Stopping nginx:                                            [  OK  ]
Starting nginx:                                            [  OK  ]

>curl localhost:5000

>curl localhost:8080
<head><title>401 Authorization Required</title></head>
<body bgcolor="white">
<center><h1>401 Authorization Required</h1></center>

Great, Docker is running, and Nginx is forwarding the request to Docker.
Now lets try connecting to Docker with the username created earlier.
>curl pete:mypassword@localhost:8080

STEP 5) Adding Secure Authentication (SSL)
In the previous step we added basic http authentication, however this is not very secure since connections to the registry are unencrypted.  In this step I'll show you how to enable SSL (https), and setup a self signed certificate.

Lets begin by editing the file: /etc/nginx/conf.d/docker-registry.conf 
Remove the # symbol in front of the SSL lines.  The result should look like this;
  ssl on;
  ssl_certificate /etc/nginx/ssl/<servername>.crt;
  ssl_certificate_key /etc/nginx/ssl/<servername>.key;

Save the file. Nginx is now configured to use SSL and will look for the certificate and key at the name and location listed in the file. Please note: As far I know there are no standards for the location of your certificate I chose the above paths for convenience .

Make sure you are logged into the server for which you want to create the SSL Certificate then enter the following, making sure to replace <servername> with the fully qualified domain name of your system.
The first thing I need to do is create the key and certificate request.
> cd /etc/httpd/conf
> openssl req -new -newkey rsa:2048 -nodes -keyout <servername>.key -out <servername>.csr

Next answer the questions The purpose of the questions is to add randomization in the certificate. Answer the questions with values suitable to your environment. Press Enter to leave the field blank, or to select the default. In my experience the last three questions can be left blank. Make sure that "Common Name", (the Fully Qualified Domain Name) matches the hostname you will use to connect to Docker.

Generating a 2048 bit RSA private key
writing new private key to 'myservername.key'
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.

Country Name (2 letter code) [GB]:US
State or Province Name (full name) [Berkshire]:MASSACHUSETTS
Locality Name (eg, city) [Newbury]:BOSTON
Organization Name (eg, company) [My Company Ltd]:
Organizational Unit Name (eg, section) []:IT
Common Name (eg, your name or your server's hostname) []: <servername>
Email Address []:
Please enter the following 'extra' attributes to be sent with your
certificate request
A challenge password []:
An optional company name []:

Next Create the following directory, then copy the private key and the Certificate Request to it.
> mkdir /etc/nginx/ssl
> cp <servername>.key /etc/nginx/ssl
> cp <servername>.csr /etc/nginx/ssl
Creating a self-signed certificate
If you do not plan to have the certificate signed by a Certificate Authority (CA) or if you plan to test the new SSL implementation while the CA is signing your certificate, you can generate a self-signed certificate. This temporary certificate generates errors in the client browser to the effect that the signing certificate authority is unknown and not trusted.
To generate a temporary certificate that is good for 365 days  Issue the following commands:

> cd /etc/nginx/ssl
> openssl x509 -req -days 365 -in <servername>.csr -signkey <servername>.key -out <servername>.crt

SSL Test

Restart Nginx to reload the configuration and SSL Keys, if all goes well there should not be any errors.
> service nginx restart
Stopping nginx: OK
Starting nginx: OK
Lets try a couple of different curl commands.
> cd /etc/nginx/ssl
> curl pete:mypassword@localhost:8080    
400 Bad Request
# You can see that Nginx is forwarding us to https, but I'm getting a 400 Bad Request.
# Lets try specifying https 

>curl  https://pete:mypassword@localhost:8080

curl: (60) Peer certificate cannot be authenticated with known CA certificates
More details here:

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

#Curl is having trouble now because I have created a self signed certificate
#and the curl command doesn't trust the the cert, due to the certificate not
# being in curl's default trust store. Lets point directly to new certificate.

> curl --cacert myserver.crt https://pete:mypassword@localhost:8080
curl: (51) SSL: certificate subject name 'myserver' does not match target host name 'localhost'

#Curl still is having trouble because we tried to connect to localhost, but 
#created the certificate with the fully qualified domain name of the server.
#Lets try one more time.

> curl --cacert myserver.crt https://pete:mypassword@myserver:8080


STEP 6) Accessing the Docker Registry from another server.

Specific to CentOS6 and possibly Redhat there is an application called docker that is a "docking application for KDE3, and GNOME2". This app will cause conflicts with docker-io. We have several tasks we must do on the client prior being able to login. 

  1. Remove the docker app if it is installed.
  2. Setup a server as a Docker client.
  3. Import the Docker Registry's self-signed certificate into the systems trust store.

Install Docker Client
> rpm -e docker 
> yum install docker-io 
...lots of output...
Is this ok [y/N]: y

Once Docker is installed, you will need to start the docker daemon, then verify that docker is configured to start at boot.
> service docker start 
Starting cgconfig service:                                 [  OK  ]
Starting docker:                                           [  OK  ]
>chkconfig --list docker
docker          0:off   1:off   2:on    3:on    4:on    5:on    6:off
If we were to attempt to login now using the command below we would encounter several issues.  I will work through each issue I encountered hopefully this will save you time.
> docker login https://myserver
If your docker registry server is not yet part of your local DNS, add the servers ip address and fully qualified domain name (FQDN) into your client systems /etc/hosts file. On start up the Docker daemon creates a file /var/run/docker.sock. This file has the following ownership (root,docker) and permissions (644).
srw-rw---- 1 root docker 0 Jan 13 17:24 /var/run/docker.sock

When attempting to run the docker login command as a client user, an error similar to the following is returned.

dial unix /var/run/docker.sock: permission denied

A temporary fix is to reset the permissions to by executing the chmod command, however once you restart the docker daemon the permissions return to 644.

> chmod 666 /var/run/docker.sock
srw-rw-rw- 1 root docker 0 Jan 13 17:24 /var/run/docker.sock

A more permanent solution is to add your clients username to the docker group. First verify that you have a docker group by looking at the /etc/group file. To add the the docker group and add your client to the docker group, execute the following commands;
> groupadd docker
> useradd -aG docker username

After adding the client user to the docker group the user must log out then log back into their account, and the error should go away.

The next thing we need to do is add our self signed certificate to the client systems list of trusted certificates. The following solution is not ideal but it worked for me, I will update this post if I come across a better solution. You will need to go back to your Docker Registry server and copy the registry servers certificate to the client server.

On the registry server,


> cd /etc/nginx/ssl
> cat myserver.crt


Copy the output and paste it to a file on your client server such as /tmp/docker-registry.crt then add it to your systems certificate authority bundle, after making a backup copy of course.

> cd /etc/pki/tls/certs
> cp ca-bundle.crt ca-bundle.crt.bak
> cat /tmp/docker-registry.crt >> ca-bundle.crt
> service docker restart

Finally we are ready to attempt our login, if successful a file (.dockercfg) will be created in the end users home directory.

> docker login
Username: pete
Login Succeeded

Lets see if we can run a simple docker command.

> docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-253:0-918242-pool
 Pool Blocksize: 65.54 kB
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 305.7 MB
 Data Space Total: 107.4 GB
 Metadata Space Used: 729.1 kB
 Metadata Space Total: 2.147 GB
 Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.el6.x86_64
Operating System: 

At this point I have successfully installed python pre-requisites, docker-io, and Nginx. I have configured docker and Nginx as a reverse proxy to forward requests to docker, setup SSL and several users. Finally I was able to log in to the docker registry and run a simple docker command.  In Part 5 of this tutorial, I will begin using this docker registry. 


How to set up a private docker registry on ubuntu