Bump to version 0.8.1

Docker-DCO-1.1-Signed-off-by: Michael Crosby <michael@crosbymichael.com> (github: crosbymichael)

CHANGELOG.md

@@ -1,5 +1,50 @@
 # Changelog
 
+## 0.8.1 (2014-02-18)
+
+#### Builder
+
+- Avoid extra mount/unmount during build. This removes an unneeded mount/unmount operation which was causing problems with devicemapper
+- Fix regression with ADD of tar files. This stops Docker from decompressing tarballs added via ADD from the local file system
+- Add error to `docker build --rm`. This adds a missing error check to ensure failures to remove containers are detected and reported
+
+#### Documentation
+
+* Update issue filing instructions
+* Warn against the use of symlinks for Docker's storage folder
+* Replace the Firefox example with an IceWeasel example
+* Rewrite the PostgresSQL example using a Dockerfile and add more details to it
+* Improve the OS X documentation
+
+#### Remote API
+
+- Fix broken images API for version less than 1.7
+- Use the right encoding for all API endpoints which return JSON
+- Move remote api client to api/
+- Queue calls to the API using generic socket wait 
+
+#### Runtime
+
+- Fix the use of custom settings for bridges and custom bridges
+- Refactor the devicemapper code to avoid many mount/unmount race conditions and failures
+- Remove two panics which could make Docker crash in some situations
+- Don't ping registry from the CLI client
+- Enable skip_block_zeroing for devicemapper. This stops devicemapper from always zeroing entire blocks
+- Fix --run in `docker commit`. This makes docker commit store `--run` in the image configuration
+- Remove directory when removing devicemapper device. This cleans up leftover mount directories
+- Drop NET_ADMIN capability for non-privileged containers. Unprivileged containers can't change their network configuration
+- Ensure `docker cp` stream is closed properly
+- Avoid extra mount/unmount during container registration. This removes an unneeded mount/unmount operation which was causing problems with devicemapper
+- Stop allowing tcp:// as a default tcp bin address which binds to 127.0.0.1:4243 and remove the default port
++ Mount-bind the PTY as container console. This allows tmux and screen to run in a container
+- Clean up archive closing. This fixes and improves archive handling
+- Fix engine tests on systems where temp directories are symlinked
+- Add test methods for save and load
+- Avoid temporarily unmounting the container when restarting it. This fixes a race for devicemapper during restart
+- Support submodules when building from a GitHub repository
+- Quote volume path to allow spaces
+- Fix remote tar ADD behavior. This fixes a regression which was causing Docker to extract tarballs
+
 ## 0.8.0 (2014-02-04)
 
 #### Notable features since 0.7.0

CONTRIBUTING.md

@@ -7,8 +7,10 @@ feels wrong or incomplete.
 ## Reporting Issues
 
 When reporting [issues](https://github.com/dotcloud/docker/issues) 
-on GitHub please include your host OS ( Ubuntu 12.04, Fedora 19, etc... )
-and the output of `docker version` along with the output of `docker info` if possible.  
+on GitHub please include your host OS (Ubuntu 12.04, Fedora 19, etc),
+the output of `uname -a` and the output of `docker version` along with
+the output of `docker info`. Please include the steps required to reproduce
+the problem if possible and applicable.
 This information will help us review and fix your issue faster.
 
 ## Build Environment
@@ -86,6 +88,8 @@ curl -o .git/hooks/pre-commit https://raw.github.com/edsrzf/gofmt-git-hook/maste
 Pull requests descriptions should be as clear as possible and include a
 reference to all the issues that they address.
 
+Pull requests mustn't contain commits from other users or branches.
+
 Code review comments may be added to your pull request. Discuss, then make the
 suggested modifications and push additional commits to your feature branch. Be
 sure to post a comment after pushing. The new commits will show up in the pull
@@ -105,6 +109,18 @@ name and email address match your git configuration. The AUTHORS file is
 regenerated occasionally from the git commit history, so a mismatch may result
 in your changes being overwritten.
 
+### Merge approval
+
+Docker maintainers use LGTM (looks good to me) in comments on the code review
+to indicate acceptance.
+
+A change requires LGTMs from an absolute majority of the maintainers of each
+component affected. For example, if a change affects docs/ and registry/, it
+needs an absolute majority from the maintainers of docs/ AND, separately, an
+absolute majority of the maintainers of registry
+
+For more details see [MAINTAINERS.md](hack/MAINTAINERS.md)
+
 ### Sign your work
 
 The sign-off is a simple line at the end of the explanation for the
@@ -113,7 +129,7 @@ pass it on as an open-source patch.  The rules are pretty simple: if you
 can certify the below:
 
 ```
-Docker Developer Grant and Certificate of Origin 1.1
+Docker Developer Certificate of Origin 1.1
 
 By making a contribution to the Docker Project ("Project"), I represent and
 warrant that:
@@ -163,7 +179,7 @@ If you have any questions, please refer to the FAQ in the [docs](http://docs.doc
 * Step 1: learn the component inside out
 * Step 2: make yourself useful by contributing code, bugfixes, support etc.
 * Step 3: volunteer on the irc channel (#docker@freenode)
-* Step 4: propose yourself at a scheduled #docker-meeting
+* Step 4: propose yourself at a scheduled docker meeting in #docker-dev
 
 Don't forget: being a maintainer is a time investment. Make sure you will have time to make yourself available.
 You don't have to be a maintainer to make a difference on the project!

FIXME

@@ -11,20 +11,14 @@ They are just like FIXME comments in the source code, except we're not sure wher
 to put them - so we put them here :)
 
 
-* Merge Runtime, Server and Builder into Runtime
 * Run linter on codebase
 * Unify build commands and regular commands
 * Move source code into src/ subdir for clarity
 * docker build: on non-existent local path for ADD, don't show full absolute path on the host
-* docker tag foo REPO:TAG
 * use size header for progress bar in pull
 * Clean up context upload in build!!!
 * Parallel pull
-* Always generate a resolv.conf per container, to avoid changing resolv.conf under thne container's feet
-* Save metadata with import/export (#1974)
 * Upgrade dockerd without stopping containers
 * Simple command to remove all untagged images (`docker rmi $(docker images | awk '/^<none>/ { print $3 }')`)
 * Simple command to clean up containers for disk space
-* Caching after an ADD (#880)
 * Clean up the ProgressReader api, it's a PITA to use
-* Use netlink instead of iproute2/iptables (#925)

Makefile

@@ -3,7 +3,7 @@
 GIT_BRANCH := $(shell git rev-parse --abbrev-ref HEAD)
 DOCKER_IMAGE := docker:$(GIT_BRANCH)
 DOCKER_DOCS_IMAGE := docker-docs:$(GIT_BRANCH)
-DOCKER_RUN_DOCKER := docker run -rm -i -t -privileged -e TESTFLAGS -v $(CURDIR)/bundles:/go/src/github.com/dotcloud/docker/bundles "$(DOCKER_IMAGE)"
+DOCKER_RUN_DOCKER := docker run -rm -i -t -privileged -e TESTFLAGS -v "$(CURDIR)/bundles:/go/src/github.com/dotcloud/docker/bundles" "$(DOCKER_IMAGE)"
 
 default: binary
 

VERSION

@@ -1 +1 @@
-0.8.0
+0.8.1

api/api.go

@@ -10,6 +10,7 @@ import (
 	"fmt"
 	"github.com/dotcloud/docker/auth"
 	"github.com/dotcloud/docker/engine"
+	"github.com/dotcloud/docker/pkg/listenbuffer"
 	"github.com/dotcloud/docker/pkg/systemd"
 	"github.com/dotcloud/docker/utils"
 	"github.com/gorilla/mux"
@@ -25,15 +26,28 @@ import (
 	"strconv"
 	"strings"
 	"syscall"
+	"time"
 )
 
+// FIXME: move code common to client and server to common.go
 const (
 	APIVERSION        = 1.9
 	DEFAULTHTTPHOST   = "127.0.0.1"
-	DEFAULTHTTPPORT   = 4243
 	DEFAULTUNIXSOCKET = "/var/run/docker.sock"
 )
 
+var (
+	activationLock chan struct{}
+)
+
+func ValidateHost(val string) (string, error) {
+	host, err := utils.ParseHost(DEFAULTHTTPHOST, DEFAULTUNIXSOCKET, val)
+	if err != nil {
+		return val, err
+	}
+	return host, nil
+}
+
 type HttpApiFunc func(eng *engine.Engine, version float64, w http.ResponseWriter, r *http.Request, vars map[string]string) error
 
 func init() {
@@ -99,6 +113,15 @@ func writeJSON(w http.ResponseWriter, code int, v engine.Env) error {
 	return v.Encode(w)
 }
 
+func streamJSON(job *engine.Job, w http.ResponseWriter, flush bool) {
+	w.Header().Set("Content-Type", "application/json")
+	if flush {
+		job.Stdout.Add(utils.NewWriteFlusher(w))
+	} else {
+		job.Stdout.Add(w)
+	}
+}
+
 func getBoolParam(value string) (bool, error) {
 	if value == "" {
 		return false, nil
@@ -205,7 +228,7 @@ func getImagesJSON(eng *engine.Engine, version float64, w http.ResponseWriter, r
 	job.Setenv("all", r.Form.Get("all"))
 
 	if version >= 1.7 {
-		job.Stdout.Add(w)
+		streamJSON(job, w, false)
 	} else if outs, err = job.Stdout.AddListTable(); err != nil {
 		return err
 	}
@@ -222,13 +245,14 @@ func getImagesJSON(eng *engine.Engine, version float64, w http.ResponseWriter, r
 				outLegacy := &engine.Env{}
 				outLegacy.Set("Repository", parts[0])
 				outLegacy.Set("Tag", parts[1])
-				outLegacy.Set("ID", out.Get("ID"))
+				outLegacy.Set("Id", out.Get("Id"))
 				outLegacy.SetInt64("Created", out.GetInt64("Created"))
 				outLegacy.SetInt64("Size", out.GetInt64("Size"))
 				outLegacy.SetInt64("VirtualSize", out.GetInt64("VirtualSize"))
 				outsLegacy.Add(outLegacy)
 			}
 		}
+		w.Header().Set("Content-Type", "application/json")
 		if _, err := outsLegacy.WriteListTo(w); err != nil {
 			return err
 		}
@@ -256,9 +280,8 @@ func getEvents(eng *engine.Engine, version float64, w http.ResponseWriter, r *ht
 		return err
 	}
 
-	w.Header().Set("Content-Type", "application/json")
 	var job = eng.Job("events", r.RemoteAddr)
-	job.Stdout.Add(utils.NewWriteFlusher(w))
+	streamJSON(job, w, true)
 	job.Setenv("since", r.Form.Get("since"))
 	return job.Run()
 }
@@ -269,7 +292,7 @@ func getImagesHistory(eng *engine.Engine, version float64, w http.ResponseWriter
 	}
 
 	var job = eng.Job("history", vars["name"])
-	job.Stdout.Add(w)
+	streamJSON(job, w, false)
 
 	if err := job.Run(); err != nil {
 		return err
@@ -282,7 +305,7 @@ func getContainersChanges(eng *engine.Engine, version float64, w http.ResponseWr
 		return fmt.Errorf("Missing parameter")
 	}
 	var job = eng.Job("changes", vars["name"])
-	job.Stdout.Add(w)
+	streamJSON(job, w, false)
 
 	return job.Run()
 }
@@ -299,7 +322,7 @@ func getContainersTop(eng *engine.Engine, version float64, w http.ResponseWriter
 	}
 
 	job := eng.Job("top", vars["name"], r.Form.Get("ps_args"))
-	job.Stdout.Add(w)
+	streamJSON(job, w, false)
 	return job.Run()
 }
 
@@ -320,7 +343,7 @@ func getContainersJSON(eng *engine.Engine, version float64, w http.ResponseWrite
 	job.Setenv("limit", r.Form.Get("limit"))
 
 	if version >= 1.5 {
-		job.Stdout.Add(w)
+		streamJSON(job, w, false)
 	} else if outs, err = job.Stdout.AddTable(); err != nil {
 		return err
 	}
@@ -333,6 +356,7 @@ func getContainersJSON(eng *engine.Engine, version float64, w http.ResponseWrite
 			ports.ReadListFrom([]byte(out.Get("Ports")))
 			out.Set("Ports", displayablePorts(ports))
 		}
+		w.Header().Set("Content-Type", "application/json")
 		if _, err = outs.WriteListTo(w); err != nil {
 			return err
 		}
@@ -366,7 +390,7 @@ func postCommit(eng *engine.Engine, version float64, w http.ResponseWriter, r *h
 		env    engine.Env
 		job    = eng.Job("commit", r.Form.Get("container"))
 	)
-	if err := config.Import(r.Body); err != nil {
+	if err := config.Decode(r.Body); err != nil {
 		utils.Errorf("%s", err)
 	}
 
@@ -425,8 +449,12 @@ func postImagesCreate(eng *engine.Engine, version float64, w http.ResponseWriter
 		job.Stdin.Add(r.Body)
 	}
 
-	job.SetenvBool("json", version > 1.0)
-	job.Stdout.Add(utils.NewWriteFlusher(w))
+	if version > 1.0 {
+		job.SetenvBool("json", true)
+		streamJSON(job, w, true)
+	} else {
+		job.Stdout.Add(utils.NewWriteFlusher(w))
+	}
 	if err := job.Run(); err != nil {
 		if !job.Stdout.Used() {
 			return err
@@ -465,7 +493,7 @@ func getImagesSearch(eng *engine.Engine, version float64, w http.ResponseWriter,
 	var job = eng.Job("search", r.Form.Get("term"))
 	job.SetenvJson("metaHeaders", metaHeaders)
 	job.SetenvJson("authConfig", authConfig)
-	job.Stdout.Add(w)
+	streamJSON(job, w, false)
 
 	return job.Run()
 }
@@ -482,8 +510,12 @@ func postImagesInsert(eng *engine.Engine, version float64, w http.ResponseWriter
 	}
 
 	job := eng.Job("insert", vars["name"], r.Form.Get("url"), r.Form.Get("path"))
-	job.SetenvBool("json", version > 1.0)
-	job.Stdout.Add(w)
+	if version > 1.0 {
+		job.SetenvBool("json", true)
+		streamJSON(job, w, false)
+	} else {
+		job.Stdout.Add(w)
+	}
 	if err := job.Run(); err != nil {
 		if !job.Stdout.Used() {
 			return err
@@ -532,8 +564,12 @@ func postImagesPush(eng *engine.Engine, version float64, w http.ResponseWriter,
 	job := eng.Job("push", vars["name"])
 	job.SetenvJson("metaHeaders", metaHeaders)
 	job.SetenvJson("authConfig", authConfig)
-	job.SetenvBool("json", version > 1.0)
-	job.Stdout.Add(utils.NewWriteFlusher(w))
+	if version > 1.0 {
+		job.SetenvBool("json", true)
+		streamJSON(job, w, true)
+	} else {
+		job.Stdout.Add(utils.NewWriteFlusher(w))
+	}
 
 	if err := job.Run(); err != nil {
 		if !job.Stdout.Used() {
@@ -635,7 +671,7 @@ func deleteImages(eng *engine.Engine, version float64, w http.ResponseWriter, r
 		return fmt.Errorf("Missing parameter")
 	}
 	var job = eng.Job("image_delete", vars["name"])
-	job.Stdout.Add(w)
+	streamJSON(job, w, false)
 	job.SetenvBool("autoPrune", version > 1.1)
 
 	return job.Run()
@@ -815,7 +851,7 @@ func getContainersByName(eng *engine.Engine, version float64, w http.ResponseWri
 		return fmt.Errorf("Missing parameter")
 	}
 	var job = eng.Job("inspect", vars["name"], "container")
-	job.Stdout.Add(w)
+	streamJSON(job, w, false)
 	job.SetenvBool("conflict", true) //conflict=true to detect conflict between containers and images in the job
 	return job.Run()
 }
@@ -825,7 +861,7 @@ func getImagesByName(eng *engine.Engine, version float64, w http.ResponseWriter,
 		return fmt.Errorf("Missing parameter")
 	}
 	var job = eng.Job("inspect", vars["name"], "image")
-	job.Stdout.Add(w)
+	streamJSON(job, w, false)
 	job.SetenvBool("conflict", true) //conflict=true to detect conflict between containers and images in the job
 	return job.Run()
 }
@@ -865,11 +901,11 @@ func postBuild(eng *engine.Engine, version float64, w http.ResponseWriter, r *ht
 	}
 
 	if version >= 1.8 {
-		w.Header().Set("Content-Type", "application/json")
 		job.SetenvBool("json", true)
+		streamJSON(job, w, true)
+	} else {
+		job.Stdout.Add(utils.NewWriteFlusher(w))
 	}
-
-	job.Stdout.Add(utils.NewWriteFlusher(w))
 	job.Stdin.Add(r.Body)
 	job.Setenv("remote", r.FormValue("remote"))
 	job.Setenv("t", r.FormValue("t"))
@@ -910,9 +946,12 @@ func postContainersCopy(eng *engine.Engine, version float64, w http.ResponseWrit
 	}
 
 	job := eng.Job("container_copy", vars["name"], copyData.Get("Resource"))
-	job.Stdout.Add(w)
+	streamJSON(job, w, false)
 	if err := job.Run(); err != nil {
 		utils.Errorf("%s", err.Error())
+		if strings.Contains(err.Error(), "No such container") {
+			w.WriteHeader(http.StatusNotFound)
+		}
 	}
 	return nil
 }
@@ -1126,7 +1165,7 @@ func ListenAndServe(proto, addr string, eng *engine.Engine, logging, enableCors
 		}
 	}
 
-	l, err := net.Listen(proto, addr)
+	l, err := listenbuffer.NewListenBuffer(proto, addr, activationLock, 15*time.Minute)
 	if err != nil {
 		return err
 	}
@@ -1168,8 +1207,15 @@ func ListenAndServe(proto, addr string, eng *engine.Engine, logging, enableCors
 // ServeApi loops through all of the protocols sent in to docker and spawns
 // off a go routine to setup a serving http.Server for each.
 func ServeApi(job *engine.Job) engine.Status {
-	protoAddrs := job.Args
-	chErrors := make(chan error, len(protoAddrs))
+	var (
+		protoAddrs = job.Args
+		chErrors   = make(chan error, len(protoAddrs))
+	)
+	activationLock = make(chan struct{})
+
+	if err := job.Eng.Register("acceptconnections", AcceptConnections); err != nil {
+		return job.Error(err)
+	}
 
 	for _, protoAddr := range protoAddrs {
 		protoAddrParts := strings.SplitN(protoAddr, "://", 2)
@@ -1186,8 +1232,15 @@ func ServeApi(job *engine.Job) engine.Status {
 		}
 	}
 
+	return engine.StatusOK
+}
+
+func AcceptConnections(job *engine.Job) engine.Status {
 	// Tell the init daemon we are accepting requests
 	go systemd.SdNotify("READY=1")
 
+	// close the lock so the listeners start accepting connections
+	close(activationLock)
+
 	return engine.StatusOK
 }

api/client.go

@@ -1,21 +1,21 @@
-package docker
+package api
 
 import (
-	"archive/tar"
 	"bufio"
 	"bytes"
 	"encoding/base64"
 	"encoding/json"
 	"errors"
 	"fmt"
-	"github.com/dotcloud/docker/api"
 	"github.com/dotcloud/docker/archive"
 	"github.com/dotcloud/docker/auth"
+	"github.com/dotcloud/docker/dockerversion"
 	"github.com/dotcloud/docker/engine"
+	"github.com/dotcloud/docker/nat"
 	flag "github.com/dotcloud/docker/pkg/mflag"
-	"github.com/dotcloud/docker/pkg/sysinfo"
 	"github.com/dotcloud/docker/pkg/term"
 	"github.com/dotcloud/docker/registry"
+	"github.com/dotcloud/docker/runconfig"
 	"github.com/dotcloud/docker/utils"
 	"io"
 	"io/ioutil"
@@ -29,7 +29,6 @@ import (
 	"reflect"
 	"regexp"
 	"runtime"
-	"sort"
 	"strconv"
 	"strings"
 	"syscall"
@@ -38,11 +37,6 @@ import (
 	"time"
 )
 
-var (
-	GITCOMMIT string
-	VERSION   string
-)
-
 var (
 	ErrConnectionRefused = errors.New("Can't connect to docker daemon. Is 'docker -d' running on this host?")
 )
@@ -80,7 +74,7 @@ func (cli *DockerCli) CmdHelp(args ...string) error {
 			return nil
 		}
 	}
-	help := fmt.Sprintf("Usage: docker [OPTIONS] COMMAND [arg...]\n -H=[unix://%s]: tcp://host:port to bind/connect to or unix://path/to/socket to use\n\nA self-sufficient runtime for linux containers.\n\nCommands:\n", api.DEFAULTUNIXSOCKET)
+	help := fmt.Sprintf("Usage: docker [OPTIONS] COMMAND [arg...]\n -H=[unix://%s]: tcp://host:port to bind/connect to or unix://path/to/socket to use\n\nA self-sufficient runtime for linux containers.\n\nCommands:\n", DEFAULTUNIXSOCKET)
 	for _, command := range [][]string{
 		{"attach", "Attach to a running container"},
 		{"build", "Build a container from a Dockerfile"},
@@ -139,35 +133,10 @@ func (cli *DockerCli) CmdInsert(args ...string) error {
 	return cli.stream("POST", "/images/"+cmd.Arg(0)+"/insert?"+v.Encode(), nil, cli.out, nil)
 }
 
-// mkBuildContext returns an archive of an empty context with the contents
-// of `dockerfile` at the path ./Dockerfile
-func MkBuildContext(dockerfile string, files [][2]string) (archive.Archive, error) {
-	buf := new(bytes.Buffer)
-	tw := tar.NewWriter(buf)
-	files = append(files, [2]string{"Dockerfile", dockerfile})
-	for _, file := range files {
-		name, content := file[0], file[1]
-		hdr := &tar.Header{
-			Name: name,
-			Size: int64(len(content)),
-		}
-		if err := tw.WriteHeader(hdr); err != nil {
-			return nil, err
-		}
-		if _, err := tw.Write([]byte(content)); err != nil {
-			return nil, err
-		}
-	}
-	if err := tw.Close(); err != nil {
-		return nil, err
-	}
-	return buf, nil
-}
-
 func (cli *DockerCli) CmdBuild(args ...string) error {
 	cmd := cli.Subcmd("build", "[OPTIONS] PATH | URL | -", "Build a new container image from the source code at PATH")
 	tag := cmd.String([]string{"t", "-tag"}, "", "Repository name (and optionally a tag) to be applied to the resulting image in case of success")
-	suppressOutput := cmd.Bool([]string{"q", "-quiet"}, false, "Suppress verbose build output")
+	suppressOutput := cmd.Bool([]string{"q", "-quiet"}, false, "Suppress the verbose output generated by the containers")
 	noCache := cmd.Bool([]string{"#no-cache", "-no-cache"}, false, "Do not use cache when building the image")
 	rm := cmd.Bool([]string{"#rm", "-rm"}, false, "Remove intermediate containers after a successful build")
 	if err := cmd.Parse(args); err != nil {
@@ -191,7 +160,7 @@ func (cli *DockerCli) CmdBuild(args ...string) error {
 		if err != nil {
 			return err
 		}
-		context, err = MkBuildContext(string(dockerfile), nil)
+		context, err = archive.Generate("Dockerfile", string(dockerfile))
 	} else if utils.IsURL(cmd.Arg(0)) || utils.IsGIT(cmd.Arg(0)) {
 		isRemote = true
 	} else {
@@ -209,7 +178,7 @@ func (cli *DockerCli) CmdBuild(args ...string) error {
 	// FIXME: ProgressReader shouldn't be this annoying to use
 	if context != nil {
 		sf := utils.NewStreamFormatter(false)
-		body = utils.ProgressReader(ioutil.NopCloser(context), 0, cli.err, sf, true, "", "Uploading context")
+		body = utils.ProgressReader(context, 0, cli.err, sf, true, "", "Uploading context")
 	}
 	// Upload the build context
 	v := &url.Values{}
@@ -266,11 +235,7 @@ func (cli *DockerCli) CmdLogin(args ...string) error {
 	}
 	serverAddress := auth.IndexServerAddress()
 	if len(cmd.Args()) > 0 {
-		serverAddress, err = registry.ExpandAndVerifyRegistryUrl(cmd.Arg(0))
-		if err != nil {
-			return err
-		}
-		fmt.Fprintf(cli.out, "Login against server at %s\n", serverAddress)
+		serverAddress = cmd.Arg(0)
 	}
 
 	promptDefault := func(prompt string, configDefault string) {
@@ -392,12 +357,12 @@ func (cli *DockerCli) CmdVersion(args ...string) error {
 		cmd.Usage()
 		return nil
 	}
-	if VERSION != "" {
-		fmt.Fprintf(cli.out, "Client version: %s\n", VERSION)
+	if dockerversion.VERSION != "" {
+		fmt.Fprintf(cli.out, "Client version: %s\n", dockerversion.VERSION)
 	}
 	fmt.Fprintf(cli.out, "Go version (client): %s\n", runtime.Version())
-	if GITCOMMIT != "" {
-		fmt.Fprintf(cli.out, "Git commit (client): %s\n", GITCOMMIT)
+	if dockerversion.GITCOMMIT != "" {
+		fmt.Fprintf(cli.out, "Git commit (client): %s\n", dockerversion.GITCOMMIT)
 	}
 
 	body, _, err := readBody(cli.call("GET", "/version", nil, false))
@@ -422,7 +387,7 @@ func (cli *DockerCli) CmdVersion(args ...string) error {
 	release := utils.GetReleaseVersion()
 	if release != "" {
 		fmt.Fprintf(cli.out, "Last stable version: %s", release)
-		if (VERSION != "" || remoteVersion.Exists("Version")) && (strings.Trim(VERSION, "-dev") != release || strings.Trim(remoteVersion.Get("Version"), "-dev") != release) {
+		if (dockerversion.VERSION != "" || remoteVersion.Exists("Version")) && (strings.Trim(dockerversion.VERSION, "-dev") != release || strings.Trim(remoteVersion.Get("Version"), "-dev") != release) {
 			fmt.Fprintf(cli.out, ", please update docker")
 		}
 		fmt.Fprintf(cli.out, "\n")
@@ -803,7 +768,7 @@ func (cli *DockerCli) CmdPort(args ...string) error {
 		return err
 	}
 
-	if frontends, exists := out.NetworkSettings.Ports[Port(port+"/"+proto)]; exists && frontends != nil {
+	if frontends, exists := out.NetworkSettings.Ports[nat.Port(port+"/"+proto)]; exists && frontends != nil {
 		for _, frontend := range frontends {
 			fmt.Fprintf(cli.out, "%s:%s\n", frontend.HostIp, frontend.HostPort)
 		}
@@ -1313,19 +1278,6 @@ func (cli *DockerCli) printTreeNode(noTrunc bool, image *engine.Env, prefix stri
 	}
 }
 
-func displayablePorts(ports *engine.Table) string {
-	result := []string{}
-	for _, port := range ports.Data {
-		if port.Get("IP") == "" {
-			result = append(result, fmt.Sprintf("%d/%s", port.GetInt("PublicPort"), port.Get("Type")))
-		} else {
-			result = append(result, fmt.Sprintf("%s:%d->%d/%s", port.Get("IP"), port.GetInt("PublicPort"), port.GetInt("PrivatePort"), port.Get("Type")))
-		}
-	}
-	sort.Strings(result)
-	return strings.Join(result, ", ")
-}
-
 func (cli *DockerCli) CmdPs(args ...string) error {
 	cmd := cli.Subcmd("ps", "[OPTIONS]", "List containers")
 	quiet := cmd.Bool([]string{"q", "-quiet"}, false, "Only display numeric IDs")
@@ -1455,11 +1407,11 @@ func (cli *DockerCli) CmdCommit(args ...string) error {
 	v.Set("comment", *flComment)
 	v.Set("author", *flAuthor)
 	var (
-		config *Config
+		config *runconfig.Config
 		env    engine.Env
 	)
 	if *flConfig != "" {
-		config = &Config{}
+		config = &runconfig.Config{}
 		if err := json.Unmarshal([]byte(*flConfig), config); err != nil {
 			return err
 		}
@@ -1620,7 +1572,7 @@ func (cli *DockerCli) CmdAttach(args ...string) error {
 		return err
 	}
 
-	if !container.State.IsRunning() {
+	if !container.State.Running {
 		return fmt.Errorf("Impossible to attach to a stopped container, start it first")
 	}
 
@@ -1749,210 +1701,9 @@ func (cli *DockerCli) CmdTag(args ...string) error {
 	return nil
 }
 
-//FIXME Only used in tests
-func ParseRun(args []string, sysInfo *sysinfo.SysInfo) (*Config, *HostConfig, *flag.FlagSet, error) {
-	cmd := flag.NewFlagSet("run", flag.ContinueOnError)
-	cmd.SetOutput(ioutil.Discard)
-	cmd.Usage = nil
-	return parseRun(cmd, args, sysInfo)
-}
-
-func parseRun(cmd *flag.FlagSet, args []string, sysInfo *sysinfo.SysInfo) (*Config, *HostConfig, *flag.FlagSet, error) {
-	var (
-		// FIXME: use utils.ListOpts for attach and volumes?
-		flAttach  = NewListOpts(ValidateAttach)
-		flVolumes = NewListOpts(ValidatePath)
-		flLinks   = NewListOpts(ValidateLink)
-		flEnv     = NewListOpts(ValidateEnv)
-
-		flPublish     ListOpts
-		flExpose      ListOpts
-		flDns         ListOpts
-		flVolumesFrom ListOpts
-		flLxcOpts     ListOpts
-
-		flAutoRemove      = cmd.Bool([]string{"#rm", "-rm"}, false, "Automatically remove the container when it exits (incompatible with -d)")
-		flDetach          = cmd.Bool([]string{"d", "-detach"}, false, "Detached mode: Run container in the background, print new container id")
-		flNetwork         = cmd.Bool([]string{"n", "-networking"}, true, "Enable networking for this container")
-		flPrivileged      = cmd.Bool([]string{"#privileged", "-privileged"}, false, "Give extended privileges to this container")
-		flPublishAll      = cmd.Bool([]string{"P", "-publish-all"}, false, "Publish all exposed ports to the host interfaces")
-		flStdin           = cmd.Bool([]string{"i", "-interactive"}, false, "Keep stdin open even if not attached")
-		flTty             = cmd.Bool([]string{"t", "-tty"}, false, "Allocate a pseudo-tty")
-		flContainerIDFile = cmd.String([]string{"#cidfile", "-cidfile"}, "", "Write the container ID to the file")
-		flEntrypoint      = cmd.String([]string{"#entrypoint", "-entrypoint"}, "", "Overwrite the default entrypoint of the image")
-		flHostname        = cmd.String([]string{"h", "-hostname"}, "", "Container host name")
-		flMemoryString    = cmd.String([]string{"m", "-memory"}, "", "Memory limit (format: <number><optional unit>, where unit = b, k, m or g)")
-		flUser            = cmd.String([]string{"u", "-user"}, "", "Username or UID")
-		flWorkingDir      = cmd.String([]string{"w", "-workdir"}, "", "Working directory inside the container")
-		flCpuShares       = cmd.Int64([]string{"c", "-cpu-shares"}, 0, "CPU shares (relative weight)")
-
-		// For documentation purpose
-		_ = cmd.Bool([]string{"#sig-proxy", "-sig-proxy"}, true, "Proxify all received signal to the process (even in non-tty mode)")
-		_ = cmd.String([]string{"#name", "-name"}, "", "Assign a name to the container")
-	)
-
-	cmd.Var(&flAttach, []string{"a", "-attach"}, "Attach to stdin, stdout or stderr.")
-	cmd.Var(&flVolumes, []string{"v", "-volume"}, "Bind mount a volume (e.g. from the host: -v /host:/container, from docker: -v /container)")
-	cmd.Var(&flLinks, []string{"#link", "-link"}, "Add link to another container (name:alias)")
-	cmd.Var(&flEnv, []string{"e", "-env"}, "Set environment variables")
-
-	cmd.Var(&flPublish, []string{"p", "-publish"}, fmt.Sprintf("Publish a container's port to the host (format: %s) (use 'docker port' to see the actual mapping)", PortSpecTemplateFormat))
-	cmd.Var(&flExpose, []string{"#expose", "-expose"}, "Expose a port from the container without publishing it to your host")
-	cmd.Var(&flDns, []string{"#dns", "-dns"}, "Set custom dns servers")
-	cmd.Var(&flVolumesFrom, []string{"#volumes-from", "-volumes-from"}, "Mount volumes from the specified container(s)")
-	cmd.Var(&flLxcOpts, []string{"#lxc-conf", "-lxc-conf"}, "Add custom lxc options -lxc-conf=\"lxc.cgroup.cpuset.cpus = 0,1\"")
-
-	if err := cmd.Parse(args); err != nil {
-		return nil, nil, cmd, err
-	}
-
-	// Check if the kernel supports memory limit cgroup.
-	if sysInfo != nil && *flMemoryString != "" && !sysInfo.MemoryLimit {
-		*flMemoryString = ""
-	}
-
-	// Validate input params
-	if *flDetach && flAttach.Len() > 0 {
-		return nil, nil, cmd, ErrConflictAttachDetach
-	}
-	if *flWorkingDir != "" && !path.IsAbs(*flWorkingDir) {
-		return nil, nil, cmd, ErrInvalidWorikingDirectory
-	}
-	if *flDetach && *flAutoRemove {
-		return nil, nil, cmd, ErrConflictDetachAutoRemove
-	}
-
-	// If neither -d or -a are set, attach to everything by default
-	if flAttach.Len() == 0 && !*flDetach {
-		if !*flDetach {
-			flAttach.Set("stdout")
-			flAttach.Set("stderr")
-			if *flStdin {
-				flAttach.Set("stdin")
-			}
-		}
-	}
-
-	var flMemory int64
-	if *flMemoryString != "" {
-		parsedMemory, err := utils.RAMInBytes(*flMemoryString)
-		if err != nil {
-			return nil, nil, cmd, err
-		}
-		flMemory = parsedMemory
-	}
-
-	var binds []string
-	// add any bind targets to the list of container volumes
-	for bind := range flVolumes.GetMap() {
-		if arr := strings.Split(bind, ":"); len(arr) > 1 {
-			if arr[0] == "/" {
-				return nil, nil, cmd, fmt.Errorf("Invalid bind mount: source can't be '/'")
-			}
-			dstDir := arr[1]
-			flVolumes.Set(dstDir)
-			binds = append(binds, bind)
-			flVolumes.Delete(bind)
-		} else if bind == "/" {
-			return nil, nil, cmd, fmt.Errorf("Invalid volume: path can't be '/'")
-		}
-	}
-
-	var (
-		parsedArgs = cmd.Args()
-		runCmd     []string
-		entrypoint []string
-		image      string
-	)
-	if len(parsedArgs) >= 1 {
-		image = cmd.Arg(0)
-	}
-	if len(parsedArgs) > 1 {
-		runCmd = parsedArgs[1:]
-	}
-	if *flEntrypoint != "" {
-		entrypoint = []string{*flEntrypoint}
-	}
-
-	lxcConf, err := parseLxcConfOpts(flLxcOpts)
-	if err != nil {
-		return nil, nil, cmd, err
-	}
-
-	var (
-		domainname string
-		hostname   = *flHostname
-		parts      = strings.SplitN(hostname, ".", 2)
-	)
-	if len(parts) > 1 {
-		hostname = parts[0]
-		domainname = parts[1]
-	}
-
-	ports, portBindings, err := parsePortSpecs(flPublish.GetAll())
-	if err != nil {
-		return nil, nil, cmd, err
-	}
-
-	// Merge in exposed ports to the map of published ports
-	for _, e := range flExpose.GetAll() {
-		if strings.Contains(e, ":") {
-			return nil, nil, cmd, fmt.Errorf("Invalid port format for --expose: %s", e)
-		}
-		p := NewPort(splitProtoPort(e))
-		if _, exists := ports[p]; !exists {
-			ports[p] = struct{}{}
-		}
-	}
-
-	config := &Config{
-		Hostname:        hostname,
-		Domainname:      domainname,
-		PortSpecs:       nil, // Deprecated
-		ExposedPorts:    ports,
-		User:            *flUser,
-		Tty:             *flTty,
-		NetworkDisabled: !*flNetwork,
-		OpenStdin:       *flStdin,
-		Memory:          flMemory,
-		CpuShares:       *flCpuShares,
-		AttachStdin:     flAttach.Get("stdin"),
-		AttachStdout:    flAttach.Get("stdout"),
-		AttachStderr:    flAttach.Get("stderr"),
-		Env:             flEnv.GetAll(),
-		Cmd:             runCmd,
-		Dns:             flDns.GetAll(),
-		Image:           image,
-		Volumes:         flVolumes.GetMap(),
-		VolumesFrom:     strings.Join(flVolumesFrom.GetAll(), ","),
-		Entrypoint:      entrypoint,
-		WorkingDir:      *flWorkingDir,
-	}
-
-	hostConfig := &HostConfig{
-		Binds:           binds,
-		ContainerIDFile: *flContainerIDFile,
-		LxcConf:         lxcConf,
-		Privileged:      *flPrivileged,
-		PortBindings:    portBindings,
-		Links:           flLinks.GetAll(),
-		PublishAllPorts: *flPublishAll,
-	}
-
-	if sysInfo != nil && flMemory > 0 && !sysInfo.SwapLimit {
-		//fmt.Fprintf(stdout, "WARNING: Your kernel does not support swap limit capabilities. Limitation discarded.\n")
-		config.MemorySwap = -1
-	}
-
-	// When allocating stdin in attached mode, close stdin at client disconnect
-	if config.OpenStdin && config.AttachStdin {
-		config.StdinOnce = true
-	}
-	return config, hostConfig, cmd, nil
-}
-
 func (cli *DockerCli) CmdRun(args ...string) error {
-	config, hostConfig, cmd, err := parseRun(cli.Subcmd("run", "[OPTIONS] IMAGE [COMMAND] [ARG...]", "Run a command in a new container"), args, nil)
+	// FIXME: just use runconfig.Parse already
+	config, hostConfig, cmd, err := runconfig.ParseSubcommand(cli.Subcmd("run", "[OPTIONS] IMAGE [COMMAND] [ARG...]", "Run a command in a new container"), args, nil)
 	if err != nil {
 		return err
 	}
@@ -1995,12 +1746,7 @@ func (cli *DockerCli) CmdRun(args ...string) error {
 	stream, statusCode, err := cli.call("POST", "/containers/create?"+containerValues.Encode(), config, false)
 	//if image not found try to pull it
 	if statusCode == 404 {
-		_, tag := utils.ParseRepositoryTag(config.Image)
-		if tag == "" {
-			tag = DEFAULTTAG
-		}
-
-		fmt.Fprintf(cli.err, "Unable to find image '%s' (tag: %s) locally\n", config.Image, tag)
+		fmt.Fprintf(cli.err, "Unable to find image '%s' locally\n", config.Image)
 
 		v := url.Values{}
 		repos, tag := utils.ParseRepositoryTag(config.Image)
@@ -2215,6 +1961,9 @@ func (cli *DockerCli) CmdCp(args ...string) error {
 	if stream != nil {
 		defer stream.Close()
 	}
+	if statusCode == 404 {
+		return fmt.Errorf("No such container: %v", info[0])
+	}
 	if err != nil {
 		return err
 	}
@@ -2283,7 +2032,7 @@ func (cli *DockerCli) call(method, path string, data interface{}, passAuthInfo b
 	re := regexp.MustCompile("/+")
 	path = re.ReplaceAllString(path, "/")
 
-	req, err := http.NewRequest(method, fmt.Sprintf("/v%g%s", api.APIVERSION, path), params)
+	req, err := http.NewRequest(method, fmt.Sprintf("/v%g%s", APIVERSION, path), params)
 	if err != nil {
 		return nil, -1, err
 	}
@@ -2307,7 +2056,7 @@ func (cli *DockerCli) call(method, path string, data interface{}, passAuthInfo b
 			}
 		}
 	}
-	req.Header.Set("User-Agent", "Docker-Client/"+VERSION)
+	req.Header.Set("User-Agent", "Docker-Client/"+dockerversion.VERSION)
 	req.Host = cli.addr
 	if data != nil {
 		req.Header.Set("Content-Type", "application/json")
@@ -2337,7 +2086,7 @@ func (cli *DockerCli) call(method, path string, data interface{}, passAuthInfo b
 			return nil, -1, err
 		}
 		if len(body) == 0 {
-			return nil, resp.StatusCode, fmt.Errorf("Error :%s", http.StatusText(resp.StatusCode))
+			return nil, resp.StatusCode, fmt.Errorf("Error: request returned %s for api route and version %s, check if the server supports the requested api version", http.StatusText(resp.StatusCode), req.URL)
 		}
 		return nil, resp.StatusCode, fmt.Errorf("Error: %s", bytes.TrimSpace(body))
 	}
@@ -2360,11 +2109,11 @@ func (cli *DockerCli) stream(method, path string, in io.Reader, out io.Writer, h
 	re := regexp.MustCompile("/+")
 	path = re.ReplaceAllString(path, "/")
 
-	req, err := http.NewRequest(method, fmt.Sprintf("/v%g%s", api.APIVERSION, path), in)
+	req, err := http.NewRequest(method, fmt.Sprintf("/v%g%s", APIVERSION, path), in)
 	if err != nil {
 		return err
 	}
-	req.Header.Set("User-Agent", "Docker-Client/"+VERSION)
+	req.Header.Set("User-Agent", "Docker-Client/"+dockerversion.VERSION)
 	req.Host = cli.addr
 	if method == "POST" {
 		req.Header.Set("Content-Type", "plain/text")
@@ -2405,7 +2154,7 @@ func (cli *DockerCli) stream(method, path string, in io.Reader, out io.Writer, h
 		return fmt.Errorf("Error: %s", bytes.TrimSpace(body))
 	}
 
-	if api.MatchesContentType(resp.Header.Get("Content-Type"), "application/json") {
+	if MatchesContentType(resp.Header.Get("Content-Type"), "application/json") {
 		return utils.DisplayJSONMessagesStream(resp.Body, out, cli.terminalFd, cli.isTerminal)
 	}
 	if _, err := io.Copy(out, resp.Body); err != nil {
@@ -2424,11 +2173,11 @@ func (cli *DockerCli) hijack(method, path string, setRawTerminal bool, in io.Rea
 	re := regexp.MustCompile("/+")
 	path = re.ReplaceAllString(path, "/")
 
-	req, err := http.NewRequest(method, fmt.Sprintf("/v%g%s", api.APIVERSION, path), nil)
+	req, err := http.NewRequest(method, fmt.Sprintf("/v%g%s", APIVERSION, path), nil)
 	if err != nil {
 		return err
 	}
-	req.Header.Set("User-Agent", "Docker-Client/"+VERSION)
+	req.Header.Set("User-Agent", "Docker-Client/"+dockerversion.VERSION)
 	req.Header.Set("Content-Type", "plain/text")
 	req.Host = cli.addr
 
@@ -2607,7 +2356,7 @@ func getExitCode(cli *DockerCli, containerId string) (bool, int, error) {
 	if err := json.Unmarshal(body, c); err != nil {
 		return false, -1, err
 	}
-	return c.State.IsRunning(), c.State.GetExitCode(), nil
+	return c.State.Running, c.State.ExitCode, nil
 }
 
 func readBody(stream io.ReadCloser, statusCode int, err error) ([]byte, int, error) {

api/container.go

@@ -0,0 +1,18 @@
+package api
+
+import (
+	"github.com/dotcloud/docker/nat"
+	"github.com/dotcloud/docker/runconfig"
+)
+
+type Container struct {
+	Config     runconfig.Config
+	HostConfig runconfig.HostConfig
+	State      struct {
+		Running  bool
+		ExitCode int
+	}
+	NetworkSettings struct {
+		Ports nat.PortMap
+	}
+}

archive/archive.go

@@ -1,10 +1,11 @@
 package archive
 
 import (
-	"archive/tar"
 	"bytes"
+	"code.google.com/p/go/src/pkg/archive/tar"
 	"compress/bzip2"
 	"compress/gzip"
+	"errors"
 	"fmt"
 	"github.com/dotcloud/docker/utils"
 	"io"
@@ -17,14 +18,19 @@ import (
 	"syscall"
 )
 
-type Archive io.Reader
+type (
+	Archive       io.ReadCloser
+	ArchiveReader io.Reader
+	Compression   int
+	TarOptions    struct {
+		Includes    []string
+		Compression Compression
+	}
+)
 
-type Compression int
-
-type TarOptions struct {
-	Includes    []string
-	Compression Compression
-}
+var (
+	ErrNotImplemented = errors.New("Function not implemented")
+)
 
 const (
 	Uncompressed Compression = iota
@@ -60,13 +66,13 @@ func DetectCompression(source []byte) Compression {
 	return Uncompressed
 }
 
-func xzDecompress(archive io.Reader) (io.Reader, error) {
+func xzDecompress(archive io.Reader) (io.ReadCloser, error) {
 	args := []string{"xz", "-d", "-c", "-q"}
 
 	return CmdStream(exec.Command(args[0], args[1:]...), archive)
 }
 
-func DecompressStream(archive io.Reader) (io.Reader, error) {
+func DecompressStream(archive io.Reader) (io.ReadCloser, error) {
 	buf := make([]byte, 10)
 	totalN := 0
 	for totalN < 10 {
@@ -85,11 +91,11 @@ func DecompressStream(archive io.Reader) (io.Reader, error) {
 
 	switch compression {
 	case Uncompressed:
-		return wrap, nil
+		return ioutil.NopCloser(wrap), nil
 	case Gzip:
 		return gzip.NewReader(wrap)
 	case Bzip2:
-		return bzip2.NewReader(wrap), nil
+		return ioutil.NopCloser(bzip2.NewReader(wrap)), nil
 	case Xz:
 		return xzDecompress(wrap)
 	default:
@@ -101,7 +107,7 @@ func CompressStream(dest io.WriteCloser, compression Compression) (io.WriteClose
 
 	switch compression {
 	case Uncompressed:
-		return dest, nil
+		return utils.NopWriteCloser(dest), nil
 	case Gzip:
 		return gzip.NewWriter(dest), nil
 	case Bzip2, Xz:
@@ -180,20 +186,25 @@ func addTarFile(path, name string, tw *tar.Writer) error {
 	return nil
 }
 
-func createTarFile(path, extractDir string, hdr *tar.Header, reader *tar.Reader) error {
+func createTarFile(path, extractDir string, hdr *tar.Header, reader io.Reader) error {
+	// hdr.Mode is in linux format, which we can use for sycalls,
+	// but for os.Foo() calls we need the mode converted to os.FileMode,
+	// so use hdrInfo.Mode() (they differ for e.g. setuid bits)
+	hdrInfo := hdr.FileInfo()
+
 	switch hdr.Typeflag {
 	case tar.TypeDir:
 		// Create directory unless it exists as a directory already.
 		// In that case we just want to merge the two
 		if fi, err := os.Lstat(path); !(err == nil && fi.IsDir()) {
-			if err := os.Mkdir(path, os.FileMode(hdr.Mode)); err != nil {
+			if err := os.Mkdir(path, hdrInfo.Mode()); err != nil {
 				return err
 			}
 		}
 
 	case tar.TypeReg, tar.TypeRegA:
 		// Source is regular file
-		file, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY, os.FileMode(hdr.Mode))
+		file, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY, hdrInfo.Mode())
 		if err != nil {
 			return err
 		}
@@ -236,14 +247,14 @@ func createTarFile(path, extractDir string, hdr *tar.Header, reader *tar.Reader)
 		return fmt.Errorf("Unhandled tar header type %d\n", hdr.Typeflag)
 	}
 
-	if err := syscall.Lchown(path, hdr.Uid, hdr.Gid); err != nil {
+	if err := os.Lchown(path, hdr.Uid, hdr.Gid); err != nil {
 		return err
 	}
 
 	// There is no LChmod, so ignore mode for symlink. Also, this
 	// must happen after chown, as that can modify the file mode
 	if hdr.Typeflag != tar.TypeSymlink {
-		if err := syscall.Chmod(path, uint32(hdr.Mode&07777)); err != nil {
+		if err := os.Chmod(path, hdrInfo.Mode()); err != nil {
 			return err
 		}
 	}
@@ -251,7 +262,7 @@ func createTarFile(path, extractDir string, hdr *tar.Header, reader *tar.Reader)
 	ts := []syscall.Timespec{timeToTimespec(hdr.AccessTime), timeToTimespec(hdr.ModTime)}
 	// syscall.UtimesNano doesn't support a NOFOLLOW flag atm, and
 	if hdr.Typeflag != tar.TypeSymlink {
-		if err := syscall.UtimesNano(path, ts); err != nil {
+		if err := UtimesNano(path, ts); err != nil {
 			return err
 		}
 	} else {
@@ -264,7 +275,7 @@ func createTarFile(path, extractDir string, hdr *tar.Header, reader *tar.Reader)
 
 // Tar creates an archive from the directory at `path`, and returns it as a
 // stream of bytes.
-func Tar(path string, compression Compression) (io.Reader, error) {
+func Tar(path string, compression Compression) (io.ReadCloser, error) {
 	return TarFilter(path, &TarOptions{Compression: compression})
 }
 
@@ -286,7 +297,7 @@ func escapeName(name string) string {
 
 // Tar creates an archive from the directory at `path`, only including files whose relative
 // paths are included in `filter`. If `filter` is nil, then all files are included.
-func TarFilter(srcPath string, options *TarOptions) (io.Reader, error) {
+func TarFilter(srcPath string, options *TarOptions) (io.ReadCloser, error) {
 	pipeReader, pipeWriter := io.Pipe()
 
 	compressWriter, err := CompressStream(pipeWriter, options.Compression)
@@ -332,6 +343,9 @@ func TarFilter(srcPath string, options *TarOptions) (io.Reader, error) {
 		if err := compressWriter.Close(); err != nil {
 			utils.Debugf("Can't close compress writer: %s\n", err)
 		}
+		if err := pipeWriter.Close(); err != nil {
+			utils.Debugf("Can't close pipe writer: %s\n", err)
+		}
 	}()
 
 	return pipeReader, nil
@@ -347,12 +361,13 @@ func Untar(archive io.Reader, dest string, options *TarOptions) error {
 		return fmt.Errorf("Empty archive")
 	}
 
-	archive, err := DecompressStream(archive)
+	decompressedArchive, err := DecompressStream(archive)
 	if err != nil {
 		return err
 	}
+	defer decompressedArchive.Close()
 
-	tr := tar.NewReader(archive)
+	tr := tar.NewReader(decompressedArchive)
 
 	var dirs []*tar.Header
 
@@ -427,15 +442,19 @@ func TarUntar(src string, dst string) error {
 	if err != nil {
 		return err
 	}
+	defer archive.Close()
 	return Untar(archive, dst, nil)
 }
 
 // UntarPath is a convenience function which looks for an archive
 // at filesystem path `src`, and unpacks it at `dst`.
 func UntarPath(src, dst string) error {
-	if archive, err := os.Open(src); err != nil {
+	archive, err := os.Open(src)
+	if err != nil {
 		return err
-	} else if err := Untar(archive, dst, nil); err != nil {
+	}
+	defer archive.Close()
+	if err := Untar(archive, dst, nil); err != nil {
 		return err
 	}
 	return nil
@@ -523,7 +542,7 @@ func CopyFileWithTar(src, dst string) (err error) {
 // CmdStream executes a command, and returns its stdout as a stream.
 // If the command fails to run or doesn't complete successfully, an error
 // will be returned, including anything written on stderr.
-func CmdStream(cmd *exec.Cmd, input io.Reader) (io.Reader, error) {
+func CmdStream(cmd *exec.Cmd, input io.Reader) (io.ReadCloser, error) {
 	if input != nil {
 		stdin, err := cmd.StdinPipe()
 		if err != nil {

archive/archive_test.go

@@ -1,8 +1,8 @@
 package archive
 
 import (
-	"archive/tar"
 	"bytes"
+	"code.google.com/p/go/src/pkg/archive/tar"
 	"fmt"
 	"io"
 	"io/ioutil"
@@ -67,12 +67,13 @@ func tarUntar(t *testing.T, origin string, compression Compression) error {
 	if err != nil {
 		t.Fatal(err)
 	}
+	defer archive.Close()
 
 	buf := make([]byte, 10)
 	if _, err := archive.Read(buf); err != nil {
 		return err
 	}
-	archive = io.MultiReader(bytes.NewReader(buf), archive)
+	wrap := io.MultiReader(bytes.NewReader(buf), archive)
 
 	detectedCompression := DetectCompression(buf)
 	if detectedCompression.Extension() != compression.Extension() {
@@ -84,7 +85,7 @@ func tarUntar(t *testing.T, origin string, compression Compression) error {
 		return err
 	}
 	defer os.RemoveAll(tmp)
-	if err := Untar(archive, tmp, nil); err != nil {
+	if err := Untar(wrap, tmp, nil); err != nil {
 		return err
 	}
 	if _, err := os.Stat(tmp); err != nil {

archive/changes.go

@@ -1,7 +1,7 @@
 package archive
 
 import (
-	"archive/tar"
+	"code.google.com/p/go/src/pkg/archive/tar"
 	"fmt"
 	"github.com/dotcloud/docker/utils"
 	"io"

archive/diff.go

@@ -1,8 +1,10 @@
 package archive
 
 import (
-	"archive/tar"
+	"code.google.com/p/go/src/pkg/archive/tar"
+	"fmt"
 	"io"
+	"io/ioutil"
 	"os"
 	"path/filepath"
 	"strings"
@@ -28,7 +30,7 @@ func timeToTimespec(time time.Time) (ts syscall.Timespec) {
 
 // ApplyLayer parses a diff in the standard layer format from `layer`, and
 // applies it to the directory `dest`.
-func ApplyLayer(dest string, layer Archive) error {
+func ApplyLayer(dest string, layer ArchiveReader) error {
 	// We need to be able to set any perms
 	oldmask := syscall.Umask(0)
 	defer syscall.Umask(oldmask)
@@ -42,6 +44,9 @@ func ApplyLayer(dest string, layer Archive) error {
 
 	var dirs []*tar.Header
 
+	aufsTempdir := ""
+	aufsHardlinks := make(map[string]*tar.Header)
+
 	// Iterate through the files in the archive.
 	for {
 		hdr, err := tr.Next()
@@ -72,6 +77,22 @@ func ApplyLayer(dest string, layer Archive) error {
 
 		// Skip AUFS metadata dirs
 		if strings.HasPrefix(hdr.Name, ".wh..wh.") {
+			// Regular files inside /.wh..wh.plnk can be used as hardlink targets
+			// We don't want this directory, but we need the files in them so that
+			// such hardlinks can be resolved.
+			if strings.HasPrefix(hdr.Name, ".wh..wh.plnk") && hdr.Typeflag == tar.TypeReg {
+				basename := filepath.Base(hdr.Name)
+				aufsHardlinks[basename] = hdr
+				if aufsTempdir == "" {
+					if aufsTempdir, err = ioutil.TempDir("", "dockerplnk"); err != nil {
+						return err
+					}
+					defer os.RemoveAll(aufsTempdir)
+				}
+				if err := createTarFile(filepath.Join(aufsTempdir, basename), dest, hdr, tr); err != nil {
+					return err
+				}
+			}
 			continue
 		}
 
@@ -96,7 +117,26 @@ func ApplyLayer(dest string, layer Archive) error {
 				}
 			}
 
-			if err := createTarFile(path, dest, hdr, tr); err != nil {
+			srcData := io.Reader(tr)
+			srcHdr := hdr
+
+			// Hard links into /.wh..wh.plnk don't work, as we don't extract that directory, so
+			// we manually retarget these into the temporary files we extracted them into
+			if hdr.Typeflag == tar.TypeLink && strings.HasPrefix(filepath.Clean(hdr.Linkname), ".wh..wh.plnk") {
+				linkBasename := filepath.Base(hdr.Linkname)
+				srcHdr = aufsHardlinks[linkBasename]
+				if srcHdr == nil {
+					return fmt.Errorf("Invalid aufs hardlink")
+				}
+				tmpFile, err := os.Open(filepath.Join(aufsTempdir, linkBasename))
+				if err != nil {
+					return err
+				}
+				defer tmpFile.Close()
+				srcData = tmpFile
+			}
+
+			if err := createTarFile(path, dest, srcHdr, srcData); err != nil {
 				return err
 			}
 

archive/stat_linux.go

@@ -30,3 +30,10 @@ func LUtimesNano(path string, ts []syscall.Timespec) error {
 
 	return nil
 }
+
+func UtimesNano(path string, ts []syscall.Timespec) error {
+	if err := syscall.UtimesNano(path, ts); err != nil {
+		return err
+	}
+	return nil
+}

archive/stat_unsupported.go

@@ -1,4 +1,4 @@
-// +build !linux !amd64
+// +build !linux
 
 package archive
 
@@ -13,5 +13,9 @@ func getLastModification(stat *syscall.Stat_t) syscall.Timespec {
 }
 
 func LUtimesNano(path string, ts []syscall.Timespec) error {
-	return nil
+	return ErrNotImplemented
+}
+
+func UtimesNano(path string, ts []syscall.Timespec) error {
+	return ErrNotImplemented
 }

archive/wrap.go

@@ -0,0 +1,59 @@
+package archive
+
+import (
+	"bytes"
+	"code.google.com/p/go/src/pkg/archive/tar"
+	"io/ioutil"
+)
+
+// Generate generates a new archive from the content provided
+// as input.
+//
+// `files` is a sequence of path/content pairs. A new file is
+// added to the archive for each pair.
+// If the last pair is incomplete, the file is created with an
+// empty content. For example:
+//
+// Generate("foo.txt", "hello world", "emptyfile")
+//
+// The above call will return an archive with 2 files:
+//  * ./foo.txt with content "hello world"
+//  * ./empty with empty content
+//
+// FIXME: stream content instead of buffering
+// FIXME: specify permissions and other archive metadata
+func Generate(input ...string) (Archive, error) {
+	files := parseStringPairs(input...)
+	buf := new(bytes.Buffer)
+	tw := tar.NewWriter(buf)
+	for _, file := range files {
+		name, content := file[0], file[1]
+		hdr := &tar.Header{
+			Name: name,
+			Size: int64(len(content)),
+		}
+		if err := tw.WriteHeader(hdr); err != nil {
+			return nil, err
+		}
+		if _, err := tw.Write([]byte(content)); err != nil {
+			return nil, err
+		}
+	}
+	if err := tw.Close(); err != nil {
+		return nil, err
+	}
+	return ioutil.NopCloser(buf), nil
+}
+
+func parseStringPairs(input ...string) (output [][2]string) {
+	output = make([][2]string, 0, len(input)/2+1)
+	for i := 0; i < len(input); i += 2 {
+		var pair [2]string
+		pair[0] = input[i]
+		if i+1 < len(input) {
+			pair[1] = input[i+1]
+		}
+		output = append(output, pair)
+	}
+	return
+}

auth/auth.go

@@ -151,12 +151,15 @@ func SaveConfig(configFile *ConfigFile) error {
 
 // try to register/login to the registry server
 func Login(authConfig *AuthConfig, factory *utils.HTTPRequestFactory) (string, error) {
-	client := &http.Client{}
-	reqStatusCode := 0
-	var status string
-	var reqBody []byte
+	var (
+		status        string
+		reqBody       []byte
+		err           error
+		client        = &http.Client{}
+		reqStatusCode = 0
+		serverAddress = authConfig.ServerAddress
+	)
 
-	serverAddress := authConfig.ServerAddress
 	if serverAddress == "" {
 		serverAddress = IndexServerAddress()
 	}

buildfile.go

@@ -9,6 +9,7 @@ import (
 	"github.com/dotcloud/docker/archive"
 	"github.com/dotcloud/docker/auth"
 	"github.com/dotcloud/docker/registry"
+	"github.com/dotcloud/docker/runconfig"
 	"github.com/dotcloud/docker/utils"
 	"io"
 	"io/ioutil"
@@ -38,7 +39,7 @@ type buildFile struct {
 
 	image      string
 	maintainer string
-	config     *Config
+	config     *runconfig.Config
 
 	contextPath string
 	context     *utils.TarSum
@@ -64,8 +65,11 @@ type buildFile struct {
 func (b *buildFile) clearTmp(containers map[string]struct{}) {
 	for c := range containers {
 		tmp := b.runtime.Get(c)
-		b.runtime.Destroy(tmp)
-		fmt.Fprintf(b.outStream, "Removing intermediate container %s\n", utils.TruncateID(c))
+		if err := b.runtime.Destroy(tmp); err != nil {
+			fmt.Fprintf(b.outStream, "Error removing intermediate container %s: %s\n", utils.TruncateID(c), err.Error())
+		} else {
+			fmt.Fprintf(b.outStream, "Removing intermediate container %s\n", utils.TruncateID(c))
+		}
 	}
 }
 
@@ -101,7 +105,7 @@ func (b *buildFile) CmdFrom(name string) error {
 		}
 	}
 	b.image = image.ID
-	b.config = &Config{}
+	b.config = &runconfig.Config{}
 	if image.Config != nil {
 		b.config = image.Config
 	}
@@ -158,14 +162,14 @@ func (b *buildFile) CmdRun(args string) error {
 	if b.image == "" {
 		return fmt.Errorf("Please provide a source image with `from` prior to run")
 	}
-	config, _, _, err := ParseRun(append([]string{b.image}, b.buildCmdFromJson(args)...), nil)
+	config, _, _, err := runconfig.Parse(append([]string{b.image}, b.buildCmdFromJson(args)...), nil)
 	if err != nil {
 		return err
 	}
 
 	cmd := b.config.Cmd
 	b.config.Cmd = nil
-	MergeConfig(b.config, config)
+	runconfig.Merge(b.config, config)
 
 	defer func(cmd []string) { b.config.Cmd = cmd }(cmd)
 
@@ -179,11 +183,20 @@ func (b *buildFile) CmdRun(args string) error {
 		return nil
 	}
 
-	cid, err := b.run()
+	c, err := b.create()
 	if err != nil {
 		return err
 	}
-	if err := b.commit(cid, cmd, "run"); err != nil {
+	// Ensure that we keep the container mounted until the commit
+	// to avoid unmounting and then mounting directly again
+	c.Mount()
+	defer c.Unmount()
+
+	err = b.run(c)
+	if err != nil {
+		return err
+	}
+	if err := b.commit(c.ID, cmd, "run"); err != nil {
 		return err
 	}
 
@@ -342,7 +355,7 @@ func (b *buildFile) checkPathForAddition(orig string) error {
 	return nil
 }
 
-func (b *buildFile) addContext(container *Container, orig, dest string) error {
+func (b *buildFile) addContext(container *Container, orig, dest string, remote bool) error {
 	var (
 		origPath = path.Join(b.contextPath, orig)
 		destPath = path.Join(container.BasefsPath(), dest)
@@ -358,20 +371,39 @@ func (b *buildFile) addContext(container *Container, orig, dest string) error {
 		}
 		return err
 	}
+
 	if fi.IsDir() {
 		if err := archive.CopyWithTar(origPath, destPath); err != nil {
 			return err
 		}
-		// First try to unpack the source as an archive
-	} else if err := archive.UntarPath(origPath, destPath); err != nil {
+		return nil
+	}
+
+	// First try to unpack the source as an archive
+	// to support the untar feature we need to clean up the path a little bit
+	// because tar is very forgiving.  First we need to strip off the archive's
+	// filename from the path but this is only added if it does not end in / .
+	tarDest := destPath
+	if strings.HasSuffix(tarDest, "/") {
+		tarDest = filepath.Dir(destPath)
+	}
+
+	// If we are adding a remote file, do not try to untar it
+	if !remote {
+		// try to successfully untar the orig
+		if err := archive.UntarPath(origPath, tarDest); err == nil {
+			return nil
+		}
 		utils.Debugf("Couldn't untar %s to %s: %s", origPath, destPath, err)
-		// If that fails, just copy it as a regular file
-		if err := os.MkdirAll(path.Dir(destPath), 0755); err != nil {
-			return err
-		}
-		if err := archive.CopyWithTar(origPath, destPath); err != nil {
-			return err
-		}
+	}
+
+	// If that fails, just copy it as a regular file
+	// but do not use all the magic path handling for the tar path
+	if err := os.MkdirAll(path.Dir(destPath), 0755); err != nil {
+		return err
+	}
+	if err := archive.CopyWithTar(origPath, destPath); err != nil {
+		return err
 	}
 	return nil
 }
@@ -399,14 +431,15 @@ func (b *buildFile) CmdAdd(args string) error {
 	b.config.Cmd = []string{"/bin/sh", "-c", fmt.Sprintf("#(nop) ADD %s in %s", orig, dest)}
 	b.config.Image = b.image
 
-	// FIXME: do we really need this?
 	var (
 		origPath   = orig
 		destPath   = dest
 		remoteHash string
+		isRemote   bool
 	)
 
 	if utils.IsURL(orig) {
+		isRemote = true
 		resp, err := utils.Download(orig)
 		if err != nil {
 			return err
@@ -435,6 +468,7 @@ func (b *buildFile) CmdAdd(args string) error {
 		}
 		tarSum := utils.TarSum{Reader: r, DisableCompression: true}
 		remoteHash = tarSum.Sum(nil)
+		r.Close()
 
 		// If the destination is a directory, figure out the filename.
 		if strings.HasSuffix(dest, "/") {
@@ -515,7 +549,7 @@ func (b *buildFile) CmdAdd(args string) error {
 	}
 	defer container.Unmount()
 
-	if err := b.addContext(container, origPath, destPath); err != nil {
+	if err := b.addContext(container, origPath, destPath, isRemote); err != nil {
 		return err
 	}
 
@@ -554,16 +588,16 @@ func (sf *StderrFormater) Write(buf []byte) (int, error) {
 	return len(buf), err
 }
 
-func (b *buildFile) run() (string, error) {
+func (b *buildFile) create() (*Container, error) {
 	if b.image == "" {
-		return "", fmt.Errorf("Please provide a source image with `from` prior to run")
+		return nil, fmt.Errorf("Please provide a source image with `from` prior to run")
 	}
 	b.config.Image = b.image
 
 	// Create the container and start it
 	c, _, err := b.runtime.Create(b.config, "")
 	if err != nil {
-		return "", err
+		return nil, err
 	}
 	b.tmpContainers[c.ID] = struct{}{}
 	fmt.Fprintf(b.outStream, " ---> Running in %s\n", utils.TruncateID(c.ID))
@@ -572,6 +606,10 @@ func (b *buildFile) run() (string, error) {
 	c.Path = b.config.Cmd[0]
 	c.Args = b.config.Cmd[1:]
 
+	return c, nil
+}
+
+func (b *buildFile) run(c *Container) error {
 	var errCh chan error
 
 	if b.verbose {
@@ -582,12 +620,12 @@ func (b *buildFile) run() (string, error) {
 
 	//start the container
 	if err := c.Start(); err != nil {
-		return "", err
+		return err
 	}
 
 	if errCh != nil {
 		if err := <-errCh; err != nil {
-			return "", err
+			return err
 		}
 	}
 
@@ -597,10 +635,10 @@ func (b *buildFile) run() (string, error) {
 			Message: fmt.Sprintf("The command %v returned a non-zero code: %d", b.config.Cmd, ret),
 			Code:    ret,
 		}
-		return "", err
+		return err
 	}
 
-	return c.ID, nil
+	return nil
 }
 
 // Commit the container <id> with the autorun command <autoCmd>
@@ -742,7 +780,7 @@ func NewBuildFile(srv *Server, outStream, errStream io.Writer, verbose, utilizeC
 	return &buildFile{
 		runtime:       srv.runtime,
 		srv:           srv,
-		config:        &Config{},
+		config:        &runconfig.Config{},
 		outStream:     outStream,
 		errStream:     errStream,
 		tmpContainers: make(map[string]struct{}),

commands_unit_test.go

@@ -1,16 +1,17 @@
 package docker
 
 import (
+	"github.com/dotcloud/docker/runconfig"
 	"strings"
 	"testing"
 )
 
-func parse(t *testing.T, args string) (*Config, *HostConfig, error) {
-	config, hostConfig, _, err := ParseRun(strings.Split(args+" ubuntu bash", " "), nil)
+func parse(t *testing.T, args string) (*runconfig.Config, *runconfig.HostConfig, error) {
+	config, hostConfig, _, err := runconfig.Parse(strings.Split(args+" ubuntu bash", " "), nil)
 	return config, hostConfig, err
 }
 
-func mustParse(t *testing.T, args string) (*Config, *HostConfig) {
+func mustParse(t *testing.T, args string) (*runconfig.Config, *runconfig.HostConfig) {
 	config, hostConfig, err := parse(t, args)
 	if err != nil {
 		t.Fatal(err)

config.go

@@ -39,6 +39,7 @@ func DaemonConfigFromJob(job *engine.Job) *DaemonConfig {
 		EnableIptables:              job.GetenvBool("EnableIptables"),
 		EnableIpForward:             job.GetenvBool("EnableIpForward"),
 		BridgeIP:                    job.Getenv("BridgeIP"),
+		BridgeIface:                 job.Getenv("BridgeIface"),
 		DefaultIp:                   net.ParseIP(job.Getenv("DefaultIp")),
 		InterContainerCommunication: job.GetenvBool("InterContainerCommunication"),
 		GraphDriver:                 job.Getenv("GraphDriver"),
@@ -51,7 +52,7 @@ func DaemonConfigFromJob(job *engine.Job) *DaemonConfig {
 	} else {
 		config.Mtu = GetDefaultNetworkMtu()
 	}
-	config.DisableNetwork = job.Getenv("BridgeIface") == DisableNetworkBridge
+	config.DisableNetwork = config.BridgeIface == DisableNetworkBridge
 
 	return config
 }

container.go

@@ -8,8 +8,10 @@ import (
 	"github.com/dotcloud/docker/engine"
 	"github.com/dotcloud/docker/execdriver"
 	"github.com/dotcloud/docker/graphdriver"
-	"github.com/dotcloud/docker/pkg/mount"
+	"github.com/dotcloud/docker/links"
+	"github.com/dotcloud/docker/nat"
 	"github.com/dotcloud/docker/pkg/term"
+	"github.com/dotcloud/docker/runconfig"
 	"github.com/dotcloud/docker/utils"
 	"github.com/kr/pty"
 	"io"
@@ -17,7 +19,6 @@ import (
 	"log"
 	"os"
 	"path"
-	"path/filepath"
 	"strings"
 	"sync"
 	"syscall"
@@ -25,8 +26,10 @@ import (
 )
 
 var (
-	ErrNotATTY = errors.New("The PTY is not a file")
-	ErrNoTTY   = errors.New("No PTY found")
+	ErrNotATTY               = errors.New("The PTY is not a file")
+	ErrNoTTY                 = errors.New("No PTY found")
+	ErrContainerStart        = errors.New("The container failed to start. Unknown error")
+	ErrContainerStartTimeout = errors.New("The container failed to start due to timed out.")
 )
 
 type Container struct {
@@ -41,7 +44,7 @@ type Container struct {
 	Path string
 	Args []string
 
-	Config *Config
+	Config *runconfig.Config
 	State  State
 	Image  string
 
@@ -67,160 +70,12 @@ type Container struct {
 	// Store rw/ro in a separate structure to preserve reverse-compatibility on-disk.
 	// Easier than migrating older container configs :)
 	VolumesRW  map[string]bool
-	hostConfig *HostConfig
+	hostConfig *runconfig.HostConfig
 
-	activeLinks map[string]*Link
-}
-
-// Note: the Config structure should hold only portable information about the container.
-// Here, "portable" means "independent from the host we are running on".
-// Non-portable information *should* appear in HostConfig.
-type Config struct {
-	Hostname        string
-	Domainname      string
-	User            string
-	Memory          int64 // Memory limit (in bytes)
-	MemorySwap      int64 // Total memory usage (memory + swap); set `-1' to disable swap
-	CpuShares       int64 // CPU shares (relative weight vs. other containers)
-	AttachStdin     bool
-	AttachStdout    bool
-	AttachStderr    bool
-	PortSpecs       []string // Deprecated - Can be in the format of 8080/tcp
-	ExposedPorts    map[Port]struct{}
-	Tty             bool // Attach standard streams to a tty, including stdin if it is not closed.
-	OpenStdin       bool // Open stdin
-	StdinOnce       bool // If true, close stdin after the 1 attached client disconnects.
-	Env             []string
-	Cmd             []string
-	Dns             []string
-	Image           string // Name of the image as it was passed by the operator (eg. could be symbolic)
-	Volumes         map[string]struct{}
-	VolumesFrom     string
-	WorkingDir      string
-	Entrypoint      []string
-	NetworkDisabled bool
-	OnBuild         []string
-}
-
-func ContainerConfigFromJob(job *engine.Job) *Config {
-	config := &Config{
-		Hostname:        job.Getenv("Hostname"),
-		Domainname:      job.Getenv("Domainname"),
-		User:            job.Getenv("User"),
-		Memory:          job.GetenvInt64("Memory"),
-		MemorySwap:      job.GetenvInt64("MemorySwap"),
-		CpuShares:       job.GetenvInt64("CpuShares"),
-		AttachStdin:     job.GetenvBool("AttachStdin"),
-		AttachStdout:    job.GetenvBool("AttachStdout"),
-		AttachStderr:    job.GetenvBool("AttachStderr"),
-		Tty:             job.GetenvBool("Tty"),
-		OpenStdin:       job.GetenvBool("OpenStdin"),
-		StdinOnce:       job.GetenvBool("StdinOnce"),
-		Image:           job.Getenv("Image"),
-		VolumesFrom:     job.Getenv("VolumesFrom"),
-		WorkingDir:      job.Getenv("WorkingDir"),
-		NetworkDisabled: job.GetenvBool("NetworkDisabled"),
-	}
-	job.GetenvJson("ExposedPorts", &config.ExposedPorts)
-	job.GetenvJson("Volumes", &config.Volumes)
-	if PortSpecs := job.GetenvList("PortSpecs"); PortSpecs != nil {
-		config.PortSpecs = PortSpecs
-	}
-	if Env := job.GetenvList("Env"); Env != nil {
-		config.Env = Env
-	}
-	if Cmd := job.GetenvList("Cmd"); Cmd != nil {
-		config.Cmd = Cmd
-	}
-	if Dns := job.GetenvList("Dns"); Dns != nil {
-		config.Dns = Dns
-	}
-	if Entrypoint := job.GetenvList("Entrypoint"); Entrypoint != nil {
-		config.Entrypoint = Entrypoint
-	}
-
-	return config
-}
-
-type HostConfig struct {
-	Binds           []string
-	ContainerIDFile string
-	LxcConf         []KeyValuePair
-	Privileged      bool
-	PortBindings    map[Port][]PortBinding
-	Links           []string
-	PublishAllPorts bool
-}
-
-func ContainerHostConfigFromJob(job *engine.Job) *HostConfig {
-	hostConfig := &HostConfig{
-		ContainerIDFile: job.Getenv("ContainerIDFile"),
-		Privileged:      job.GetenvBool("Privileged"),
-		PublishAllPorts: job.GetenvBool("PublishAllPorts"),
-	}
-	job.GetenvJson("LxcConf", &hostConfig.LxcConf)
-	job.GetenvJson("PortBindings", &hostConfig.PortBindings)
-	if Binds := job.GetenvList("Binds"); Binds != nil {
-		hostConfig.Binds = Binds
-	}
-	if Links := job.GetenvList("Links"); Links != nil {
-		hostConfig.Links = Links
-	}
-
-	return hostConfig
-}
-
-type BindMap struct {
-	SrcPath string
-	DstPath string
-	Mode    string
-}
-
-var (
-	ErrContainerStart           = errors.New("The container failed to start. Unknown error")
-	ErrContainerStartTimeout    = errors.New("The container failed to start due to timed out.")
-	ErrInvalidWorikingDirectory = errors.New("The working directory is invalid. It needs to be an absolute path.")
-	ErrConflictAttachDetach     = errors.New("Conflicting options: -a and -d")
-	ErrConflictDetachAutoRemove = errors.New("Conflicting options: -rm and -d")
-)
-
-type KeyValuePair struct {
-	Key   string
-	Value string
-}
-
-type PortBinding struct {
-	HostIp   string
-	HostPort string
-}
-
-// 80/tcp
-type Port string
-
-func (p Port) Proto() string {
-	parts := strings.Split(string(p), "/")
-	if len(parts) == 1 {
-		return "tcp"
-	}
-	return parts[1]
-}
-
-func (p Port) Port() string {
-	return strings.Split(string(p), "/")[0]
-}
-
-func (p Port) Int() int {
-	i, err := parsePort(p.Port())
-	if err != nil {
-		panic(err)
-	}
-	return i
-}
-
-func NewPort(proto, port string) Port {
-	return Port(fmt.Sprintf("%s/%s", port, proto))
+	activeLinks map[string]*links.Link
 }
 
+// FIXME: move deprecated port stuff to nat to clean up the core.
 type PortMapping map[string]string // Deprecated
 
 type NetworkSettings struct {
@@ -229,13 +84,13 @@ type NetworkSettings struct {
 	Gateway     string
 	Bridge      string
 	PortMapping map[string]PortMapping // Deprecated
-	Ports       map[Port][]PortBinding
+	Ports       nat.PortMap
 }
 
 func (settings *NetworkSettings) PortMappingAPI() *engine.Table {
 	var outs = engine.NewTable("", 0)
 	for port, bindings := range settings.Ports {
-		p, _ := parsePort(port.Port())
+		p, _ := nat.ParsePort(port.Port())
 		if len(bindings) == 0 {
 			out := &engine.Env{}
 			out.SetInt("PublicPort", p)
@@ -245,7 +100,7 @@ func (settings *NetworkSettings) PortMappingAPI() *engine.Table {
 		}
 		for _, binding := range bindings {
 			out := &engine.Env{}
-			h, _ := parsePort(binding.HostPort)
+			h, _ := nat.ParsePort(binding.HostPort)
 			out.SetInt("PrivatePort", p)
 			out.SetInt("PublicPort", h)
 			out.Set("Type", port.Proto())
@@ -322,7 +177,7 @@ func (container *Container) ToDisk() (err error) {
 }
 
 func (container *Container) readHostConfig() error {
-	container.hostConfig = &HostConfig{}
+	container.hostConfig = &runconfig.HostConfig{}
 	// If the hostconfig file does not exist, do not read it.
 	// (We still have to initialize container.hostConfig,
 	// but that's OK, since we just did that above.)
@@ -366,6 +221,7 @@ func (container *Container) setupPty() error {
 	container.ptyMaster = ptyMaster
 	container.command.Stdout = ptySlave
 	container.command.Stderr = ptySlave
+	container.command.Console = ptySlave.Name()
 
 	// Copy the PTYs to our broadcasters
 	go func() {
@@ -637,17 +493,7 @@ func (container *Container) Start() (err error) {
 		log.Printf("WARNING: IPv4 forwarding is disabled. Networking will not work")
 	}
 
-	if container.Volumes == nil || len(container.Volumes) == 0 {
-		container.Volumes = make(map[string]string)
-		container.VolumesRW = make(map[string]bool)
-	}
-
-	// Apply volumes from another container if requested
-	if err := container.applyExternalVolumes(); err != nil {
-		return err
-	}
-
-	if err := container.createVolumes(); err != nil {
+	if err := prepareVolumesForContainer(container); err != nil {
 		return err
 	}
 
@@ -671,7 +517,7 @@ func (container *Container) Start() (err error) {
 	}
 
 	if len(children) > 0 {
-		container.activeLinks = make(map[string]*Link, len(children))
+		container.activeLinks = make(map[string]*links.Link, len(children))
 
 		// If we encounter an error make sure that we rollback any network
 		// config and ip table changes
@@ -682,8 +528,19 @@ func (container *Container) Start() (err error) {
 			container.activeLinks = nil
 		}
 
-		for p, child := range children {
-			link, err := NewLink(container, child, p, runtime.eng)
+		for linkAlias, child := range children {
+			if !child.State.IsRunning() {
+				return fmt.Errorf("Cannot link to a non running container: %s AS %s", child.Name, linkAlias)
+			}
+
+			link, err := links.NewLink(
+				container.NetworkSettings.IPAddress,
+				child.NetworkSettings.IPAddress,
+				linkAlias,
+				child.Config.Env,
+				child.Config.ExposedPorts,
+				runtime.eng)
+
 			if err != nil {
 				rollback()
 				return err
@@ -721,62 +578,10 @@ func (container *Container) Start() (err error) {
 		return err
 	}
 
-	// Setup the root fs as a bind mount of the base fs
-	root := container.RootfsPath()
-	if err := os.MkdirAll(root, 0755); err != nil && !os.IsExist(err) {
-		return nil
-	}
-
-	// Create a bind mount of the base fs as a place where we can add mounts
-	// without affecting the ability to access the base fs
-	if err := mount.Mount(container.basefs, root, "none", "bind,rw"); err != nil {
+	if err := mountVolumesForContainer(container, envPath); err != nil {
 		return err
 	}
 
-	// Make sure the root fs is private so the mounts here don't propagate to basefs
-	if err := mount.ForceMount(root, root, "none", "private"); err != nil {
-		return err
-	}
-
-	// Mount docker specific files into the containers root fs
-	if err := mount.Mount(runtime.sysInitPath, path.Join(root, "/.dockerinit"), "none", "bind,ro"); err != nil {
-		return err
-	}
-	if err := mount.Mount(envPath, path.Join(root, "/.dockerenv"), "none", "bind,ro"); err != nil {
-		return err
-	}
-	if err := mount.Mount(container.ResolvConfPath, path.Join(root, "/etc/resolv.conf"), "none", "bind,ro"); err != nil {
-		return err
-	}
-
-	if container.HostnamePath != "" && container.HostsPath != "" {
-		if err := mount.Mount(container.HostnamePath, path.Join(root, "/etc/hostname"), "none", "bind,ro"); err != nil {
-			return err
-		}
-		if err := mount.Mount(container.HostsPath, path.Join(root, "/etc/hosts"), "none", "bind,ro"); err != nil {
-			return err
-		}
-	}
-
-	// Mount user specified volumes
-	for r, v := range container.Volumes {
-		mountAs := "ro"
-		if container.VolumesRW[r] {
-			mountAs = "rw"
-		}
-
-		r = path.Join(root, r)
-		if p, err := utils.FollowSymlinkInScope(r, root); err != nil {
-			return err
-		} else {
-			r = p
-		}
-
-		if err := mount.Mount(v, r, "none", fmt.Sprintf("bind,%s", mountAs)); err != nil {
-			return err
-		}
-	}
-
 	populateCommand(container)
 
 	// Setup logging of stdout and stderr to disk
@@ -829,205 +634,6 @@ func (container *Container) Start() (err error) {
 	return nil
 }
 
-func (container *Container) getBindMap() (map[string]BindMap, error) {
-	// Create the requested bind mounts
-	binds := make(map[string]BindMap)
-	// Define illegal container destinations
-	illegalDsts := []string{"/", "."}
-
-	for _, bind := range container.hostConfig.Binds {
-		// FIXME: factorize bind parsing in parseBind
-		var src, dst, mode string
-		arr := strings.Split(bind, ":")
-		if len(arr) == 2 {
-			src = arr[0]
-			dst = arr[1]
-			mode = "rw"
-		} else if len(arr) == 3 {
-			src = arr[0]
-			dst = arr[1]
-			mode = arr[2]
-		} else {
-			return nil, fmt.Errorf("Invalid bind specification: %s", bind)
-		}
-
-		// Bail if trying to mount to an illegal destination
-		for _, illegal := range illegalDsts {
-			if dst == illegal {
-				return nil, fmt.Errorf("Illegal bind destination: %s", dst)
-			}
-		}
-
-		bindMap := BindMap{
-			SrcPath: src,
-			DstPath: dst,
-			Mode:    mode,
-		}
-		binds[path.Clean(dst)] = bindMap
-	}
-	return binds, nil
-}
-
-func (container *Container) createVolumes() error {
-	binds, err := container.getBindMap()
-	if err != nil {
-		return err
-	}
-	volumesDriver := container.runtime.volumes.driver
-	// Create the requested volumes if they don't exist
-	for volPath := range container.Config.Volumes {
-		volPath = path.Clean(volPath)
-		volIsDir := true
-		// Skip existing volumes
-		if _, exists := container.Volumes[volPath]; exists {
-			continue
-		}
-		var srcPath string
-		var isBindMount bool
-		srcRW := false
-		// If an external bind is defined for this volume, use that as a source
-		if bindMap, exists := binds[volPath]; exists {
-			isBindMount = true
-			srcPath = bindMap.SrcPath
-			if strings.ToLower(bindMap.Mode) == "rw" {
-				srcRW = true
-			}
-			if stat, err := os.Stat(bindMap.SrcPath); err != nil {
-				return err
-			} else {
-				volIsDir = stat.IsDir()
-			}
-			// Otherwise create an directory in $ROOT/volumes/ and use that
-		} else {
-
-			// Do not pass a container as the parameter for the volume creation.
-			// The graph driver using the container's information ( Image ) to
-			// create the parent.
-			c, err := container.runtime.volumes.Create(nil, nil, "", "", nil)
-			if err != nil {
-				return err
-			}
-			srcPath, err = volumesDriver.Get(c.ID)
-			if err != nil {
-				return fmt.Errorf("Driver %s failed to get volume rootfs %s: %s", volumesDriver, c.ID, err)
-			}
-			srcRW = true // RW by default
-		}
-
-		if p, err := filepath.EvalSymlinks(srcPath); err != nil {
-			return err
-		} else {
-			srcPath = p
-		}
-
-		container.Volumes[volPath] = srcPath
-		container.VolumesRW[volPath] = srcRW
-
-		// Create the mountpoint
-		volPath = path.Join(container.basefs, volPath)
-		rootVolPath, err := utils.FollowSymlinkInScope(volPath, container.basefs)
-		if err != nil {
-			return err
-		}
-
-		if _, err := os.Stat(rootVolPath); err != nil {
-			if os.IsNotExist(err) {
-				if volIsDir {
-					if err := os.MkdirAll(rootVolPath, 0755); err != nil {
-						return err
-					}
-				} else {
-					if err := os.MkdirAll(path.Dir(rootVolPath), 0755); err != nil {
-						return err
-					}
-					if f, err := os.OpenFile(rootVolPath, os.O_CREATE, 0755); err != nil {
-						return err
-					} else {
-						f.Close()
-					}
-				}
-			}
-		}
-
-		// Do not copy or change permissions if we are mounting from the host
-		if srcRW && !isBindMount {
-			volList, err := ioutil.ReadDir(rootVolPath)
-			if err != nil {
-				return err
-			}
-			if len(volList) > 0 {
-				srcList, err := ioutil.ReadDir(srcPath)
-				if err != nil {
-					return err
-				}
-				if len(srcList) == 0 {
-					// If the source volume is empty copy files from the root into the volume
-					if err := archive.CopyWithTar(rootVolPath, srcPath); err != nil {
-						return err
-					}
-
-					var stat syscall.Stat_t
-					if err := syscall.Stat(rootVolPath, &stat); err != nil {
-						return err
-					}
-					var srcStat syscall.Stat_t
-					if err := syscall.Stat(srcPath, &srcStat); err != nil {
-						return err
-					}
-					// Change the source volume's ownership if it differs from the root
-					// files that were just copied
-					if stat.Uid != srcStat.Uid || stat.Gid != srcStat.Gid {
-						if err := os.Chown(srcPath, int(stat.Uid), int(stat.Gid)); err != nil {
-							return err
-						}
-					}
-				}
-			}
-		}
-	}
-	return nil
-}
-
-func (container *Container) applyExternalVolumes() error {
-	if container.Config.VolumesFrom != "" {
-		containerSpecs := strings.Split(container.Config.VolumesFrom, ",")
-		for _, containerSpec := range containerSpecs {
-			mountRW := true
-			specParts := strings.SplitN(containerSpec, ":", 2)
-			switch len(specParts) {
-			case 0:
-				return fmt.Errorf("Malformed volumes-from specification: %s", container.Config.VolumesFrom)
-			case 2:
-				switch specParts[1] {
-				case "ro":
-					mountRW = false
-				case "rw": // mountRW is already true
-				default:
-					return fmt.Errorf("Malformed volumes-from specification: %s", containerSpec)
-				}
-			}
-			c := container.runtime.Get(specParts[0])
-			if c == nil {
-				return fmt.Errorf("Container %s not found. Impossible to mount its volumes", container.ID)
-			}
-			for volPath, id := range c.Volumes {
-				if _, exists := container.Volumes[volPath]; exists {
-					continue
-				}
-				if err := os.MkdirAll(path.Join(container.basefs, volPath), 0755); err != nil {
-					return err
-				}
-				container.Volumes[volPath] = id
-				if isRW, exists := c.VolumesRW[volPath]; exists {
-					container.VolumesRW[volPath] = isRW && mountRW
-				}
-			}
-
-		}
-	}
-	return nil
-}
-
 func (container *Container) Run() error {
 	if err := container.Start(); err != nil {
 		return err
@@ -1152,8 +758,8 @@ func (container *Container) allocateNetwork() error {
 	}
 
 	var (
-		portSpecs = make(map[Port]struct{})
-		bindings  = make(map[Port][]PortBinding)
+		portSpecs = make(nat.PortSet)
+		bindings  = make(nat.PortMap)
 	)
 
 	if !container.State.IsGhost() {
@@ -1177,7 +783,7 @@ func (container *Container) allocateNetwork() error {
 	for port := range portSpecs {
 		binding := bindings[port]
 		if container.hostConfig.PublishAllPorts && len(binding) == 0 {
-			binding = append(binding, PortBinding{})
+			binding = append(binding, nat.PortBinding{})
 		}
 
 		for i := 0; i < len(binding); i++ {
@@ -1300,29 +906,7 @@ func (container *Container) cleanup() {
 		}
 	}
 
-	var (
-		root   = container.RootfsPath()
-		mounts = []string{
-			root,
-			path.Join(root, "/.dockerinit"),
-			path.Join(root, "/.dockerenv"),
-			path.Join(root, "/etc/resolv.conf"),
-		}
-	)
-
-	if container.HostnamePath != "" && container.HostsPath != "" {
-		mounts = append(mounts, path.Join(root, "/etc/hostname"), path.Join(root, "/etc/hosts"))
-	}
-
-	for r := range container.Volumes {
-		mounts = append(mounts, path.Join(root, r))
-	}
-
-	for i := len(mounts) - 1; i >= 0; i-- {
-		if lastError := mount.Unmount(mounts[i]); lastError != nil {
-			log.Printf("Failed to umount %v: %v", mounts[i], lastError)
-		}
-	}
+	unmountVolumesForContainer(container)
 
 	if err := container.Unmount(); err != nil {
 		log.Printf("%v: Failed to umount filesystem: %v", container.ID, err)
@@ -1390,6 +974,13 @@ func (container *Container) Stop(seconds int) error {
 }
 
 func (container *Container) Restart(seconds int) error {
+	// Avoid unnecessarily unmounting and then directly mounting
+	// the container when the container stops and then starts
+	// again
+	if err := container.Mount(); err == nil {
+		defer container.Unmount()
+	}
+
 	if err := container.Stop(seconds); err != nil {
 		return err
 	}
@@ -1422,7 +1013,11 @@ func (container *Container) ExportRw() (archive.Archive, error) {
 		container.Unmount()
 		return nil, err
 	}
-	return EofReader(archive, func() { container.Unmount() }), nil
+	return utils.NewReadCloserWrapper(archive, func() error {
+		err := archive.Close()
+		container.Unmount()
+		return err
+	}), nil
 }
 
 func (container *Container) Export() (archive.Archive, error) {
@@ -1435,7 +1030,11 @@ func (container *Container) Export() (archive.Archive, error) {
 		container.Unmount()
 		return nil, err
 	}
-	return EofReader(archive, func() { container.Unmount() }), nil
+	return utils.NewReadCloserWrapper(archive, func() error {
+		err := archive.Close()
+		container.Unmount()
+		return err
+	}), nil
 }
 
 func (container *Container) WaitTimeout(timeout time.Duration) error {
@@ -1562,7 +1161,7 @@ func (container *Container) GetSize() (int64, int64) {
 	return sizeRw, sizeRootfs
 }
 
-func (container *Container) Copy(resource string) (archive.Archive, error) {
+func (container *Container) Copy(resource string) (io.ReadCloser, error) {
 	if err := container.Mount(); err != nil {
 		return nil, err
 	}
@@ -1589,11 +1188,15 @@ func (container *Container) Copy(resource string) (archive.Archive, error) {
 	if err != nil {
 		return nil, err
 	}
-	return EofReader(archive, func() { container.Unmount() }), nil
+	return utils.NewReadCloserWrapper(archive, func() error {
+		err := archive.Close()
+		container.Unmount()
+		return err
+	}), nil
 }
 
 // Returns true if the container exposes a certain port
-func (container *Container) Exposes(p Port) bool {
+func (container *Container) Exposes(p nat.Port) bool {
 	_, exists := container.Config.ExposedPorts[p]
 	return exists
 }

container_unit_test.go

@@ -1,28 +1,12 @@
 package docker
 
 import (
+	"github.com/dotcloud/docker/nat"
 	"testing"
 )
 
-func TestParseLxcConfOpt(t *testing.T) {
-	opts := []string{"lxc.utsname=docker", "lxc.utsname = docker "}
-
-	for _, o := range opts {
-		k, v, err := parseLxcOpt(o)
-		if err != nil {
-			t.FailNow()
-		}
-		if k != "lxc.utsname" {
-			t.Fail()
-		}
-		if v != "docker" {
-			t.Fail()
-		}
-	}
-}
-
 func TestParseNetworkOptsPrivateOnly(t *testing.T) {
-	ports, bindings, err := parsePortSpecs([]string{"192.168.1.100::80"})
+	ports, bindings, err := nat.ParsePortSpecs([]string{"192.168.1.100::80"})
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -64,7 +48,7 @@ func TestParseNetworkOptsPrivateOnly(t *testing.T) {
 }
 
 func TestParseNetworkOptsPublic(t *testing.T) {
-	ports, bindings, err := parsePortSpecs([]string{"192.168.1.100:8080:80"})
+	ports, bindings, err := nat.ParsePortSpecs([]string{"192.168.1.100:8080:80"})
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -106,7 +90,7 @@ func TestParseNetworkOptsPublic(t *testing.T) {
 }
 
 func TestParseNetworkOptsUdp(t *testing.T) {
-	ports, bindings, err := parsePortSpecs([]string{"192.168.1.100::6000/udp"})
+	ports, bindings, err := nat.ParsePortSpecs([]string{"192.168.1.100::6000/udp"})
 	if err != nil {
 		t.Fatal(err)
 	}

contrib/desktop-integration/README.md

@@ -8,4 +8,4 @@ Examples
 ========
 
 * Data container: ./data/Dockerfile creates a data image sharing /data volume
-* Firefox: ./firefox/Dockerfile shows a way to dockerize a common multimedia application
+* Iceweasel: ./iceweasel/Dockerfile shows a way to dockerize a common multimedia application

contrib/desktop-integration/data/Dockerfile

@@ -11,28 +11,28 @@
 #   # Build data image
 #   docker build -t data -rm .
 #
-#   # Create a data container. (eg: firefox-data)
-#   docker run -name firefox-data data true
+#   # Create a data container. (eg: iceweasel-data)
+#   docker run -name iceweasel-data data true
 #
 #   # List data from it
-#   docker run -volumes-from firefox-data busybox ls -al /data
+#   docker run -volumes-from iceweasel-data busybox ls -al /data
 
 docker-version 0.6.5
 
 # Smallest base image, just to launch a container
-from busybox
-maintainer	Daniel Mizyrycki <daniel@docker.com>
+FROM busybox
+MAINTAINER Daniel Mizyrycki <daniel@docker.com>
 
 # Create a regular user
-run echo 'sysadmin:x:1000:1000::/data:/bin/sh' >> /etc/passwd
-run echo 'sysadmin:x:1000:' >> /etc/group
+RUN echo 'sysadmin:x:1000:1000::/data:/bin/sh' >> /etc/passwd
+RUN echo 'sysadmin:x:1000:' >> /etc/group
 
 # Create directory for that user
-run mkdir /data
-run chown sysadmin.sysadmin /data
+RUN mkdir /data
+RUN chown sysadmin.sysadmin /data
 
 # Add content to /data. This will keep sysadmin ownership
-run touch /data/init_volume
+RUN touch /data/init_volume
 
 # Create /data volume
 VOLUME /data

contrib/desktop-integration/firefox/Dockerfile

@@ -1,49 +0,0 @@
-# VERSION:        0.7
-# DESCRIPTION:    Create firefox container with its dependencies
-# AUTHOR:         Daniel Mizyrycki <daniel@dotcloud.com>
-# COMMENTS:
-#   This file describes how to build a Firefox container with all
-#   dependencies installed. It uses native X11 unix socket and alsa
-#   sound devices. Tested on Debian 7.2
-# USAGE:
-#   # Download Firefox Dockerfile
-#   wget http://raw.github.com/dotcloud/docker/master/contrib/desktop-integration/firefox/Dockerfile
-#
-#   # Build firefox image
-#   docker build -t firefox -rm .
-#
-#   # Run stateful data-on-host firefox. For ephemeral, remove -v /data/firefox:/data
-#   docker run -v /data/firefox:/data -v /tmp/.X11-unix:/tmp/.X11-unix \
-#     -v /dev/snd:/dev/snd -lxc-conf='lxc.cgroup.devices.allow = c 116:* rwm' \
-#     -e DISPLAY=unix$DISPLAY firefox
-#
-#   # To run stateful dockerized data containers
-#   docker run -volumes-from firefox-data -v /tmp/.X11-unix:/tmp/.X11-unix \
-#     -v /dev/snd:/dev/snd -lxc-conf='lxc.cgroup.devices.allow = c 116:* rwm' \
-#     -e DISPLAY=unix$DISPLAY firefox
-
-docker-version 0.6.5
-
-# Base docker image
-from tianon/debian:wheezy
-maintainer	Daniel Mizyrycki <daniel@docker.com>
-
-# Install firefox dependencies
-run echo "deb http://ftp.debian.org/debian/ wheezy main contrib" > /etc/apt/sources.list
-run apt-get update
-run DEBIAN_FRONTEND=noninteractive apt-get install -y libXrender1 libasound2 \
-    libdbus-glib-1-2 libgtk2.0-0 libpango1.0-0 libxt6 wget bzip2 sudo
-
-# Install Firefox
-run mkdir /application
-run cd /application; wget -O - \
-    http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/25.0/linux-x86_64/en-US/firefox-25.0.tar.bz2 | tar jx
-
-# create sysadmin account
-run useradd -m -d /data -p saIVpsc0EVTwA sysadmin
-run sed -Ei 's/sudo:x:27:/sudo:x:27:sysadmin/' /etc/group
-run sed -Ei 's/(\%sudo\s+ALL=\(ALL\:ALL\) )ALL/\1 NOPASSWD:ALL/' /etc/sudoers
-
-# Autorun firefox. -no-remote is necessary to create a new container, as firefox
-# appears to communicate with itself through X11.
-cmd ["/bin/sh", "-c", "/usr/bin/sudo -u sysadmin -H -E /application/firefox/firefox -no-remote"]

contrib/desktop-integration/iceweasel/Dockerfile

@@ -0,0 +1,41 @@
+# VERSION:        0.7
+# DESCRIPTION:    Create iceweasel container with its dependencies
+# AUTHOR:         Daniel Mizyrycki <daniel@dotcloud.com>
+# COMMENTS:
+#   This file describes how to build a Iceweasel container with all
+#   dependencies installed. It uses native X11 unix socket and alsa
+#   sound devices. Tested on Debian 7.2
+# USAGE:
+#   # Download Iceweasel Dockerfile
+#   wget http://raw.github.com/dotcloud/docker/master/contrib/desktop-integration/iceweasel/Dockerfile
+#
+#   # Build iceweasel image
+#   docker build -t iceweasel -rm .
+#
+#   # Run stateful data-on-host iceweasel. For ephemeral, remove -v /data/iceweasel:/data
+#   docker run -v /data/iceweasel:/data -v /tmp/.X11-unix:/tmp/.X11-unix \
+#     -v /dev/snd:/dev/snd -lxc-conf='lxc.cgroup.devices.allow = c 116:* rwm' \
+#     -e DISPLAY=unix$DISPLAY iceweasel
+#
+#   # To run stateful dockerized data containers
+#   docker run -volumes-from iceweasel-data -v /tmp/.X11-unix:/tmp/.X11-unix \
+#     -v /dev/snd:/dev/snd -lxc-conf='lxc.cgroup.devices.allow = c 116:* rwm' \
+#     -e DISPLAY=unix$DISPLAY iceweasel
+
+docker-version 0.6.5
+
+# Base docker image
+FROM debian:wheezy
+MAINTAINER Daniel Mizyrycki <daniel@docker.com>
+
+# Install Iceweasel and "sudo"
+RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -yq iceweasel sudo
+
+# create sysadmin account
+RUN useradd -m -d /data -p saIVpsc0EVTwA sysadmin
+RUN sed -Ei 's/sudo:x:27:/sudo:x:27:sysadmin/' /etc/group
+RUN sed -Ei 's/(\%sudo\s+ALL=\(ALL\:ALL\) )ALL/\1 NOPASSWD:ALL/' /etc/sudoers
+
+# Autorun iceweasel. -no-remote is necessary to create a new container, as
+# iceweasel appears to communicate with itself through X11.
+CMD ["/usr/bin/sudo", "-u", "sysadmin", "-H", "-E", "/usr/bin/iceweasel", "-no-remote"]

contrib/init/sysvinit-redhat/docker

@@ -0,0 +1,123 @@
+#!/bin/sh
+#
+#       /etc/rc.d/init.d/docker
+#
+#       Daemon for docker.io
+#       
+# chkconfig:   2345 95 95
+# description: Daemon for docker.io
+
+### BEGIN INIT INFO
+# Provides:       docker
+# Required-Start: $network cgconfig
+# Required-Stop:
+# Should-Start:
+# Should-Stop:
+# Default-Start: 2 3 4 5
+# Default-Stop:  0 1 6
+# Short-Description: start and stop docker
+# Description: Daemon for docker.io
+### END INIT INFO
+
+# Source function library.
+. /etc/rc.d/init.d/functions
+
+prog="docker"
+exec="/usr/bin/$prog"
+pidfile="/var/run/$prog.pid"
+lockfile="/var/lock/subsys/$prog"
+logfile="/var/log/$prog"
+
+[ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog
+
+prestart() {
+    service cgconfig status > /dev/null
+
+    if [[ $? != 0 ]]; then
+        service cgconfig start
+    fi
+
+}
+
+start() {
+    [ -x $exec ] || exit 5
+
+    if ! [ -f $pidfile ]; then
+        prestart
+        printf "Starting $prog:\t"
+        echo "\n$(date)\n" >> $logfile
+        $exec -d $other_args &>> $logfile &
+        pid=$!
+        touch $lockfile
+        success
+        echo
+    else
+        failure
+        echo
+        printf "$pidfile still exists...\n"
+        exit 7
+    fi
+}
+
+stop() {
+    echo -n $"Stopping $prog: "
+    killproc -p $pidfile $prog
+    retval=$?
+    echo
+    [ $retval -eq 0 ] && rm -f $lockfile
+    return $retval
+}
+
+restart() {
+    stop
+    start
+}
+
+reload() {
+    restart
+}
+
+force_reload() {
+    restart
+}
+
+rh_status() {
+    status -p $pidfile $prog
+}
+
+rh_status_q() {
+    rh_status >/dev/null 2>&1
+}
+
+case "$1" in
+    start)
+        rh_status_q && exit 0
+        $1
+        ;;
+    stop)
+        rh_status_q || exit 0
+        $1
+        ;;
+    restart)
+        $1
+        ;;
+    reload)
+        rh_status_q || exit 7
+        $1
+        ;;
+    force-reload)
+        force_reload
+        ;;
+    status)
+        rh_status
+        ;;
+    condrestart|try-restart)
+        rh_status_q || exit 0
+        restart
+        ;;
+    *)
+        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
+        exit 2
+esac
+
+exit $?

contrib/init/sysvinit-redhat/docker.sysconfig

@@ -0,0 +1,7 @@
+# /etc/sysconfig/docker
+# 
+# Other arguments to pass to the docker daemon process
+# These will be parsed by the sysv initscript and appended
+# to the arguments list passed to docker -d
+
+other_args=""

contrib/mkimage-yum.sh

@@ -51,7 +51,7 @@ done
 
 yum -c "$yum_config" --installroot="$target" --setopt=tsflags=nodocs \
     --setopt=group_package_types=mandatory -y groupinstall Core
-yum -c "$yum_config" --installroot="$mount" -y clean all
+yum -c "$yum_config" --installroot="$target" -y clean all
 
 cat > "$target"/etc/sysconfig/network <<EOF
 NETWORKING=yes

contrib/syntax/textmate/Docker.tmbundle/Preferences/Dockerfile.tmPreferences

@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>name</key>
+	<string>Comments</string>
+	<key>scope</key>
+	<string>source.dockerfile</string>
+	<key>settings</key>
+	<dict>
+		<key>shellVariables</key>
+		<array>
+			<dict>
+				<key>name</key>
+				<string>TM_COMMENT_START</string>
+				<key>value</key>
+				<string># </string>
+			</dict>
+		</array>
+	</dict>
+	<key>uuid</key>
+	<string>2B215AC0-A7F3-4090-9FF6-F4842BD56CA7</string>
+</dict>
+</plist>

contrib/syntax/textmate/Docker.tmbundle/Syntaxes/Dockerfile.tmLanguage

@@ -12,15 +12,37 @@
 	<array>
 		<dict>
 			<key>match</key>
-			<string>^\s*(FROM|MAINTAINER|RUN|CMD|EXPOSE|ENV|ADD)\s</string>
-			<key>name</key>
-			<string>keyword.control.dockerfile</string>
+			<string>^\s*(ONBUILD\s+)?(FROM|MAINTAINER|RUN|EXPOSE|ENV|ADD|VOLUME|USER|WORKDIR)\s</string>
+			<key>captures</key>
+			<dict>
+				<key>0</key>
+				<dict>
+					<key>name</key>
+					<string>keyword.control.dockerfile</string>					
+				</dict>
+				<key>1</key>
+				<dict>
+					<key>name</key>
+					<string>keyword.other.special-method.dockerfile</string>					
+				</dict>
+			</dict>
 		</dict>
 		<dict>
 			<key>match</key>
-			<string>^\s*(ENTRYPOINT|VOLUME|USER|WORKDIR)\s</string>
-			<key>name</key>
-			<string>keyword.operator.dockerfile</string>
+			<string>^\s*(ONBUILD\s+)?(CMD|ENTRYPOINT)\s</string>
+			<key>captures</key>
+			<dict>
+				<key>0</key>
+				<dict>
+					<key>name</key>
+					<string>keyword.operator.dockerfile</string>					
+				</dict>
+				<key>1</key>
+				<dict>
+					<key>name</key>
+					<string>keyword.other.special-method.dockerfile</string>					
+				</dict>
+			</dict>
 		</dict>
 		<dict>
 			<key>begin</key>
@@ -39,6 +61,23 @@
 				</dict>
 			</array>
 		</dict>
+		<dict>
+			<key>begin</key>
+			<string>'</string>
+			<key>end</key>
+			<string>'</string>
+			<key>name</key>
+			<string>string.quoted.single.dockerfile</string>
+			<key>patterns</key>
+			<array>
+				<dict>
+					<key>match</key>
+					<string>\\.</string>
+					<key>name</key>
+					<string>constant.character.escaped.dockerfile</string>
+				</dict>
+			</array>
+		</dict>
 		<dict>
 			<key>match</key>
 			<string>^\s*#.*$</string>

contrib/syntax/textmate/Docker.tmbundle/info.plist

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>contactEmailRot13</key>
+	<string>germ@andz.com.ar</string>
+	<key>contactName</key>
+	<string>GermanDZ</string>
+	<key>description</key>
+	<string>Helpers for Docker.</string>
+	<key>name</key>
+	<string>Docker</string>
+	<key>uuid</key>
+	<string>8B9DDBAF-E65C-4E12-FFA7-467D4AA535B1</string>
+</dict>
+</plist>

contrib/syntax/textmate/Dockerfile.YAML-tmLanguage

@@ -1,23 +0,0 @@
-# [PackageDev] target_format: plist, ext: tmLanguage
----
-name: Dockerfile
-scopeName: source.dockerfile
-uuid: a39d8795-59d2-49af-aa00-fe74ee29576e
-
-patterns:
-# Keywords
-- name: keyword.control.dockerfile
-  match: ^\s*(FROM|MAINTAINER|RUN|CMD|EXPOSE|ENV|ADD)\s
-- name: keyword.operator.dockerfile
-  match: ^\s*(ENTRYPOINT|VOLUME|USER|WORKDIR)\s
-# String
-- name: string.quoted.double.dockerfile
-  begin: "\""
-  end: "\""
-  patterns:
-  - name: constant.character.escaped.dockerfile
-    match: \\.
-# Comment
-- name: comment.block.dockerfile
-  match: ^\s*#.*$
-...

contrib/syntax/textmate/README.md

@@ -1,9 +1,16 @@
-# Dockerfile.tmLanguage
+# Docker.tmbundle
 
-Pretty basic Dockerfile.tmLanguage for Sublime Text syntax highlighting.
+Dockerfile syntaxt highlighting for TextMate and Sublime Text.
 
-PR's with syntax updates, suggestions etc. are all very much appreciated!
+## Install
 
-I'll get to making this installable via Package Control soon!
+### Sublime Text
+
+Available for Sublime Text under [package control](https://sublime.wbond.net/packages/Dockerfile%20Syntax%20Highlighting).
+Search for *Dockerfile Syntax Highlighting*
+
+### TextMate 2
+
+Copy the directory `Docker.tmbundle` (showed as a Package in OSX) to `~/Library/Application Support/TextMate/Managed/Bundles`
 
 enjoy.

contrib/syntax/vim/syntax/dockerfile.vim

@@ -11,8 +11,7 @@ let b:current_syntax = "dockerfile"
 
 syntax case ignore
 
-syntax match dockerfileKeyword /\v^\s*(FROM|MAINTAINER|RUN|CMD|EXPOSE|ENV|ADD)\s/
-syntax match dockerfileKeyword /\v^\s*(ENTRYPOINT|VOLUME|USER|WORKDIR)\s/
+syntax match dockerfileKeyword /\v^\s*(ONBUILD\s+)?(ADD|CMD|ENTRYPOINT|ENV|EXPOSE|FROM|MAINTAINER|RUN|USER|VOLUME|WORKDIR)\s/
 highlight link dockerfileKeyword Keyword
 
 syntax region dockerfileString start=/\v"/ skip=/\v\\./ end=/\v"/

docker/docker.go

@@ -6,19 +6,16 @@ import (
 	"os"
 	"strings"
 
-	"github.com/dotcloud/docker"
+	_ "github.com/dotcloud/docker"
 	"github.com/dotcloud/docker/api"
+	"github.com/dotcloud/docker/dockerversion"
 	"github.com/dotcloud/docker/engine"
 	flag "github.com/dotcloud/docker/pkg/mflag"
+	"github.com/dotcloud/docker/pkg/opts"
 	"github.com/dotcloud/docker/sysinit"
 	"github.com/dotcloud/docker/utils"
 )
 
-var (
-	GITCOMMIT string
-	VERSION   string
-)
-
 func main() {
 	if selfPath := utils.SelfPath(); selfPath == "/sbin/init" || selfPath == "/.dockerinit" {
 		// Running in init mode
@@ -36,13 +33,13 @@ func main() {
 		pidfile              = flag.String([]string{"p", "-pidfile"}, "/var/run/docker.pid", "Path to use for daemon PID file")
 		flRoot               = flag.String([]string{"g", "-graph"}, "/var/lib/docker", "Path to use as the root of the docker runtime")
 		flEnableCors         = flag.Bool([]string{"#api-enable-cors", "-api-enable-cors"}, false, "Enable CORS headers in the remote API")
-		flDns                = docker.NewListOpts(docker.ValidateIp4Address)
+		flDns                = opts.NewListOpts(opts.ValidateIp4Address)
 		flEnableIptables     = flag.Bool([]string{"#iptables", "-iptables"}, true, "Disable docker's addition of iptables rules")
 		flEnableIpForward    = flag.Bool([]string{"#ip-forward", "-ip-forward"}, true, "Disable enabling of net.ipv4.ip_forward")
 		flDefaultIp          = flag.String([]string{"#ip", "-ip"}, "0.0.0.0", "Default IP address to use when binding container ports")
 		flInterContainerComm = flag.Bool([]string{"#icc", "-icc"}, true, "Enable inter-container communication")
 		flGraphDriver        = flag.String([]string{"s", "-storage-driver"}, "", "Force the docker runtime to use a specific storage driver")
-		flHosts              = docker.NewListOpts(docker.ValidateHost)
+		flHosts              = opts.NewListOpts(api.ValidateHost)
 		flMtu                = flag.Int([]string{"#mtu", "-mtu"}, 0, "Set the containers network MTU; if no value is provided: default to the default route MTU or 1500 if not default route is available")
 	)
 	flag.Var(&flDns, []string{"#dns", "-dns"}, "Force docker to use specific DNS servers")
@@ -61,6 +58,9 @@ func main() {
 			// If we do not have a host, default to unix socket
 			defaultHost = fmt.Sprintf("unix://%s", api.DEFAULTUNIXSOCKET)
 		}
+		if _, err := api.ValidateHost(defaultHost); err != nil {
+			log.Fatal(err)
+		}
 		flHosts.Set(defaultHost)
 	}
 
@@ -71,8 +71,6 @@ func main() {
 	if *flDebug {
 		os.Setenv("DEBUG", "1")
 	}
-	docker.GITCOMMIT = GITCOMMIT
-	docker.VERSION = VERSION
 	if *flDaemon {
 		if flag.NArg() != 0 {
 			flag.Usage()
@@ -83,28 +81,39 @@ func main() {
 		if err != nil {
 			log.Fatal(err)
 		}
-		// Load plugin: httpapi
-		job := eng.Job("initserver")
-		job.Setenv("Pidfile", *pidfile)
-		job.Setenv("Root", *flRoot)
-		job.SetenvBool("AutoRestart", *flAutoRestart)
-		job.SetenvList("Dns", flDns.GetAll())
-		job.SetenvBool("EnableIptables", *flEnableIptables)
-		job.SetenvBool("EnableIpForward", *flEnableIpForward)
-		job.Setenv("BridgeIface", *bridgeName)
-		job.Setenv("BridgeIP", *bridgeIp)
-		job.Setenv("DefaultIp", *flDefaultIp)
-		job.SetenvBool("InterContainerCommunication", *flInterContainerComm)
-		job.Setenv("GraphDriver", *flGraphDriver)
-		job.SetenvInt("Mtu", *flMtu)
-		if err := job.Run(); err != nil {
-			log.Fatal(err)
-		}
+		// load the daemon in the background so we can immediately start
+		// the http api so that connections don't fail while the daemon
+		// is booting
+		go func() {
+			// Load plugin: httpapi
+			job := eng.Job("initserver")
+			job.Setenv("Pidfile", *pidfile)
+			job.Setenv("Root", *flRoot)
+			job.SetenvBool("AutoRestart", *flAutoRestart)
+			job.SetenvList("Dns", flDns.GetAll())
+			job.SetenvBool("EnableIptables", *flEnableIptables)
+			job.SetenvBool("EnableIpForward", *flEnableIpForward)
+			job.Setenv("BridgeIface", *bridgeName)
+			job.Setenv("BridgeIP", *bridgeIp)
+			job.Setenv("DefaultIp", *flDefaultIp)
+			job.SetenvBool("InterContainerCommunication", *flInterContainerComm)
+			job.Setenv("GraphDriver", *flGraphDriver)
+			job.SetenvInt("Mtu", *flMtu)
+			if err := job.Run(); err != nil {
+				log.Fatal(err)
+			}
+			// after the daemon is done setting up we can tell the api to start
+			// accepting connections
+			if err := eng.Job("acceptconnections").Run(); err != nil {
+				log.Fatal(err)
+			}
+		}()
+
 		// Serve api
-		job = eng.Job("serveapi", flHosts.GetAll()...)
+		job := eng.Job("serveapi", flHosts.GetAll()...)
 		job.SetenvBool("Logging", true)
 		job.SetenvBool("EnableCors", *flEnableCors)
-		job.Setenv("Version", VERSION)
+		job.Setenv("Version", dockerversion.VERSION)
 		if err := job.Run(); err != nil {
 			log.Fatal(err)
 		}
@@ -113,7 +122,7 @@ func main() {
 			log.Fatal("Please specify only one -H")
 		}
 		protoAddrParts := strings.SplitN(flHosts.GetAll()[0], "://", 2)
-		if err := docker.ParseCommands(protoAddrParts[0], protoAddrParts[1], flag.Args()...); err != nil {
+		if err := api.ParseCommands(protoAddrParts[0], protoAddrParts[1], flag.Args()...); err != nil {
 			if sterr, ok := err.(*utils.StatusError); ok {
 				if sterr.Status != "" {
 					log.Println(sterr.Status)
@@ -126,5 +135,5 @@ func main() {
 }
 
 func showVersion() {
-	fmt.Printf("Docker version %s, build %s\n", VERSION, GITCOMMIT)
+	fmt.Printf("Docker version %s, build %s\n", dockerversion.VERSION, dockerversion.GITCOMMIT)
 }

dockerinit/dockerinit.go

@@ -4,11 +4,6 @@ import (
 	"github.com/dotcloud/docker/sysinit"
 )
 
-var (
-	GITCOMMIT string
-	VERSION   string
-)
-
 func main() {
 	// Running in init mode
 	sysinit.SysInit()

dockerversion/dockerversion.go

@@ -0,0 +1,15 @@
+package dockerversion
+
+// FIXME: this should be embedded in the docker/docker.go,
+// but we can't because distro policy requires us to
+// package a separate dockerinit binary, and that binary needs
+// to know its version too.
+
+var (
+	GITCOMMIT string
+	VERSION   string
+
+	IAMSTATIC bool   // whether or not Docker itself was compiled statically via ./hack/make.sh binary
+	INITSHA1  string // sha1sum of separate static dockerinit, if Docker itself was compiled dynamically via ./hack/make.sh dynbinary
+	INITPATH  string // custom location to search for a valid dockerinit binary (available for packagers as a last resort escape hatch)
+)

docs/sources/contributing/devenvironment.rst

@@ -24,7 +24,17 @@ a working, up-to-date docker installation, then continue to the next
 step.
 
 
-Step 2: Check out the Source
+Step 2: Install tools used for this tutorial
+--------------------------------------------
+
+Install ``git``; honest, it's very good. You can use other ways to get the Docker
+source, but they're not anywhere near as easy.
+
+Install ``make``. This tutorial uses our base Makefile to kick off the docker
+containers in a repeatable and consistent way. Again, you can do it in other ways
+but you need to do more work.
+
+Step 3: Check out the Source
 ----------------------------
 
 .. code-block:: bash
@@ -35,7 +45,7 @@ Step 2: Check out the Source
 To checkout a different revision just use ``git checkout`` with the name of branch or revision number.
 
 
-Step 3: Build the Environment
+Step 4: Build the Environment
 -----------------------------
 
 This following command will build a development environment using the Dockerfile in the current directory. Essentially, it will install all the build and runtime dependencies necessary to build and test Docker. This command will take some time to complete when you first execute it.
@@ -48,7 +58,7 @@ If the build is successful, congratulations! You have produced a clean build of
 docker, neatly encapsulated in a standard build environment. 
 
 
-Step 4: Build the Docker Binary
+Step 5: Build the Docker Binary
 -------------------------------
 
 To create the Docker binary, run this command:

docs/sources/examples/postgresql_service.Dockerfile

@@ -0,0 +1,53 @@
+#
+# example Dockerfile for http://docs.docker.io/en/latest/examples/postgresql_service/
+#
+
+FROM ubuntu
+MAINTAINER SvenDowideit@docker.com
+
+# Add the PostgreSQL PGP key to verify their Debian packages.
+# It should be the same key as https://www.postgresql.org/media/keys/ACCC4CF8.asc 
+RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8
+
+# Add PostgreSQL's repository. It contains the most recent stable release
+#     of PostgreSQL, ``9.3``.
+RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main" > /etc/apt/sources.list.d/pgdg.list
+
+# Update the Ubuntu and PostgreSQL repository indexes
+RUN apt-get update
+
+# Install ``python-software-properties``, ``software-properties-common`` and PostgreSQL 9.3
+#  There are some warnings (in red) that show up during the build. You can hide
+#  them by prefixing each apt-get statement with DEBIAN_FRONTEND=noninteractive
+RUN apt-get -y -q install python-software-properties software-properties-common
+RUN apt-get -y -q install postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3
+
+# Note: The official Debian and Ubuntu images automatically ``apt-get clean``
+# after each ``apt-get`` 
+
+# Run the rest of the commands as the ``postgres`` user created by the ``postgres-9.3`` package when it was ``apt-get installed``
+USER postgres
+
+# Create a PostgreSQL role named ``docker`` with ``docker`` as the password and
+# then create a database `docker` owned by the ``docker`` role.
+# Note: here we use ``&&\`` to run commands one after the other - the ``\``
+#       allows the RUN command to span multiple lines.
+RUN    /etc/init.d/postgresql start &&\
+    psql --command "CREATE USER docker WITH SUPERUSER PASSWORD 'docker';" &&\
+    createdb -O docker docker
+
+# Adjust PostgreSQL configuration so that remote connections to the
+# database are possible. 
+RUN echo "host all  all    0.0.0.0/0  md5" >> /etc/postgresql/9.3/main/pg_hba.conf
+
+# And add ``listen_addresses`` to ``/etc/postgresql/9.3/main/postgresql.conf``
+RUN echo "listen_addresses='*'" >> /etc/postgresql/9.3/main/postgresql.conf
+
+# Expose the PostgreSQL port
+EXPOSE 5432
+
+# Add VOLUMEs to allow backup of config, logs and databases
+VOLUME	["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
+
+# Set the default command to run when starting the container
+CMD ["/usr/lib/postgresql/9.3/bin/postgres", "-D", "/var/lib/postgresql/9.3/main", "-c", "config_file=/etc/postgresql/9.3/main/postgresql.conf"]

docs/sources/examples/postgresql_service.rst

@@ -9,152 +9,109 @@ PostgreSQL Service
 
 .. include:: example_header.inc
 
-.. note::
-
-    A shorter version of `this blog post`_.
-
-.. _this blog post: http://zaiste.net/2013/08/docker_postgresql_how_to/
-
 Installing PostgreSQL on Docker
 -------------------------------
 
-Run an interactive shell in a Docker container.
+Assuming there is no Docker image that suits your needs in `the index`_, you 
+can create one yourself.
 
-.. code-block:: bash
+.. _the index: http://index.docker.io
 
-    sudo docker run -i -t ubuntu /bin/bash
-
-Update its dependencies.
-
-.. code-block:: bash
-
-    apt-get update
-
-Install ``python-software-properties``, ``software-properties-common``, ``wget`` and ``vim``.
-
-.. code-block:: bash
-
-    apt-get -y install python-software-properties software-properties-common wget vim
-
-Add PostgreSQL's repository. It contains the most recent stable release
-of PostgreSQL, ``9.3``.
-
-.. code-block:: bash
-
-    wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
-    echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main" > /etc/apt/sources.list.d/pgdg.list
-    apt-get update
-
-Finally, install PostgreSQL 9.3
-
-.. code-block:: bash
-
-    apt-get -y install postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3
-
-Now, create a PostgreSQL superuser role that can create databases and
-other roles.  Following Vagrant's convention the role will be named
-``docker`` with ``docker`` password assigned to it.
-
-.. code-block:: bash
-
-    su postgres -c "createuser -P -d -r -s docker"
-
-Create a test database also named ``docker`` owned by previously created ``docker``
-role.
-
-.. code-block:: bash
-
-    su postgres -c "createdb -O docker docker"
-
-Adjust PostgreSQL configuration so that remote connections to the
-database are possible. Make sure that inside
-``/etc/postgresql/9.3/main/pg_hba.conf`` you have following line:
-
-.. code-block:: bash
-
-    host    all             all             0.0.0.0/0               md5
-
-Additionaly, inside ``/etc/postgresql/9.3/main/postgresql.conf``
-uncomment ``listen_addresses`` like so:
-
-.. code-block:: bash
-
-    listen_addresses='*'
+Start by creating a new Dockerfile:
 
 .. note::
 
     This PostgreSQL setup is for development only purposes. Refer
-    to PostgreSQL documentation how to fine-tune these settings so that it
-    is secure enough.
+    to the PostgreSQL documentation to fine-tune these settings so that it
+    is suitably secure.
 
-Exit.
+.. literalinclude:: postgresql_service.Dockerfile
+
+Build an image from the Dockerfile assign it a name.
 
 .. code-block:: bash
 
-    exit
+    $ sudo docker build -t eg_postgresql .
 
-Create an image from our container and assign it a name. The ``<container_id>``
-is in the Bash prompt; you can also locate it using ``docker ps -a``.
+And run the PostgreSQL server container (in the foreground):
 
 .. code-block:: bash
 
-    sudo docker commit <container_id> <your username>/postgresql
+    $ sudo docker run -rm -P -name pg_test eg_postgresql
 
-Finally, run the PostgreSQL server via ``docker``.
+There are  2 ways to connect to the PostgreSQL server. We can use 
+:ref:`working_with_links_names`, or we can access it from our host (or the network).
+
+.. note:: The ``-rm`` removes the container and its image when the container 
+          exists successfully.
+
+Using container linking
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Containers can be linked to another container's ports directly using 
+``-link remote_name:local_alias`` in the client's ``docker run``. This will
+set a number of environment variables that can then be used to connect:
 
 .. code-block:: bash
 
-    CONTAINER=$(sudo docker run -d -p 5432 \
-      -t <your username>/postgresql \
-      /bin/su postgres -c '/usr/lib/postgresql/9.3/bin/postgres \
-        -D /var/lib/postgresql/9.3/main \
-        -c config_file=/etc/postgresql/9.3/main/postgresql.conf')
+    $ sudo docker run -rm -t -i -link pg_test:pg eg_postgresql bash
 
-Connect the PostgreSQL server using ``psql`` (You will need the
-postgresql client installed on the machine.  For ubuntu, use something
-like ``sudo apt-get install postgresql-client``).
+    postgres@7ef98b1b7243:/$ psql -h $PG_PORT_5432_TCP_ADDR -p $PG_PORT_5432_TCP_PORT -d docker -U docker --password
+
+Connecting from your host system
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Assuming you have the postgresql-client installed, you can use the host-mapped port
+to test as well. You need to use ``docker ps`` to find out what local host port the 
+container is mapped to first:
 
 .. code-block:: bash
 
-    CONTAINER_IP=$(sudo docker inspect -format='{{.NetworkSettings.IPAddress}}' $CONTAINER)
-    psql -h $CONTAINER_IP -p 5432 -d docker -U docker -W
+    $ docker ps
+    CONTAINER ID        IMAGE                  COMMAND                CREATED             STATUS              PORTS                                      NAMES
+    5e24362f27f6        eg_postgresql:latest   /usr/lib/postgresql/   About an hour ago   Up About an hour    0.0.0.0:49153->5432/tcp                    pg_test
+    $ psql -h localhost -p 49153 -d docker -U docker --password
 
-As before, create roles or databases if needed.
+Testing the database
+^^^^^^^^^^^^^^^^^^^^
+
+Once you have authenticated and have a ``docker =#`` prompt, you can
+create a table and populate it.
 
 .. code-block:: bash
 
     psql (9.3.1)
     Type "help" for help.
 
-    docker=# CREATE DATABASE foo OWNER=docker;
-    CREATE DATABASE
+    docker=# CREATE TABLE cities (
+    docker(#     name            varchar(80),
+    docker(#     location        point
+    docker(# );
+    CREATE TABLE
+    docker=# INSERT INTO cities VALUES ('San Francisco', '(-194.0, 53.0)');
+    INSERT 0 1
+    docker=# select * from cities;
+         name      | location  
+    ---------------+-----------
+     San Francisco | (-194,53)
+    (1 row)
 
-Additionally, publish your newly created image on the Docker Index.
+Using the container volumes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can use the defined volumes to inspect the PostgreSQL log files and to backup your
+configuration and data:
 
 .. code-block:: bash
 
-    sudo docker login
-    Username: <your username>
-    [...]
+    docker run -rm --volumes-from pg_test -t -i busybox sh
 
-.. code-block:: bash
+    / # ls
+    bin      etc      lib      linuxrc  mnt      proc     run      sys      usr
+    dev      home     lib64    media    opt      root     sbin     tmp      var
+    / # ls /etc/postgresql/9.3/main/
+    environment      pg_hba.conf      postgresql.conf
+    pg_ctl.conf      pg_ident.conf    start.conf
+    /tmp # ls /var/log
+    ldconfig    postgresql
 
-    sudo docker push <your username>/postgresql
-
-PostgreSQL service auto-launch
-------------------------------
-
-Running our image seems complicated. We have to specify the whole command with
-``docker run``. Let's simplify it so the service starts automatically when the
-container starts.
-
-.. code-block:: bash
-
-    sudo docker commit -run='{"Cmd": \
-      ["/bin/su", "postgres", "-c", "/usr/lib/postgresql/9.3/bin/postgres -D \
-      /var/lib/postgresql/9.3/main -c \
-      config_file=/etc/postgresql/9.3/main/postgresql.conf"], "PortSpecs": ["5432"]}' \
-      <container_id> <your username>/postgresql
-
-From now on, just type ``docker run <your username>/postgresql`` and
-PostgreSQL should automatically start.

docs/sources/examples/using_supervisord.rst

@@ -112,7 +112,7 @@ Once we've got a built image we can launch a container from it.
 
 .. code-block:: bash
 
-    sudo docker run -p 22 -p 80 -t -i <yourname>/supervisor
+    sudo docker run -p 22 -p 80 -t -i <yourname>/supervisord
     2013-11-25 18:53:22,312 CRIT Supervisor running as root (no user in config file)
     2013-11-25 18:53:22,312 WARN Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
     2013-11-25 18:53:22,342 INFO supervisord started with pid 1

docs/sources/faq.rst

@@ -175,6 +175,7 @@ Linux:
 - Gentoo
 - ArchLinux
 - openSUSE 12.3+
+- CRUX 3.0+
 
 Cloud:
 
@@ -182,6 +183,12 @@ Cloud:
 - Google Compute Engine
 - Rackspace
 
+How do I report a security issue with Docker?
+.............................................
+
+You can learn about the project's security policy `here <http://www.docker.io/security/>`_
+and report security issues to this `mailbox <mailto:security@docker.com>`_.
+
 Can I help by adding some questions and answers?
 ................................................
 

docs/sources/installation/amazon.rst

@@ -1,5 +1,5 @@
 :title: Installation on Amazon EC2
-:description: Docker installation on Amazon EC2 
+:description: Please note this project is currently under heavy development. It should not be used in production. 
 :keywords: amazon ec2, virtualization, cloud, docker, documentation, installation
 
 Amazon EC2

docs/sources/installation/archlinux.rst

@@ -1,5 +1,5 @@
 :title: Installation on Arch Linux
-:description: Docker installation on Arch Linux.
+:description: Please note this project is currently under heavy development. It should not be used in production.
 :keywords: arch linux, virtualization, docker, documentation, installation
 
 .. _arch_linux:

docs/sources/installation/cruxlinux.rst

@@ -0,0 +1,98 @@
+:title: Installation on CRUX Linux
+:description: Docker installation on CRUX Linux.
+:keywords: crux linux, virtualization, Docker, documentation, installation
+
+.. _crux_linux:
+
+
+CRUX Linux
+==========
+
+.. include:: install_header.inc
+
+.. include:: install_unofficial.inc
+
+Installing on CRUX Linux can be handled via the ports from `James Mills <http://prologic.shortcircuit.net.au/>`_:
+
+* `docker <https://bitbucket.org/prologic/ports/src/tip/docker/>`_
+
+* `docker-bin <https://bitbucket.org/prologic/ports/src/tip/docker-bin/>`_
+
+* `docker-git <https://bitbucket.org/prologic/ports/src/tip/docker-git/>`_
+
+The ``docker`` port will install the latest tagged version of Docker. 
+The ``docker-bin`` port will install the latest tagged versin of Docker from upstream built binaries.
+The ``docker-git`` package will build from the current master branch.
+
+
+Installation
+------------
+
+For the time being (*until the CRUX Docker port(s) get into the official contrib repository*) you will need to install
+`James Mills' <https://bitbucket.org/prologic/ports>`_ ports repository. You can do so via:
+
+Download the ``httpup`` file to ``/etc/ports/``:
+::
+    
+    curl -q -o - http://crux.nu/portdb/?a=getup&q=prologic > /etc/ports/prologic.httpup
+    
+
+Add ``prtdir /usr/ports/prologic`` to ``/etc/prt-get.conf``:
+::
+    
+    vim /etc/prt-get.conf
+
+    # or:
+    echo "prtdir /usr/ports/prologic" >> /etc/prt-get.conf
+    
+
+Update ports and prt-get cache:
+::
+    
+    ports -u
+    prt-get cache
+        
+
+To install (*and its dependencies*):
+::
+
+    prt-get depinst docker
+    
+
+Use ``docker-bin`` for the upstream binary or ``docker-git`` to build and install from the master branch from git.
+
+
+Kernel Requirements
+-------------------
+
+To have a working **CRUX+Docker** Host you must ensure your Kernel
+has the necessary modules enabled for LXC containers to function
+correctly and Docker Daemon to work properly.
+
+Please read the ``README.rst``:
+::
+    
+    prt-get readme docker
+    
+There is a ``test_kernel_config.sh`` script in the above ports which you can use to test your Kernel configuration:
+
+::
+    
+    cd /usr/ports/prologic/docker
+    ./test_kernel_config.sh /usr/src/linux/.config
+    
+
+Starting Docker
+---------------
+
+There is a rc script created for Docker. To start the Docker service:
+
+::
+    
+    sudo su -
+    /etc/rc.d/docker start
+    
+To start on system boot:
+
+- Edit ``/etc/rc.conf``
+- Put ``docker`` into the ``SERVICES=(...)`` array after ``net``.

docs/sources/installation/fedora.rst

@@ -1,4 +1,4 @@
-:title: Requirements and Installation on Fedora
+:title: Installation on Fedora
 :description: Please note this project is currently under heavy development. It should not be used in production.
 :keywords: Docker, Docker documentation, Fedora, requirements, virtualbox, vagrant, git, ssh, putty, cygwin, linux
 

docs/sources/installation/frugalware.rst

@@ -1,5 +1,5 @@
 :title: Installation on FrugalWare
-:description: Docker installation on FrugalWare.
+:description: Please note this project is currently under heavy development. It should not be used in production.
 :keywords: frugalware linux, virtualization, docker, documentation, installation
 
 .. _frugalware:

docs/sources/installation/gentoolinux.rst

@@ -1,5 +1,5 @@
-:title: Installation on Gentoo Linux
-:description: Docker installation instructions and nuances for Gentoo Linux.
+:title: Installation on Gentoo
+:description: Please note this project is currently under heavy development. It should not be used in production.
 :keywords: gentoo linux, virtualization, docker, documentation, installation
 
 .. _gentoo_linux:

docs/sources/installation/google.rst

@@ -50,18 +50,9 @@
     docker-playground:~$ curl get.docker.io | bash
     docker-playground:~$ sudo update-rc.d docker defaults
 
-6. If running in zones: ``us-central1-a``, ``europe-west1-1``, and ``europe-west1-b``, the docker daemon must be started with the ``-mtu`` flag. Without the flag, you may experience intermittent network pauses. 
-`See this issue <https://code.google.com/p/google-compute-engine/issues/detail?id=57>`_ for more details.
-
-.. code-block:: bash
-
-    docker-playground:~$ echo 'DOCKER_OPTS="$DOCKER_OPTS -mtu 1460"' | sudo tee -a /etc/default/docker
-    docker-playground:~$ sudo service docker restart
-
-7. Start a new container:
+6. Start a new container:
 
 .. code-block:: bash
 
     docker-playground:~$ sudo docker run busybox echo 'docker on GCE \o/'
     docker on GCE \o/
-

docs/sources/installation/index.rst

@@ -21,6 +21,7 @@ Contents:
    rhel
    fedora
    archlinux
+   cruxlinux
    gentoolinux
    openSUSE
    frugalware

docs/sources/installation/mac.rst

@@ -1,4 +1,4 @@
-:title: Requirements and Installation on Mac OS X 10.6 Snow Leopard
+:title: Installation on Mac OS X 10.6 Snow Leopard
 :description: Please note this project is currently under heavy development. It should not be used in production.
 :keywords: Docker, Docker documentation, requirements, virtualbox, ssh, linux, os x, osx, mac
 
@@ -49,10 +49,10 @@ Run the following commands to get boot2docker:
 
     # Enter the installation directory
     cd ~/bin
-    
+
     # Get the file
     curl https://raw.github.com/steeve/boot2docker/master/boot2docker > boot2docker
-    
+
     # Mark it executable
     chmod +x boot2docker
 
@@ -66,14 +66,14 @@ Run the following commands to get it downloaded and set up:
 .. code-block:: bash
 
     # Get the file
-    curl -o docker http://get.docker.io/builds/Darwin/x86_64/docker-latest
-    
+    curl -o docker https://get.docker.io/builds/Darwin/x86_64/docker-latest
+
     # Mark it executable
     chmod +x docker
 
     # Set the environment variable for the docker daemon
-    export DOCKER_HOST=tcp://
-    
+    export DOCKER_HOST=tcp://127.0.0.1:4243
+
     # Copy the executable file
     sudo cp docker /usr/local/bin/
 
@@ -94,7 +94,7 @@ Inside the ``~/bin`` directory, run the following commands:
 
     # Run the VM (the docker daemon)
     ./boot2docker up
-    
+
     # To see all available commands:
     ./boot2docker
 
@@ -116,6 +116,21 @@ client just like any other application.
     # Git commit (server): c348c04
     # Go version (server): go1.2
 
+Forwarding VM Port Range to Host
+--------------------------------
+
+If we take the port range that docker uses by default with the -P option
+(49000-49900), and forward same range from host to vm, we'll be able to interact
+with our containers as if they were running locally:
+
+.. code-block:: bash
+
+   # vm must be powered off
+   for i in {4900..49900}; do
+    VBoxManage modifyvm "boot2docker-vm" --natpf1 "tcp-port$i,tcp,,$i,,$i";
+    VBoxManage modifyvm "boot2docker-vm" --natpf1 "udp-port$i,udp,,$i,,$i";
+   done
+
 SSH-ing The VM
 --------------
 
@@ -147,6 +162,18 @@ If SSH complains about keys:
 
     ssh-keygen -R '[localhost]:2022'
 
+Upgrading to a newer release of boot2docker
+-------------------------------------------
+
+To upgrade an initialised VM, you can use the following 3 commands. Your persistence
+disk will not be changed, so you won't lose your images and containers:
+
+.. code-block:: bash
+
+    ./boot2docker stop
+    ./boot2docker download
+    ./boot2docker start
+
 About the way Docker works on Mac OS X:
 ---------------------------------------
 

docs/sources/installation/openSUSE.rst

@@ -1,5 +1,5 @@
 :title: Installation on openSUSE
-:description: Docker installation on openSUSE.
+:description: Please note this project is currently under heavy development. It should not be used in production.
 :keywords: openSUSE, virtualbox, docker, documentation, installation
 
 .. _openSUSE:

docs/sources/installation/rackspace.rst

@@ -1,5 +1,5 @@
-:title: Rackspace Cloud Installation
-:description: Installing Docker on Ubuntu proviced by Rackspace
+:title: Installation on Rackspace Cloud
+:description: Please note this project is currently under heavy development. It should not be used in production.
 :keywords: Rackspace Cloud, installation, docker, linux, ubuntu
 
 Rackspace Cloud

docs/sources/installation/rhel.rst

@@ -1,4 +1,4 @@
-:title: Requirements and Installation on Red Hat Enterprise Linux
+:title: Installation on Red Hat Enterprise Linux
 :description: Please note this project is currently under heavy development. It should not be used in production.
 :keywords: Docker, Docker documentation, requirements, linux, rhel, centos
 

docs/sources/installation/ubuntulinux.rst

@@ -1,4 +1,4 @@
-:title: Requirements and Installation on Ubuntu Linux
+:title: Installation on Ubuntu
 :description: Please note this project is currently under heavy development. It should not be used in production.
 :keywords: Docker, Docker documentation, requirements, virtualbox, vagrant, git, ssh, putty, cygwin, linux
 

docs/sources/installation/windows.rst

@@ -1,11 +1,11 @@
-:title: Requirements and Installation on Windows
-:description: Docker's tutorial to run docker on Windows
+:title: Installation on Windows
+:description: Please note this project is currently under heavy development. It should not be used in production.
 :keywords: Docker, Docker documentation, Windows, requirements, virtualbox, vagrant, git, ssh, putty, cygwin
 
 .. _windows:
 
-Installing Docker on Windows
-============================
+Windows
+=======
 
 Docker can run on Windows using a VM like VirtualBox. You then run
 Linux within the VM.

docs/sources/reference/api/docker_remote_api_v1.0.rst

@@ -732,11 +732,11 @@ Tag an image into a repository
 
         .. sourcecode:: http
 
-           HTTP/1.1 200 OK
+           HTTP/1.1 201 OK
 
 	:query repo: The repository to tag in
 	:query force: 1/True/true or 0/False/false, default false
-	:statuscode 200: no error
+	:statuscode 201: no error
 	:statuscode 400: bad parameter
 	:statuscode 404: no such image
         :statuscode 500: server error

docs/sources/reference/api/docker_remote_api_v1.1.rst

@@ -742,11 +742,11 @@ Tag an image into a repository
 
         .. sourcecode:: http
 
-           HTTP/1.1 200 OK
+           HTTP/1.1 201 OK
 
 	:query repo: The repository to tag in
 	:query force: 1/True/true or 0/False/false, default false
-	:statuscode 200: no error
+	:statuscode 201: no error
 	:statuscode 400: bad parameter
 	:statuscode 404: no such image
 	:statuscode 409: conflict

docs/sources/reference/api/docker_remote_api_v1.2.rst

@@ -761,11 +761,11 @@ Tag an image into a repository
 
         .. sourcecode:: http
 
-           HTTP/1.1 200 OK
+           HTTP/1.1 201 OK
 
 	:query repo: The repository to tag in
 	:query force: 1/True/true or 0/False/false, default false
-	:statuscode 200: no error
+	:statuscode 201: no error
 	:statuscode 400: bad parameter
 	:statuscode 404: no such image
 	:statuscode 409: conflict

docs/sources/reference/api/docker_remote_api_v1.3.rst

@@ -808,11 +808,11 @@ Tag an image into a repository
 
         .. sourcecode:: http
 
-           HTTP/1.1 200 OK
+           HTTP/1.1 201 OK
 
 	:query repo: The repository to tag in
 	:query force: 1/True/true or 0/False/false, default false
-	:statuscode 200: no error
+	:statuscode 201: no error
 	:statuscode 400: bad parameter
 	:statuscode 404: no such image
 	:statuscode 409: conflict

docs/sources/reference/api/docker_remote_api_v1.4.rst

@@ -852,11 +852,11 @@ Tag an image into a repository
 
         .. sourcecode:: http
 
-           HTTP/1.1 200 OK
+           HTTP/1.1 201 OK
 
 	:query repo: The repository to tag in
 	:query force: 1/True/true or 0/False/false, default false
-	:statuscode 200: no error
+	:statuscode 201: no error
 	:statuscode 400: bad parameter
 	:statuscode 404: no such image
 	:statuscode 409: conflict

docs/sources/reference/api/docker_remote_api_v1.5.rst

@@ -831,11 +831,11 @@ Tag an image into a repository
 
   .. sourcecode:: http
 
-    HTTP/1.1 200 OK
+    HTTP/1.1 201 OK
 
   :query repo: The repository to tag in
   :query force: 1/True/true or 0/False/false, default false
-  :statuscode 200: no error
+  :statuscode 201: no error
   :statuscode 400: bad parameter
   :statuscode 404: no such image
   :statuscode 409: conflict

docs/sources/reference/api/docker_remote_api_v1.6.rst

@@ -958,11 +958,11 @@ Tag an image into a repository
 
         .. sourcecode:: http
 
-           HTTP/1.1 200 OK
+           HTTP/1.1 201 OK
 
 	:query repo: The repository to tag in
 	:query force: 1/True/true or 0/False/false, default false
-	:statuscode 200: no error
+	:statuscode 201: no error
 	:statuscode 400: bad parameter
 	:statuscode 404: no such image
 	:statuscode 409: conflict

docs/sources/reference/api/docker_remote_api_v1.7.rst

@@ -877,11 +877,11 @@ Tag an image into a repository
 
         .. sourcecode:: http
 
-           HTTP/1.1 200 OK
+           HTTP/1.1 201 OK
 
 	:query repo: The repository to tag in
 	:query force: 1/True/true or 0/False/false, default false
-	:statuscode 200: no error
+	:statuscode 201: no error
 	:statuscode 400: bad parameter
 	:statuscode 404: no such image
 	:statuscode 409: conflict

docs/sources/reference/api/docker_remote_api_v1.8.rst

@@ -892,11 +892,11 @@ Tag an image into a repository
 
         .. sourcecode:: http
 
-           HTTP/1.1 200 OK
+           HTTP/1.1 201 OK
 
         :query repo: The repository to tag in
         :query force: 1/True/true or 0/False/false, default false
-        :statuscode 200: no error
+        :statuscode 201: no error
         :statuscode 400: bad parameter
         :statuscode 404: no such image
         :statuscode 409: conflict

docs/sources/reference/api/docker_remote_api_v1.9.rst

@@ -892,11 +892,11 @@ Tag an image into a repository
 
         .. sourcecode:: http
 
-           HTTP/1.1 200 OK
+           HTTP/1.1 201 OK
 
         :query repo: The repository to tag in
         :query force: 1/True/true or 0/False/false, default false
-        :statuscode 200: no error
+        :statuscode 201: no error
         :statuscode 400: bad parameter
         :statuscode 404: no such image
         :statuscode 409: conflict

docs/sources/reference/api/remote_api_client_libraries.rst

@@ -1,6 +1,6 @@
 :title: Remote API Client Libraries
 :description: Various client libraries available to use with the Docker remote API
-:keywords: API, Docker, index, registry, REST, documentation, clients, Python, Ruby, Javascript, Erlang, Go
+:keywords: API, Docker, index, registry, REST, documentation, clients, Python, Ruby, JavaScript, Erlang, Go
 
 
 ==================================
@@ -21,12 +21,18 @@ and we will add the libraries here.
 +----------------------+----------------+--------------------------------------------+----------+
 | Ruby                 | docker-api     | https://github.com/swipely/docker-api      | Active   |
 +----------------------+----------------+--------------------------------------------+----------+
-| Javascript (NodeJS)  | docker.io      | https://github.com/appersonlabs/docker.io  | Active   |
+| JavaScript (NodeJS)  | dockerode      | https://github.com/apocas/dockerode        | Active   |
+|                      |                | Install via NPM: `npm install dockerode`   |          |
++----------------------+----------------+--------------------------------------------+----------+
+| JavaScript (NodeJS)  | docker.io      | https://github.com/appersonlabs/docker.io  | Active   |
 |                      |                | Install via NPM: `npm install docker.io`   |          |
 +----------------------+----------------+--------------------------------------------+----------+
-| Javascript           | docker-js      | https://github.com/dgoujard/docker-js      | Active   |
+| JavaScript           | docker-js      | https://github.com/dgoujard/docker-js      | Active   |
 +----------------------+----------------+--------------------------------------------+----------+
-| Javascript (Angular) | dockerui       | https://github.com/crosbymichael/dockerui  | Active   |
+| JavaScript (Angular) | docker-cp      | https://github.com/13W/docker-cp           | Active   |
+| **WebUI**            |                |                                            |          |
++----------------------+----------------+--------------------------------------------+----------+
+| JavaScript (Angular) | dockerui       | https://github.com/crosbymichael/dockerui  | Active   |
 | **WebUI**            |                |                                            |          |
 +----------------------+----------------+--------------------------------------------+----------+
 | Java                 | docker-java    | https://github.com/kpelykh/docker-java     | Active   |

docs/sources/reference/builder.rst

@@ -251,9 +251,14 @@ value ``<value>``. This value will be passed to all future ``RUN``
 instructions. This is functionally equivalent to prefixing the command
 with ``<key>=<value>``
 
+The environment variables set using ``ENV`` will persist when a container is run
+from the resulting image. You can view the values using ``docker inspect``, and change them using ``docker run --env <key>=<value>``.
+
 .. note::
-    The environment variables will persist when a container is run
-    from the resulting image.
+    One example where this can cause unexpected consequenses, is setting 
+    ``ENV DEBIAN_FRONTEND noninteractive``.
+    Which will persist when the container is run interactively; for example: 
+    ``docker run -t -i image bash``
 
 .. _dockerfile_add:
 
@@ -269,7 +274,7 @@ the container's filesystem at path ``<dest>``.
 source directory being built (also called the *context* of the build) or
 a remote file URL.
 
-``<dest>`` is the path at which the source will be copied in the
+``<dest>`` is the absolute path to which the source will be copied inside the
 destination container.
 
 All new files and directories are created with mode 0755, uid and gid
@@ -399,8 +404,10 @@ the image.
 
     ``WORKDIR /path/to/workdir``
 
-The ``WORKDIR`` instruction sets the working directory in which
-the command given by ``CMD`` is executed.
+The ``WORKDIR`` instruction sets the working directory for the ``RUN``, ``CMD`` and
+``ENTRYPOINT``  Dockerfile commands that follow it.
+
+It can be used multiple times in the one Dockerfile.
 
 3.11 ONBUILD
 ------------

docs/sources/reference/commandline/cli.rst

@@ -12,7 +12,7 @@ To list available commands, either run ``docker`` with no parameters or execute
 
   $ sudo docker
     Usage: docker [OPTIONS] COMMAND [arg...]
-      -H=[unix:///var/run/docker.sock]: tcp://[host[:port]] to bind/connect to or unix://[/path/to/socket] to use. When host=[0.0.0.0], port=[4243] or path=[/var/run/docker.sock] is omitted, default values are used.
+      -H=[unix:///var/run/docker.sock]: tcp://[host]:port to bind/connect to or unix://[/path/to/socket] to use. When host=[127.0.0.1] is omitted for tcp or path=[/var/run/docker.sock] is omitted for unix sockets, default values are used.
 
     A self-sufficient runtime for linux containers.
 
@@ -102,12 +102,17 @@ the ``-H`` flag for the client.
         docker ps
         # both are equal
 
-
 To run the daemon with `systemd socket activation <http://0pointer.de/blog/projects/socket-activation.html>`_, use ``docker -d -H fd://``.
 Using ``fd://`` will work perfectly for most setups but you can also specify individual sockets too ``docker -d -H fd://3``.
 If the specified socket activated files aren't found then docker will exit.
 You can find examples of using systemd socket activation with docker and systemd in the `docker source tree <https://github.com/dotcloud/docker/blob/master/contrib/init/systemd/socket-activation/>`_.
 
+.. warning::
+  Docker and LXC do not support the use of softlinks for either the Docker data directory (``/var/lib/docker``) or for ``/tmp``.
+  If your system is likely to be set up in that way, you can use ``readlink -f`` to canonicalise the links:
+
+  ``TMPDIR=$(readlink -f /tmp) /usr/local/bin/docker -d -D -g $(readlink -f /var/lib/docker) -H unix:// $EXPOSE_ALL > /var/lib/boot2docker/docker.log 2>&1``
+
 .. _cli_attach:
 
 ``attach``
@@ -181,7 +186,7 @@ Examples:
     Build a new container image from the source code at PATH
       -t, --time="": Repository name (and optionally a tag) to be applied 
              to the resulting image in case of success.
-      -q, --quiet=false: Suppress verbose build output.
+      -q, --quiet=false: Suppress the verbose output generated by the containers.
       --no-cache: Do not use the cache when building the image.
       --rm: Remove intermediate containers after a successful build
 
@@ -189,7 +194,8 @@ The files at ``PATH`` or ``URL`` are called the "context" of the build. The
 build process may refer to any of the files in the context, for example when
 using an :ref:`ADD <dockerfile_add>` instruction.  When a single ``Dockerfile``
 is given as ``URL``, then no context is set.  When a Git repository is set as
-``URL``, then the repository is used as the context
+``URL``, then the repository is used as the context. Git repositories are
+cloned with their submodules (`git clone --recursive`).
 
 .. _cli_build_examples:
 
@@ -1083,6 +1089,10 @@ is, ``docker run`` is equivalent to the API ``/containers/create`` then
 The ``docker run`` command can be used in combination with ``docker commit`` to
 :ref:`change the command that a container runs <cli_commit_examples>`.
 
+See :ref:`port_redirection` for more detailed information about the ``--expose``, 
+``-p``, ``-P`` and ``--link`` parameters, and :ref:`working_with_links_names` for 
+specific examples using ``--link``.
+
 Known Issues (run -volumes-from)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/sources/reference/run.rst

@@ -143,6 +143,7 @@ Network Settings
 ----------------
 
 ::
+
    -n=true   : Enable networking for this container
    -dns=[]   : Set custom dns servers for the container
 

docs/sources/use/basics.rst

@@ -59,10 +59,10 @@ Bind Docker to another host/port or a Unix socket
 
 .. warning:: Changing the default ``docker`` daemon binding to a TCP
    port or Unix *docker* user group will increase your security risks
-   by allowing non-root users to potentially gain *root* access on the
-   host (`e.g. #1369
-   <https://github.com/dotcloud/docker/issues/1369>`_). Make sure you
-   control access to ``docker``.
+   by allowing non-root users to gain *root* access on the
+   host. Make sure you control access to ``docker``. If you are binding 
+   to a TCP port, anyone with access to that port has full Docker access;
+   so it is not advisable on an open network.
 
 With ``-H`` it is possible to make the Docker daemon to listen on a
 specific IP and port. By default, it will listen on

docs/sources/use/port_redirection.rst

@@ -31,6 +31,15 @@ container, Docker provide ways to bind the container port to an
 interface of the host system. To simplify communication between
 containers, Docker provides the linking mechanism.
 
+Auto map all exposed ports on the host
+--------------------------------------
+
+To bind all the exposed container ports to the host automatically, use 
+``docker run -P <imageid>``.  The mapped host ports will be auto-selected 
+from a pool of unused ports (49000..49900), and you will need to use 
+``docker ps``, ``docker inspect <container_id>``  or 
+``docker port <container_id> <port>`` to determine what they are.
+
 Binding a port to a host interface
 -----------------------------------
 

docs/sources/use/working_with_volumes.rst

@@ -101,13 +101,23 @@ might not work on any other machine.
 
 For example::
 
-  sudo docker run -v /var/logs:/var/host_logs:ro ubuntu bash
+  sudo docker run -t -i -v /var/logs:/var/host_logs:ro ubuntu bash
 
 The command above mounts the host directory ``/var/logs`` into the
 container with read only permissions as ``/var/host_logs``.
 
 .. versionadded:: v0.5.0
 
+
+Note for OS/X users and remote daemon users:
+--------------------------------------------
+ 
+OS/X users run ``boot2docker`` to create a minimalist virtual machine running the docker daemon.  That 
+virtual machine then launches docker commands on behalf of the OS/X command line.   The means that ``host
+directories`` refer to directories in the ``boot2docker`` virtual machine, not the OS/X filesystem.
+
+Similarly, anytime when the docker daemon is on a remote machine, the ``host directories`` always refer to directories on the daemon's machine.
+
 Known Issues
 ............
 

engine/engine_test.go

@@ -4,6 +4,7 @@ import (
 	"io/ioutil"
 	"os"
 	"path"
+	"path/filepath"
 	"testing"
 )
 
@@ -64,6 +65,18 @@ func TestEngineRoot(t *testing.T) {
 		t.Fatal(err)
 	}
 	defer os.RemoveAll(tmp)
+	// We expect Root to resolve to an absolute path.
+	// FIXME: this should not be necessary.
+	// Until the above FIXME is implemented, let's check for the
+	// current behavior.
+	tmp, err = filepath.EvalSymlinks(tmp)
+	if err != nil {
+		t.Fatal(err)
+	}
+	tmp, err = filepath.Abs(tmp)
+	if err != nil {
+		t.Fatal(err)
+	}
 	dir := path.Join(tmp, "dir")
 	eng, err := New(dir)
 	if err != nil {

execdriver/driver.go

@@ -99,6 +99,8 @@ type Command struct {
 	Network    *Network   `json:"network"` // if network is nil then networking is disabled
 	Config     []string   `json:"config"`  //  generic values that specific drivers can consume
 	Resources  *Resources `json:"resources"`
+
+	Console string `json:"-"`
 }
 
 // Return the pid of the process

execdriver/lxc/driver.go

@@ -279,7 +279,8 @@ func (i *info) IsRunning() bool {
 
 	output, err := i.driver.getInfo(i.ID)
 	if err != nil {
-		panic(err)
+		utils.Errorf("Error getting info for lxc container %s: %s (%s)", i.ID, err, output)
+		return false
 	}
 	if strings.Contains(string(output), "RUNNING") {
 		running = true

execdriver/lxc/init.go

@@ -4,11 +4,10 @@ import (
 	"fmt"
 	"github.com/dotcloud/docker/execdriver"
 	"github.com/dotcloud/docker/pkg/netlink"
-	"github.com/dotcloud/docker/utils"
+	"github.com/dotcloud/docker/pkg/user"
 	"github.com/syndtr/gocapability/capability"
 	"net"
 	"os"
-	"strconv"
 	"strings"
 	"syscall"
 )
@@ -79,35 +78,28 @@ func setupWorkingDirectory(args *execdriver.InitArgs) error {
 
 // Takes care of dropping privileges to the desired user
 func changeUser(args *execdriver.InitArgs) error {
-	if args.User == "" {
-		return nil
-	}
-	userent, err := utils.UserLookup(args.User)
+	uid, gid, suppGids, err := user.GetUserGroupSupplementary(
+		args.User,
+		syscall.Getuid(), syscall.Getgid(),
+	)
 	if err != nil {
-		return fmt.Errorf("Unable to find user %v: %v", args.User, err)
+		return err
 	}
 
-	uid, err := strconv.Atoi(userent.Uid)
-	if err != nil {
-		return fmt.Errorf("Invalid uid: %v", userent.Uid)
+	if err := syscall.Setgroups(suppGids); err != nil {
+		return fmt.Errorf("Setgroups failed: %v", err)
 	}
-	gid, err := strconv.Atoi(userent.Gid)
-	if err != nil {
-		return fmt.Errorf("Invalid gid: %v", userent.Gid)
-	}
-
 	if err := syscall.Setgid(gid); err != nil {
-		return fmt.Errorf("setgid failed: %v", err)
+		return fmt.Errorf("Setgid failed: %v", err)
 	}
 	if err := syscall.Setuid(uid); err != nil {
-		return fmt.Errorf("setuid failed: %v", err)
+		return fmt.Errorf("Setuid failed: %v", err)
 	}
 
 	return nil
 }
 
 func setupCapabilities(args *execdriver.InitArgs) error {
-
 	if args.Privileged {
 		return nil
 	}
@@ -127,6 +119,7 @@ func setupCapabilities(args *execdriver.InitArgs) error {
 		capability.CAP_AUDIT_CONTROL,
 		capability.CAP_MAC_OVERRIDE,
 		capability.CAP_MAC_ADMIN,
+		capability.CAP_NET_ADMIN,
 	}
 
 	c, err := capability.NewPid(os.Getpid())

execdriver/lxc/lxc_template.go

@@ -15,6 +15,7 @@ lxc.network.name = eth0
 {{else}}
 # network is disabled (-n=false)
 lxc.network.type = empty
+lxc.network.flags = up
 {{end}}
 
 # root filesystem
@@ -79,6 +80,10 @@ lxc.mount.entry = proc {{escapeFstabSpaces $ROOTFS}}/proc proc nosuid,nodev,noex
 # if your userspace allows it. eg. see http://bit.ly/T9CkqJ
 lxc.mount.entry = sysfs {{escapeFstabSpaces $ROOTFS}}/sys sysfs nosuid,nodev,noexec 0 0
 
+{{if .Tty}}
+lxc.mount.entry = {{.Console}} {{escapeFstabSpaces $ROOTFS}}/dev/console none bind,rw 0 0
+{{end}}
+
 lxc.mount.entry = devpts {{escapeFstabSpaces $ROOTFS}}/dev/pts devpts newinstance,ptmxmode=0666,nosuid,noexec 0 0
 lxc.mount.entry = shm {{escapeFstabSpaces $ROOTFS}}/dev/shm tmpfs size=65536k,nosuid,nodev,noexec 0 0
 

graph.go

@@ -3,7 +3,9 @@ package docker
 import (
 	"fmt"
 	"github.com/dotcloud/docker/archive"
+	"github.com/dotcloud/docker/dockerversion"
 	"github.com/dotcloud/docker/graphdriver"
+	"github.com/dotcloud/docker/runconfig"
 	"github.com/dotcloud/docker/utils"
 	"io"
 	"io/ioutil"
@@ -125,12 +127,12 @@ func (graph *Graph) Get(name string) (*Image, error) {
 }
 
 // Create creates a new image and registers it in the graph.
-func (graph *Graph) Create(layerData archive.Archive, container *Container, comment, author string, config *Config) (*Image, error) {
+func (graph *Graph) Create(layerData archive.ArchiveReader, container *Container, comment, author string, config *runconfig.Config) (*Image, error) {
 	img := &Image{
 		ID:            GenerateID(),
 		Comment:       comment,
 		Created:       time.Now().UTC(),
-		DockerVersion: VERSION,
+		DockerVersion: dockerversion.VERSION,
 		Author:        author,
 		Config:        config,
 		Architecture:  runtime.GOARCH,
@@ -149,7 +151,7 @@ func (graph *Graph) Create(layerData archive.Archive, container *Container, comm
 
 // Register imports a pre-existing image into the graph.
 // FIXME: pass img as first argument
-func (graph *Graph) Register(jsonData []byte, layerData archive.Archive, img *Image) (err error) {
+func (graph *Graph) Register(jsonData []byte, layerData archive.ArchiveReader, img *Image) (err error) {
 	defer func() {
 		// If any error occurs, remove the new dir from the driver.
 		// Don't check for errors since the dir might not have been created.
@@ -224,7 +226,9 @@ func (graph *Graph) TempLayerArchive(id string, compression archive.Compression,
 	if err != nil {
 		return nil, err
 	}
-	return archive.NewTempArchive(utils.ProgressReader(ioutil.NopCloser(a), 0, output, sf, false, utils.TruncateID(id), "Buffering to disk"), tmp)
+	progress := utils.ProgressReader(a, 0, output, sf, false, utils.TruncateID(id), "Buffering to disk")
+	defer progress.Close()
+	return archive.NewTempArchive(progress, tmp)
 }
 
 // Mktemp creates a temporary sub-directory inside the graph's filesystem.

graphdriver/aufs/aufs.go

@@ -271,7 +271,7 @@ func (a *Driver) Diff(id string) (archive.Archive, error) {
 	})
 }
 
-func (a *Driver) ApplyDiff(id string, diff archive.Archive) error {
+func (a *Driver) ApplyDiff(id string, diff archive.ArchiveReader) error {
 	return archive.Untar(diff, path.Join(a.rootPath(), "diff", id), nil)
 }
 

graphdriver/devmapper/deviceset.go

@@ -12,6 +12,7 @@ import (
 	"path"
 	"path/filepath"
 	"strconv"
+	"strings"
 	"sync"
 	"time"
 )
@@ -29,6 +30,15 @@ type DevInfo struct {
 	TransactionId uint64     `json:"transaction_id"`
 	Initialized   bool       `json:"initialized"`
 	devices       *DeviceSet `json:"-"`
+
+	mountCount int    `json:"-"`
+	mountPath  string `json:"-"`
+	// A floating mount means one reference is not owned and
+	// will be stolen by the next mount. This allows us to
+	// avoid unmounting directly after creation before the
+	// first get (since we need to mount to set up the device
+	// a bit first).
+	floating bool `json:"-"`
 }
 
 type MetaData struct {
@@ -43,7 +53,7 @@ type DeviceSet struct {
 	TransactionId    uint64
 	NewTransactionId uint64
 	nextFreeDevice   int
-	activeMounts     map[string]int
+	sawBusy          bool
 }
 
 type DiskUsage struct {
@@ -69,6 +79,14 @@ type DevStatus struct {
 	HighestMappedSector uint64
 }
 
+type UnmountMode int
+
+const (
+	UnmountRegular UnmountMode = iota
+	UnmountFloat
+	UnmountSink
+)
+
 func getDevName(name string) string {
 	return "/dev/mapper/" + name
 }
@@ -290,7 +308,7 @@ func (devices *DeviceSet) setupBaseImage() error {
 
 	if oldInfo != nil && !oldInfo.Initialized {
 		utils.Debugf("Removing uninitialized base image")
-		if err := devices.removeDevice(""); err != nil {
+		if err := devices.deleteDevice(""); err != nil {
 			utils.Debugf("\n--->Err: %s\n", err)
 			return err
 		}
@@ -355,6 +373,10 @@ func (devices *DeviceSet) log(level int, file string, line int, dmError int, mes
 		return // Ignore _LOG_DEBUG
 	}
 
+	if strings.Contains(message, "busy") {
+		devices.sawBusy = true
+	}
+
 	utils.Debugf("libdevmapper(%d): %s:%d (%d) %s", level, file, line, dmError, message)
 }
 
@@ -562,7 +584,7 @@ func (devices *DeviceSet) AddDevice(hash, baseHash string) error {
 	return nil
 }
 
-func (devices *DeviceSet) removeDevice(hash string) error {
+func (devices *DeviceSet) deleteDevice(hash string) error {
 	info := devices.Devices[hash]
 	if info == nil {
 		return fmt.Errorf("hash %s doesn't exists", hash)
@@ -579,7 +601,7 @@ func (devices *DeviceSet) removeDevice(hash string) error {
 
 	devinfo, _ := getInfo(info.Name())
 	if devinfo != nil && devinfo.Exists != 0 {
-		if err := removeDevice(info.Name()); err != nil {
+		if err := devices.removeDeviceAndWait(info.Name()); err != nil {
 			utils.Debugf("Error removing device: %s\n", err)
 			return err
 		}
@@ -610,50 +632,87 @@ func (devices *DeviceSet) removeDevice(hash string) error {
 	return nil
 }
 
-func (devices *DeviceSet) RemoveDevice(hash string) error {
+func (devices *DeviceSet) DeleteDevice(hash string) error {
 	devices.Lock()
 	defer devices.Unlock()
 
-	return devices.removeDevice(hash)
+	return devices.deleteDevice(hash)
 }
 
-func (devices *DeviceSet) deactivateDevice(hash string) error {
-	utils.Debugf("[devmapper] deactivateDevice(%s)", hash)
-	defer utils.Debugf("[devmapper] deactivateDevice END")
-	var devname string
-	// FIXME: shouldn't we just register the pool into devices?
-	devname, err := devices.byHash(hash)
-	if err != nil {
-		return err
-	}
+func (devices *DeviceSet) deactivatePool() error {
+	utils.Debugf("[devmapper] deactivatePool()")
+	defer utils.Debugf("[devmapper] deactivatePool END")
+	devname := devices.getPoolDevName()
 	devinfo, err := getInfo(devname)
 	if err != nil {
 		utils.Debugf("\n--->Err: %s\n", err)
 		return err
 	}
 	if devinfo.Exists != 0 {
-		if err := removeDevice(devname); err != nil {
+		return removeDevice(devname)
+	}
+
+	return nil
+}
+
+func (devices *DeviceSet) deactivateDevice(hash string) error {
+	utils.Debugf("[devmapper] deactivateDevice(%s)", hash)
+	defer utils.Debugf("[devmapper] deactivateDevice END")
+
+	info := devices.Devices[hash]
+	if info == nil {
+		return fmt.Errorf("Unknown device %s", hash)
+	}
+	devinfo, err := getInfo(info.Name())
+	if err != nil {
+		utils.Debugf("\n--->Err: %s\n", err)
+		return err
+	}
+	if devinfo.Exists != 0 {
+		if err := devices.removeDeviceAndWait(info.Name()); err != nil {
 			utils.Debugf("\n--->Err: %s\n", err)
 			return err
 		}
-		if err := devices.waitRemove(hash); err != nil {
-			return err
-		}
 	}
 
 	return nil
 }
 
-// waitRemove blocks until either:
-// a) the device registered at <device_set_prefix>-<hash> is removed,
-// or b) the 1 second timeout expires.
-func (devices *DeviceSet) waitRemove(hash string) error {
-	utils.Debugf("[deviceset %s] waitRemove(%s)", devices.devicePrefix, hash)
-	defer utils.Debugf("[deviceset %s] waitRemove(%) END", devices.devicePrefix, hash)
-	devname, err := devices.byHash(hash)
+// Issues the underlying dm remove operation and then waits
+// for it to finish.
+func (devices *DeviceSet) removeDeviceAndWait(devname string) error {
+	var err error
+
+	for i := 0; i < 10; i++ {
+		devices.sawBusy = false
+		err = removeDevice(devname)
+		if err == nil {
+			break
+		}
+		if !devices.sawBusy {
+			return err
+		}
+
+		// If we see EBUSY it may be a transient error,
+		// sleep a bit a retry a few times.
+		time.Sleep(5 * time.Millisecond)
+	}
 	if err != nil {
 		return err
 	}
+
+	if err := devices.waitRemove(devname); err != nil {
+		return err
+	}
+	return nil
+}
+
+// waitRemove blocks until either:
+// a) the device registered at <device_set_prefix>-<hash> is removed,
+// or b) the 1 second timeout expires.
+func (devices *DeviceSet) waitRemove(devname string) error {
+	utils.Debugf("[deviceset %s] waitRemove(%s)", devices.devicePrefix, devname)
+	defer utils.Debugf("[deviceset %s] waitRemove(%s) END", devices.devicePrefix, devname)
 	i := 0
 	for ; i < 1000; i += 1 {
 		devinfo, err := getInfo(devname)
@@ -681,18 +740,18 @@ func (devices *DeviceSet) waitRemove(hash string) error {
 // a) the device registered at <device_set_prefix>-<hash> is closed,
 // or b) the 1 second timeout expires.
 func (devices *DeviceSet) waitClose(hash string) error {
-	devname, err := devices.byHash(hash)
-	if err != nil {
-		return err
+	info := devices.Devices[hash]
+	if info == nil {
+		return fmt.Errorf("Unknown device %s", hash)
 	}
 	i := 0
 	for ; i < 1000; i += 1 {
-		devinfo, err := getInfo(devname)
+		devinfo, err := getInfo(info.Name())
 		if err != nil {
 			return err
 		}
 		if i%100 == 0 {
-			utils.Debugf("Waiting for unmount of %s: opencount=%d", devname, devinfo.OpenCount)
+			utils.Debugf("Waiting for unmount of %s: opencount=%d", hash, devinfo.OpenCount)
 		}
 		if devinfo.OpenCount == 0 {
 			break
@@ -700,26 +759,11 @@ func (devices *DeviceSet) waitClose(hash string) error {
 		time.Sleep(1 * time.Millisecond)
 	}
 	if i == 1000 {
-		return fmt.Errorf("Timeout while waiting for device %s to close", devname)
+		return fmt.Errorf("Timeout while waiting for device %s to close", hash)
 	}
 	return nil
 }
 
-// byHash is a hack to allow looking up the deviceset's pool by the hash "pool".
-// FIXME: it seems probably cleaner to register the pool in devices.Devices,
-// but I am afraid of arcane implications deep in the devicemapper code,
-// so this will do.
-func (devices *DeviceSet) byHash(hash string) (devname string, err error) {
-	if hash == "pool" {
-		return devices.getPoolDevName(), nil
-	}
-	info := devices.Devices[hash]
-	if info == nil {
-		return "", fmt.Errorf("hash %s doesn't exists", hash)
-	}
-	return info.Name(), nil
-}
-
 func (devices *DeviceSet) Shutdown() error {
 	devices.Lock()
 	defer devices.Unlock()
@@ -728,13 +772,12 @@ func (devices *DeviceSet) Shutdown() error {
 	utils.Debugf("[devmapper] Shutting down DeviceSet: %s", devices.root)
 	defer utils.Debugf("[deviceset %s] shutdown END", devices.devicePrefix)
 
-	for path, count := range devices.activeMounts {
-		for i := count; i > 0; i-- {
-			if err := sysUnmount(path, 0); err != nil {
-				utils.Debugf("Shutdown unmounting %s, error: %s\n", path, err)
+	for _, info := range devices.Devices {
+		if info.mountCount > 0 {
+			if err := sysUnmount(info.mountPath, 0); err != nil {
+				utils.Debugf("Shutdown unmounting %s, error: %s\n", info.mountPath, err)
 			}
 		}
-		delete(devices.activeMounts, path)
 	}
 
 	for _, d := range devices.Devices {
@@ -746,32 +789,42 @@ func (devices *DeviceSet) Shutdown() error {
 		}
 	}
 
-	pool := devices.getPoolDevName()
-	if devinfo, err := getInfo(pool); err == nil && devinfo.Exists != 0 {
-		if err := devices.deactivateDevice("pool"); err != nil {
-			utils.Debugf("Shutdown deactivate %s , error: %s\n", pool, err)
-		}
+	if err := devices.deactivatePool(); err != nil {
+		utils.Debugf("Shutdown deactivate pool , error: %s\n", err)
 	}
 
 	return nil
 }
 
-func (devices *DeviceSet) MountDevice(hash, path string, readOnly bool) error {
+func (devices *DeviceSet) MountDevice(hash, path string) error {
 	devices.Lock()
 	defer devices.Unlock()
 
+	info := devices.Devices[hash]
+	if info == nil {
+		return fmt.Errorf("Unknown device %s", hash)
+	}
+
+	if info.mountCount > 0 {
+		if path != info.mountPath {
+			return fmt.Errorf("Trying to mount devmapper device in multple places (%s, %s)", info.mountPath, path)
+		}
+
+		if info.floating {
+			// Steal floating ref
+			info.floating = false
+		} else {
+			info.mountCount++
+		}
+		return nil
+	}
+
 	if err := devices.activateDeviceIfNeeded(hash); err != nil {
 		return fmt.Errorf("Error activating devmapper device for '%s': %s", hash, err)
 	}
 
-	info := devices.Devices[hash]
-
 	var flags uintptr = sysMsMgcVal
 
-	if readOnly {
-		flags = flags | sysMsRdOnly
-	}
-
 	err := sysMount(info.DevName(), path, "ext4", flags, "discard")
 	if err != nil && err == sysEInval {
 		err = sysMount(info.DevName(), path, "ext4", flags, "")
@@ -780,20 +833,53 @@ func (devices *DeviceSet) MountDevice(hash, path string, readOnly bool) error {
 		return fmt.Errorf("Error mounting '%s' on '%s': %s", info.DevName(), path, err)
 	}
 
-	count := devices.activeMounts[path]
-	devices.activeMounts[path] = count + 1
+	info.mountCount = 1
+	info.mountPath = path
+	info.floating = false
 
 	return devices.setInitialized(hash)
 }
 
-func (devices *DeviceSet) UnmountDevice(hash, path string, deactivate bool) error {
-	utils.Debugf("[devmapper] UnmountDevice(hash=%s path=%s)", hash, path)
+func (devices *DeviceSet) UnmountDevice(hash string, mode UnmountMode) error {
+	utils.Debugf("[devmapper] UnmountDevice(hash=%s, mode=%d)", hash, mode)
 	defer utils.Debugf("[devmapper] UnmountDevice END")
 	devices.Lock()
 	defer devices.Unlock()
 
-	utils.Debugf("[devmapper] Unmount(%s)", path)
-	if err := sysUnmount(path, 0); err != nil {
+	info := devices.Devices[hash]
+	if info == nil {
+		return fmt.Errorf("UnmountDevice: no such device %s\n", hash)
+	}
+
+	if mode == UnmountFloat {
+		if info.floating {
+			return fmt.Errorf("UnmountDevice: can't float floating reference %s\n", hash)
+		}
+
+		// Leave this reference floating
+		info.floating = true
+		return nil
+	}
+
+	if mode == UnmountSink {
+		if !info.floating {
+			// Someone already sunk this
+			return nil
+		}
+		// Otherwise, treat this as a regular unmount
+	}
+
+	if info.mountCount == 0 {
+		return fmt.Errorf("UnmountDevice: device not-mounted id %s\n", hash)
+	}
+
+	info.mountCount--
+	if info.mountCount > 0 {
+		return nil
+	}
+
+	utils.Debugf("[devmapper] Unmount(%s)", info.mountPath)
+	if err := sysUnmount(info.mountPath, 0); err != nil {
 		utils.Debugf("\n--->Err: %s\n", err)
 		return err
 	}
@@ -804,15 +890,9 @@ func (devices *DeviceSet) UnmountDevice(hash, path string, deactivate bool) erro
 		return err
 	}
 
-	if count := devices.activeMounts[path]; count > 1 {
-		devices.activeMounts[path] = count - 1
-	} else {
-		delete(devices.activeMounts, path)
-	}
+	devices.deactivateDevice(hash)
 
-	if deactivate {
-		devices.deactivateDevice(hash)
-	}
+	info.mountPath = ""
 
 	return nil
 }
@@ -955,9 +1035,8 @@ func NewDeviceSet(root string, doInit bool) (*DeviceSet, error) {
 	SetDevDir("/dev")
 
 	devices := &DeviceSet{
-		root:         root,
-		MetaData:     MetaData{Devices: make(map[string]*DevInfo)},
-		activeMounts: make(map[string]int),
+		root:     root,
+		MetaData: MetaData{Devices: make(map[string]*DevInfo)},
 	}
 
 	if err := devices.initDevmapper(doInit); err != nil {

graphdriver/devmapper/devmapper.go

@@ -324,7 +324,7 @@ func createPool(poolName string, dataFile, metadataFile *osFile) error {
 		return fmt.Errorf("Can't get data size")
 	}
 
-	params := metadataFile.Name() + " " + dataFile.Name() + " 128 32768"
+	params := metadataFile.Name() + " " + dataFile.Name() + " 128 32768 1 skip_block_zeroing"
 	if err := task.AddTarget(0, size/512, "thin-pool", params); err != nil {
 		return fmt.Errorf("Can't add target")
 	}

graphdriver/devmapper/driver.go

@@ -7,8 +7,8 @@ import (
 	"github.com/dotcloud/docker/graphdriver"
 	"github.com/dotcloud/docker/utils"
 	"io/ioutil"
+	"os"
 	"path"
-	"sync"
 )
 
 func init() {
@@ -22,9 +22,7 @@ func init() {
 
 type Driver struct {
 	*DeviceSet
-	home       string
-	sync.Mutex // Protects concurrent modification to active
-	active     map[string]int
+	home string
 }
 
 var Init = func(home string) (graphdriver.Driver, error) {
@@ -35,7 +33,6 @@ var Init = func(home string) (graphdriver.Driver, error) {
 	d := &Driver{
 		DeviceSet: deviceSet,
 		home:      home,
-		active:    make(map[string]int),
 	}
 	return d, nil
 }
@@ -83,55 +80,45 @@ func (d *Driver) Create(id, parent string) error {
 		return err
 	}
 
+	// We float this reference so that the next Get call can
+	// steal it, so we don't have to unmount
+	if err := d.DeviceSet.UnmountDevice(id, UnmountFloat); err != nil {
+		return err
+	}
+
 	return nil
 }
 
 func (d *Driver) Remove(id string) error {
-	// Protect the d.active from concurrent access
-	d.Lock()
-	defer d.Unlock()
-
-	if d.active[id] != 0 {
-		utils.Errorf("Warning: removing active id %s\n", id)
+	// Sink the float from create in case no Get() call was made
+	if err := d.DeviceSet.UnmountDevice(id, UnmountSink); err != nil {
+		return err
+	}
+	// This assumes the device has been properly Get/Put:ed and thus is unmounted
+	if err := d.DeviceSet.DeleteDevice(id); err != nil {
+		return err
 	}
 
 	mp := path.Join(d.home, "mnt", id)
-	if err := d.unmount(id, mp); err != nil {
+	if err := os.RemoveAll(mp); err != nil && !os.IsNotExist(err) {
 		return err
 	}
-	return d.DeviceSet.RemoveDevice(id)
+
+	return nil
 }
 
 func (d *Driver) Get(id string) (string, error) {
-	// Protect the d.active from concurrent access
-	d.Lock()
-	defer d.Unlock()
-
-	count := d.active[id]
-
 	mp := path.Join(d.home, "mnt", id)
-	if count == 0 {
-		if err := d.mount(id, mp); err != nil {
-			return "", err
-		}
+	if err := d.mount(id, mp); err != nil {
+		return "", err
 	}
 
-	d.active[id] = count + 1
-
 	return path.Join(mp, "rootfs"), nil
 }
 
 func (d *Driver) Put(id string) {
-	// Protect the d.active from concurrent access
-	d.Lock()
-	defer d.Unlock()
-
-	if count := d.active[id]; count > 1 {
-		d.active[id] = count - 1
-	} else {
-		mp := path.Join(d.home, "mnt", id)
-		d.unmount(id, mp)
-		delete(d.active, id)
+	if err := d.DeviceSet.UnmountDevice(id, UnmountRegular); err != nil {
+		utils.Errorf("Warning: error unmounting device %s: %s\n", id, err)
 	}
 }
 
@@ -140,25 +127,8 @@ func (d *Driver) mount(id, mountPoint string) error {
 	if err := osMkdirAll(mountPoint, 0755); err != nil && !osIsExist(err) {
 		return err
 	}
-	// If mountpoint is already mounted, do nothing
-	if mounted, err := Mounted(mountPoint); err != nil {
-		return fmt.Errorf("Error checking mountpoint: %s", err)
-	} else if mounted {
-		return nil
-	}
 	// Mount the device
-	return d.DeviceSet.MountDevice(id, mountPoint, false)
-}
-
-func (d *Driver) unmount(id, mountPoint string) error {
-	// If mountpoint is not mounted, do nothing
-	if mounted, err := Mounted(mountPoint); err != nil {
-		return fmt.Errorf("Error checking mountpoint: %s", err)
-	} else if !mounted {
-		return nil
-	}
-	// Unmount the device
-	return d.DeviceSet.UnmountDevice(id, mountPoint, true)
+	return d.DeviceSet.MountDevice(id, mountPoint)
 }
 
 func (d *Driver) Exists(id string) bool {

graphdriver/devmapper/driver_test.go

@@ -136,7 +136,12 @@ type Set map[string]bool
 
 func (r Set) Assert(t *testing.T, names ...string) {
 	for _, key := range names {
-		if _, exists := r[key]; !exists {
+		required := true
+		if strings.HasPrefix(key, "?") {
+			key = key[1:]
+			required = false
+		}
+		if _, exists := r[key]; !exists && required {
 			t.Fatalf("Key not set: %s", key)
 		}
 		delete(r, key)
@@ -486,6 +491,7 @@ func TestDriverCreate(t *testing.T) {
 			"ioctl.blkgetsize",
 			"ioctl.loopsetfd",
 			"ioctl.loopsetstatus",
+			"?ioctl.loopctlgetfree",
 		)
 
 		if err := d.Create("1", ""); err != nil {
@@ -495,7 +501,6 @@ func TestDriverCreate(t *testing.T) {
 			"DmTaskCreate",
 			"DmTaskGetInfo",
 			"sysMount",
-			"Mounted",
 			"DmTaskRun",
 			"DmTaskSetTarget",
 			"DmTaskSetSector",
@@ -604,6 +609,7 @@ func TestDriverRemove(t *testing.T) {
 			"ioctl.blkgetsize",
 			"ioctl.loopsetfd",
 			"ioctl.loopsetstatus",
+			"?ioctl.loopctlgetfree",
 		)
 
 		if err := d.Create("1", ""); err != nil {
@@ -614,7 +620,6 @@ func TestDriverRemove(t *testing.T) {
 			"DmTaskCreate",
 			"DmTaskGetInfo",
 			"sysMount",
-			"Mounted",
 			"DmTaskRun",
 			"DmTaskSetTarget",
 			"DmTaskSetSector",
@@ -645,7 +650,6 @@ func TestDriverRemove(t *testing.T) {
 			"DmTaskSetTarget",
 			"DmTaskSetAddNode",
 			"DmUdevWait",
-			"Mounted",
 			"sysUnmount",
 		)
 	}()

graphdriver/driver.go

@@ -28,7 +28,7 @@ type Driver interface {
 type Differ interface {
 	Diff(id string) (archive.Archive, error)
 	Changes(id string) ([]archive.Change, error)
-	ApplyDiff(id string, diff archive.Archive) error
+	ApplyDiff(id string, diff archive.ArchiveReader) error
 	DiffSize(id string) (bytes int64, err error)
 }
 

hack/infrastructure/docker-ci.rst

@@ -1,56 +0,0 @@
-docker-ci
-=========
-
-docker-ci is our buildbot continuous integration server,
-building and testing docker, hosted on EC2 and reachable at
-http://docker-ci.dotcloud.com
-
-
-Deployment
-==========
-
-# Load AWS credentials
-export AWS_ACCESS_KEY_ID=''
-export AWS_SECRET_ACCESS_KEY=''
-export AWS_KEYPAIR_NAME=''
-export AWS_SSH_PRIVKEY=''
-
-# Load buildbot credentials and config
-export BUILDBOT_PWD=''
-export IRC_PWD=''
-export IRC_CHANNEL='docker-dev'
-export SMTP_USER=''
-export SMTP_PWD=''
-export EMAIL_RCP=''
-
-# Load registry test credentials
-export REGISTRY_USER=''
-export REGISTRY_PWD=''
-
-cd docker/testing
-vagrant up --provider=aws
-
-
-github pull request
-===================
-
-The entire docker pull request test workflow is event driven by github. Its
-usage is fully automatic and the results are logged in docker-ci.dotcloud.com
-
-Each time there is a pull request on docker's github project, github connects
-to docker-ci using github's rest API documented in http://developer.github.com/v3/repos/hooks
-The issued command to program github's notification PR event was:
-curl -u GITHUB_USER:GITHUB_PASSWORD -d '{"name":"web","active":true,"events":["pull_request"],"config":{"url":"http://docker-ci.dotcloud.com:8011/change_hook/github?project=docker"}}' https://api.github.com/repos/dotcloud/docker/hooks
-
-buildbot (0.8.7p1) was patched using ./testing/buildbot/github.py, so it
-can understand the PR data github sends to it. Originally PR #1603 (ee64e099e0)
-implemented this capability. Also we added a new scheduler to exclusively filter
-PRs. and the 'pullrequest' builder to rebase the PR on top of master and test it.
-
-
-nighthly release
-================
-
-The nightly release process is done by buildbot, running a DinD container that downloads
-the docker repository and builds the release container. The resulting docker
-binary is then tested, and if everything is fine, the release is done.

hack/infrastructure/docker-ci/Dockerfile

@@ -1,47 +1,29 @@
-# VERSION:        0.25
-# DOCKER-VERSION  0.6.6
-# AUTHOR:         Daniel Mizyrycki <daniel@docker.com>
-# DESCRIPTION:    Deploy docker-ci on Digital Ocean
-# COMMENTS:
-#     CONFIG_JSON is an environment variable json string loaded as:
-#
-# export CONFIG_JSON='
-#     { "DROPLET_NAME":        "docker-ci",
-#       "DO_CLIENT_ID":        "Digital_Ocean_client_id",
-#       "DO_API_KEY":          "Digital_Ocean_api_key",
-#       "DOCKER_KEY_ID":       "Digital_Ocean_ssh_key_id",
-#       "DOCKER_CI_KEY_PATH":  "docker-ci_private_key_path",
-#       "DOCKER_CI_PUB":       "$(cat docker-ci_ssh_public_key.pub)",
-#       "DOCKER_CI_KEY":       "$(cat docker-ci_ssh_private_key.key)",
-#       "BUILDBOT_PWD":        "Buildbot_server_password",
-#       "IRC_PWD":             "Buildbot_IRC_password",
-#       "SMTP_USER":           "SMTP_server_user",
-#       "SMTP_PWD":            "SMTP_server_password",
-#       "PKG_ACCESS_KEY":      "Docker_release_S3_bucket_access_key",
-#       "PKG_SECRET_KEY":      "Docker_release_S3_bucket_secret_key",
-#       "PKG_GPG_PASSPHRASE":  "Docker_release_gpg_passphrase",
-#       "INDEX_AUTH":          "Index_encripted_user_password",
-#       "REGISTRY_USER":       "Registry_test_user",
-#       "REGISTRY_PWD":        "Registry_test_password",
-#       "REGISTRY_BUCKET":     "Registry_S3_bucket_name",
-#       "REGISTRY_ACCESS_KEY": "Registry_S3_bucket_access_key",
-#       "REGISTRY_SECRET_KEY": "Registry_S3_bucket_secret_key",
-#       "IRC_CHANNEL":         "Buildbot_IRC_channel",
-#       "EMAIL_RCP":           "Buildbot_mailing_receipient" }'
-#
-#
-# TO_BUILD:   docker build -t docker-ci .
-# TO_DEPLOY:  docker run -e CONFIG_JSON="${CONFIG_JSON}" docker-ci
+# DOCKER-VERSION: 0.7.6
+# AUTHOR:         Daniel Mizyrycki <daniel@dotcloud.com>
+# DESCRIPTION:    docker-ci continuous integration service
+# TO_BUILD:       docker build -rm -t docker-ci/docker-ci .
+# TO_RUN:         docker run -rm -i -t -p 8000:80 -p 2222:22 -v /run:/var/socket \
+#                     -v /data/docker-ci:/data/docker-ci docker-ci/docker-ci
 
 from ubuntu:12.04
+maintainer Daniel Mizyrycki <daniel@dotcloud.com>
 
-run echo 'deb http://archive.ubuntu.com/ubuntu precise main universe' \
-    > /etc/apt/sources.list
-run apt-get update; apt-get install -y git python2.7 python-dev libevent-dev \
-    python-pip ssh rsync less vim
-run pip install requests fabric
+ENV DEBIAN_FRONTEND noninteractive
+RUN echo 'deb http://archive.ubuntu.com/ubuntu precise main universe' > \
+    /etc/apt/sources.list; apt-get update
+RUN apt-get install -y --no-install-recommends  python2.7 python-dev \
+    libevent-dev git supervisor ssh rsync less vim sudo gcc wget nginx
+RUN cd /tmp; wget http://python-distribute.org/distribute_setup.py
+RUN cd /tmp; python distribute_setup.py; easy_install pip; rm distribute_setup.py
 
-# Add deployment code and set default container command
-add . /docker-ci
-cmd "/docker-ci/deployment.py"
+RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
+RUN echo 'deb http://get.docker.io/ubuntu docker main' > \
+      /etc/apt/sources.list.d/docker.list; apt-get update
+RUN apt-get install -y lxc-docker-0.8.0
+RUN pip install SQLAlchemy==0.7.10 buildbot buildbot-slave pyopenssl boto
+RUN ln -s /var/socket/docker.sock /run/docker.sock
 
+ADD . /docker-ci
+RUN /docker-ci/setup.sh
+
+ENTRYPOINT ["supervisord", "-n"]

hack/infrastructure/docker-ci/README.rst

@@ -1,26 +1,65 @@
-=======
-testing
-=======
+=========
+docker-ci
+=========
 
-This directory contains docker-ci testing related files.
+This directory contains docker-ci continuous integration system.
+As expected, it is a fully dockerized and deployed using
+docker-container-runner.
+docker-ci is based on Buildbot, a continuous integration system designed
+to automate the build/test cycle. By automatically rebuilding and testing
+the tree each time something has changed, build problems are pinpointed
+quickly, before other developers are inconvenienced by the failure.
+We are running buildbot at Rackspace to verify docker and docker-registry
+pass tests, and check for coverage code details.
+
+docker-ci instance is at https://docker-ci.docker.io/waterfall
+
+Inside docker-ci container we have the following directory structure:
+
+/docker-ci                       source code of docker-ci
+/data/backup/docker-ci/          daily backup (replicated over S3)
+/data/docker-ci/coverage/{docker,docker-registry}/    mapped to host volumes
+/data/buildbot/{master,slave}/   main docker-ci buildbot config and database
+/var/socket/{docker.sock}        host volume access to docker socket
 
 
-Buildbot
-========
+Production deployment
+=====================
 
-Buildbot is a continuous integration system designed to automate the
-build/test cycle. By automatically rebuilding and testing the tree each time
-something has changed, build problems are pinpointed quickly, before other
-developers are inconvenienced by the failure.
+::
 
-We are running buildbot in Amazon's EC2 to verify docker passes all
-tests when commits get pushed to the master branch and building
-nightly releases using Docker in Docker awesome implementation made
-by Jerome Petazzoni.
+  # Clone docker-ci repository
+  git clone https://github.com/dotcloud/docker
+  cd docker/hack/infrastructure/docker-ci
 
-https://github.com/jpetazzo/dind
+  export DOCKER_PROD=[PRODUCTION_SERVER_IP]
 
-Docker's buildbot instance is at http://docker-ci.dotcloud.com/waterfall
+  # Create data host volume. (only once)
+  docker -H $DOCKER_PROD run -v /home:/data ubuntu:12.04 \
+    mkdir -p /data/docker-ci/coverage/docker
+  docker -H $DOCKER_PROD run -v /home:/data ubuntu:12.04 \
+    mkdir -p /data/docker-ci/coverage/docker-registry
+  docker -H $DOCKER_PROD run -v /home:/data ubuntu:12.04 \
+    chown -R 1000.1000 /data/docker-ci
 
-For deployment instructions, please take a look at
-hack/infrastructure/docker-ci/Dockerfile
+  # dcr deployment. Define credentials and special environment dcr variables
+  # ( retrieved at /hack/infrastructure/docker-ci/dcr/prod/docker-ci.yml )
+  export WEB_USER=[DOCKER-CI-WEBSITE-USERNAME]
+  export WEB_IRC_PWD=[DOCKER-CI-WEBSITE-PASSWORD]
+  export BUILDBOT_PWD=[BUILDSLAVE_PASSWORD]
+  export AWS_ACCESS_KEY=[DOCKER_RELEASE_S3_ACCESS]
+  export AWS_SECRET_KEY=[DOCKER_RELEASE_S3_SECRET]
+  export GPG_PASSPHRASE=[DOCKER_RELEASE_PASSPHRASE]
+  export BACKUP_AWS_ID=[S3_BUCKET_CREDENTIAL_ACCESS]
+  export BACKUP_AWS_SECRET=[S3_BUCKET_CREDENTIAL_SECRET]
+  export SMTP_USER=[MAILGUN_SMTP_USERNAME]
+  export SMTP_PWD=[MAILGUN_SMTP_PASSWORD]
+  export EMAIL_RCP=[EMAIL_FOR_BUILD_ERRORS]
+
+  # Build docker-ci and testbuilder docker images
+  docker -H $DOCKER_PROD build -rm -t docker-ci/docker-ci .
+  (cd testbuilder; docker -H $DOCKER_PROD build -rm -t docker-ci/testbuilder .)
+
+  # Run docker-ci container ( assuming no previous container running )
+  (cd dcr/prod; dcr docker-ci.yml start)
+  (cd dcr/prod; dcr docker-ci.yml register docker-ci.docker.io)

hack/infrastructure/docker-ci/VERSION

@@ -1 +1 @@
-0.4.5
+0.5.6

hack/infrastructure/docker-ci/buildbot/README.rst

@@ -1 +0,0 @@
-Buildbot configuration and setup files

hack/infrastructure/docker-ci/buildbot/buildbot.conf

@@ -1,18 +0,0 @@
-[program:buildmaster]
-command=twistd --nodaemon --no_save -y buildbot.tac
-directory=/data/buildbot/master
-chown= root:root
-redirect_stderr=true
-stdout_logfile=/var/log/supervisor/buildbot-master.log
-stderr_logfile=/var/log/supervisor/buildbot-master.log
-
-[program:buildworker]
-command=twistd --nodaemon --no_save -y buildbot.tac
-directory=/data/buildbot/slave
-chown= root:root
-redirect_stderr=true
-stdout_logfile=/var/log/supervisor/buildbot-slave.log
-stderr_logfile=/var/log/supervisor/buildbot-slave.log
-
-[group:buildbot]
-programs=buildmaster,buildworker

hack/infrastructure/docker-ci/buildbot/github.py

@@ -17,7 +17,7 @@
 """
 github_buildbot.py is based on git_buildbot.py
 
-github_buildbot.py will determine the repository information from the JSON 
+github_buildbot.py will determine the repository information from the JSON
 HTTP POST it receives from github.com and build the appropriate repository.
 If your github repository is private, you must add a ssh key to the github
 repository for the user who initiated the build on the buildslave.
@@ -88,7 +88,8 @@ def getChanges(request, options = None):
         payload = json.loads(request.args['payload'][0])
         import urllib,datetime
         fname = str(datetime.datetime.now()).replace(' ','_').replace(':','-')[:19]
-        open('github_{0}.json'.format(fname),'w').write(json.dumps(json.loads(urllib.unquote(request.args['payload'][0])), sort_keys = True, indent = 2))
+        # Github event debug
+        # open('github_{0}.json'.format(fname),'w').write(json.dumps(json.loads(urllib.unquote(request.args['payload'][0])), sort_keys = True, indent = 2))
 
         if 'pull_request' in payload:
             user = payload['pull_request']['user']['login']
@@ -142,13 +143,13 @@ def process_change(payload, user, repo, repo_url, project):
                     'category'   : 'github_pullrequest',
                     'who'        : '{0} - PR#{1}'.format(user,payload['number']),
                     'files'      : [],
-                    'comments'   : payload['pull_request']['title'], 
+                    'comments'   : payload['pull_request']['title'],
                     'revision'   : newrev,
                     'when'       : convertTime(payload['pull_request']['updated_at']),
                     'branch'     : branch,
                     'revlink'    : '{0}/commit/{1}'.format(repo_url,newrev),
                     'repository' : repo_url,
-                    'project'  : project  }] 
+                    'project'  : project  }]
                 return changes
             for commit in payload['commits']:
                 files = []

hack/infrastructure/docker-ci/buildbot/master.cfg

@@ -1,4 +1,4 @@
-import os
+import os, re
 from buildbot.buildslave import BuildSlave
 from buildbot.schedulers.forcesched import ForceScheduler
 from buildbot.schedulers.basic import SingleBranchScheduler
@@ -6,127 +6,156 @@ from buildbot.schedulers.timed import Nightly
 from buildbot.changes import filter
 from buildbot.config import BuilderConfig
 from buildbot.process.factory import BuildFactory
-from buildbot.process.properties import Interpolate
+from buildbot.process.properties import Property
 from buildbot.steps.shell import ShellCommand
 from buildbot.status import html, words
 from buildbot.status.web import authz, auth
 from buildbot.status.mail import MailNotifier
 
-PORT_WEB = 80           # Buildbot webserver port
-PORT_GITHUB = 8011      # Buildbot github hook port
-PORT_MASTER = 9989      # Port where buildbot master listen buildworkers
-TEST_USER = 'buildbot'  # Credential to authenticate build triggers
-TEST_PWD = 'docker'     # Credential to authenticate build triggers
-GITHUB_DOCKER = 'github.com/dotcloud/docker'
-BUILDBOT_PATH = '/data/buildbot'
-DOCKER_PATH = '/go/src/github.com/dotcloud/docker'
-DOCKER_CI_PATH = '/docker-ci'
+
+def ENV(x):
+    '''Promote an environment variable for global use returning its value'''
+    retval = os.environ.get(x, '')
+    globals()[x] = retval
+    return retval
+
+
+class TestCommand(ShellCommand):
+    '''Extend ShellCommand with optional summary logs'''
+    def __init__(self, *args, **kwargs):
+        super(TestCommand, self).__init__(*args, **kwargs)
+
+    def createSummary(self, log):
+        exit_status = re.sub(r'.+\n\+ exit (\d+).+',
+            r'\1', log.getText()[-100:], flags=re.DOTALL)
+        if exit_status != '0':
+            return
+        # Infer coverage path from log
+        if '+ COVERAGE_PATH' in log.getText():
+            path = re.sub(r'.+\+ COVERAGE_PATH=((.+?)-\d+).+',
+              r'\2/\1', log.getText(), flags=re.DOTALL)
+            url = '{}coverage/{}/index.html'.format(c['buildbotURL'], path)
+            self.addURL('coverage', url)
+        elif 'COVERAGE_FILE' in log.getText():
+            path = re.sub(r'.+\+ COVERAGE_FILE=((.+?)-\d+).+',
+              r'\2/\1', log.getText(), flags=re.DOTALL)
+            url = '{}coverage/{}/index.html'.format(c['buildbotURL'], path)
+            self.addURL('coverage', url)
+
+
+PORT_WEB = 8000      # Buildbot webserver port
+PORT_GITHUB = 8011   # Buildbot github hook port
+PORT_MASTER = 9989   # Port where buildbot master listen buildworkers
+
+BUILDBOT_URL = '//localhost:{}/'.format(PORT_WEB)
+DOCKER_REPO = 'https://github.com/docker-test/docker'
+DOCKER_TEST_ARGV = 'HEAD {}'.format(DOCKER_REPO)
+REGISTRY_REPO = 'https://github.com/docker-test/docker-registry'
+REGISTRY_TEST_ARGV = 'HEAD {}'.format(REGISTRY_REPO)
+if ENV('DEPLOYMENT') == 'staging':
+    BUILDBOT_URL = "//docker-ci-stage.docker.io/"
+if ENV('DEPLOYMENT') == 'production':
+    BUILDBOT_URL = '//docker-ci.docker.io/'
+    DOCKER_REPO = 'https://github.com/dotcloud/docker'
+    DOCKER_TEST_ARGV = ''
+    REGISTRY_REPO = 'https://github.com/dotcloud/docker-registry'
+    REGISTRY_TEST_ARGV = ''
 
 # Credentials set by setup.sh from deployment.py
-BUILDBOT_PWD = ''
-IRC_PWD = ''
-IRC_CHANNEL = ''
-SMTP_USER = ''
-SMTP_PWD = ''
-EMAIL_RCP = ''
+ENV('WEB_USER')
+ENV('WEB_IRC_PWD')
+ENV('BUILDBOT_PWD')
+ENV('SMTP_USER')
+ENV('SMTP_PWD')
+ENV('EMAIL_RCP')
+ENV('IRC_CHANNEL')
 
 
 c = BuildmasterConfig = {}
 
-c['title'] = "Docker"
+c['title'] = "docker-ci"
 c['titleURL'] = "waterfall"
-c['buildbotURL'] = "http://docker-ci.dotcloud.com/"
+c['buildbotURL'] = BUILDBOT_URL
 c['db'] = {'db_url':"sqlite:///state.sqlite"}
 c['slaves'] = [BuildSlave('buildworker', BUILDBOT_PWD)]
 c['slavePortnum'] = PORT_MASTER
 
 
 # Schedulers
-c['schedulers'] = [ForceScheduler(name='trigger', builderNames=['docker',
-    'index','registry','docker-coverage','registry-coverage','nightlyrelease'])]
-c['schedulers'] += [SingleBranchScheduler(name="all", treeStableTimer=None,
+c['schedulers'] = [ForceScheduler(name='trigger', builderNames=[
+    'docker', 'docker-registry', 'nightlyrelease', 'backup'])]
+c['schedulers'] += [SingleBranchScheduler(name="docker", treeStableTimer=None,
     change_filter=filter.ChangeFilter(branch='master',
-    repository='https://github.com/dotcloud/docker'), builderNames=['docker'])]
-c['schedulers'] += [SingleBranchScheduler(name='pullrequest',
-    change_filter=filter.ChangeFilter(category='github_pullrequest'), treeStableTimer=None,
-    builderNames=['pullrequest'])]
-c['schedulers'] += [Nightly(name='daily', branch=None, builderNames=['nightlyrelease',
-    'docker-coverage','registry-coverage'], hour=7, minute=00)]
-c['schedulers'] += [Nightly(name='every4hrs', branch=None, builderNames=['registry','index'],
-    hour=range(0,24,4), minute=15)]
+    repository=DOCKER_REPO), builderNames=['docker'])]
+c['schedulers'] += [SingleBranchScheduler(name="registry", treeStableTimer=None,
+    change_filter=filter.ChangeFilter(branch='master',
+    repository=REGISTRY_REPO), builderNames=['docker-registry'])]
+c['schedulers'] += [SingleBranchScheduler(name='docker-pr', treeStableTimer=None,
+    change_filter=filter.ChangeFilter(category='github_pullrequest',
+    project='docker'), builderNames=['docker-pr'])]
+c['schedulers'] += [SingleBranchScheduler(name='docker-registry-pr', treeStableTimer=None,
+    change_filter=filter.ChangeFilter(category='github_pullrequest',
+    project='docker-registry'), builderNames=['docker-registry-pr'])]
+c['schedulers'] += [Nightly(name='daily', branch=None, builderNames=[
+    'nightlyrelease', 'backup'], hour=7, minute=00)]
+
 
 # Builders
-# Docker commit test
-test_cmd = ('docker run -privileged mzdaniel/test_docker hack/dind'
-    ' test_docker.sh %(src::revision)s')
+
+# Backup
 factory = BuildFactory()
-factory.addStep(ShellCommand(description='Docker', logEnviron=False,
-    usePTY=True, command=["sh", "-c", Interpolate(test_cmd)]))
-c['builders'] = [BuilderConfig(name='docker',slavenames=['buildworker'],
+factory.addStep(TestCommand(description='backup', logEnviron=False,
+    usePTY=True, command='/docker-ci/tool/backup.py'))
+c['builders'] = [BuilderConfig(name='backup',slavenames=['buildworker'],
+    factory=factory)]
+
+# Docker test
+factory = BuildFactory()
+factory.addStep(TestCommand(description='docker', logEnviron=False,
+    usePTY=True, command='/docker-ci/dockertest/docker {}'.format(DOCKER_TEST_ARGV)))
+c['builders'] += [BuilderConfig(name='docker',slavenames=['buildworker'],
     factory=factory)]
 
 # Docker pull request test
-test_cmd = ('docker run -privileged mzdaniel/test_docker hack/dind'
-    ' test_docker.sh %(src::revision)s %(src::repository)s %(src::branch)s')
 factory = BuildFactory()
-factory.addStep(ShellCommand(description='pull_request', logEnviron=False,
-    usePTY=True, command=["sh", "-c", Interpolate(test_cmd)]))
-c['builders'] += [BuilderConfig(name='pullrequest',slavenames=['buildworker'],
+factory.addStep(TestCommand(description='docker-pr', logEnviron=False,
+    usePTY=True, command=['/docker-ci/dockertest/docker',
+    Property('revision'), Property('repository'), Property('branch')]))
+c['builders'] += [BuilderConfig(name='docker-pr',slavenames=['buildworker'],
     factory=factory)]
 
-# Docker coverage test
+# docker-registry test
 factory = BuildFactory()
-factory.addStep(ShellCommand(description='docker-coverage', logEnviron=False,
-    usePTY=True, command='{0}/docker-coverage/coverage-docker.sh'.format(
-    DOCKER_CI_PATH)))
-c['builders'] += [BuilderConfig(name='docker-coverage',slavenames=['buildworker'],
+factory.addStep(TestCommand(description='docker-registry', logEnviron=False,
+    usePTY=True, command='/docker-ci/dockertest/docker-registry {}'.format(REGISTRY_TEST_ARGV)))
+c['builders'] += [BuilderConfig(name='docker-registry',slavenames=['buildworker'],
     factory=factory)]
 
-# Docker registry coverage test
+# Docker registry pull request test
 factory = BuildFactory()
-factory.addStep(ShellCommand(description='registry-coverage', logEnviron=False,
-    usePTY=True, command='docker run registry_coverage'.format(
-    DOCKER_CI_PATH)))
-c['builders'] += [BuilderConfig(name='registry-coverage',slavenames=['buildworker'],
-    factory=factory)]
-
-# Registry functional test
-factory = BuildFactory()
-factory.addStep(ShellCommand(description='registry', logEnviron=False,
-    command='. {0}/master/credentials.cfg; '
-    '{1}/functionaltests/test_registry.sh'.format(BUILDBOT_PATH, DOCKER_CI_PATH),
-    usePTY=True))
-c['builders'] += [BuilderConfig(name='registry',slavenames=['buildworker'],
-    factory=factory)]
-
-# Index functional test
-factory = BuildFactory()
-factory.addStep(ShellCommand(description='index', logEnviron=False,
-    command='. {0}/master/credentials.cfg; '
-    '{1}/functionaltests/test_index.py'.format(BUILDBOT_PATH, DOCKER_CI_PATH),
-    usePTY=True))
-c['builders'] += [BuilderConfig(name='index',slavenames=['buildworker'],
+factory.addStep(TestCommand(description='docker-registry-pr', logEnviron=False,
+    usePTY=True, command=['/docker-ci/dockertest/docker-registry',
+    Property('revision'), Property('repository'), Property('branch')]))
+c['builders'] += [BuilderConfig(name='docker-registry-pr',slavenames=['buildworker'],
     factory=factory)]
 
 # Docker nightly release
-nightlyrelease_cmd = ('docker version; docker run -i -t -privileged -e AWS_S3_BUCKET='
-    'test.docker.io dockerbuilder hack/dind dockerbuild.sh')
 factory = BuildFactory()
 factory.addStep(ShellCommand(description='NightlyRelease',logEnviron=False,
-    usePTY=True, command=nightlyrelease_cmd))
+    usePTY=True, command=['/docker-ci/dockertest/nightlyrelease']))
 c['builders'] += [BuilderConfig(name='nightlyrelease',slavenames=['buildworker'],
     factory=factory)]
 
 # Status
-authz_cfg = authz.Authz(auth=auth.BasicAuth([(TEST_USER, TEST_PWD)]),
+authz_cfg = authz.Authz(auth=auth.BasicAuth([(WEB_USER, WEB_IRC_PWD)]),
     forceBuild='auth')
 c['status'] = [html.WebStatus(http_port=PORT_WEB, authz=authz_cfg)]
 c['status'].append(html.WebStatus(http_port=PORT_GITHUB, allowForce=True,
     change_hook_dialects={ 'github': True }))
-c['status'].append(MailNotifier(fromaddr='buildbot@docker.io',
+c['status'].append(MailNotifier(fromaddr='docker-test@docker.io',
     sendToInterestedUsers=False, extraRecipients=[EMAIL_RCP],
     mode='failing', relayhost='smtp.mailgun.org', smtpPort=587, useTls=True,
     smtpUser=SMTP_USER, smtpPassword=SMTP_PWD))
 c['status'].append(words.IRC("irc.freenode.net", "dockerqabot",
-    channels=[IRC_CHANNEL], password=IRC_PWD, allowForce=True,
+    channels=[IRC_CHANNEL], password=WEB_IRC_PWD, allowForce=True,
     notify_events={'exception':1, 'successToFailure':1, 'failureToSuccess':1}))

hack/infrastructure/docker-ci/buildbot/requirements.txt

@@ -1,9 +0,0 @@
-sqlalchemy<=0.7.9
-sqlalchemy-migrate>=0.7.2
-buildbot==0.8.7p1
-buildbot_slave==0.8.7p1
-nose==1.2.1
-requests==1.1.0
-flask==0.10.1
-simplejson==2.3.2
-selenium==2.35.0

hack/infrastructure/docker-ci/buildbot/setup.sh

@@ -1,59 +0,0 @@
-#!/usr/bin/env bash
-
-# Setup of buildbot configuration. Package installation is being done by
-# Vagrantfile
-# Dependencies: buildbot, buildbot-slave, supervisor
-
-USER=$1
-CFG_PATH=$2
-DOCKER_PATH=$3
-BUILDBOT_PWD=$4
-IRC_PWD=$5
-IRC_CHANNEL=$6
-SMTP_USER=$7
-SMTP_PWD=$8
-EMAIL_RCP=$9
-REGISTRY_USER=${10}
-REGISTRY_PWD=${11}
-REGISTRY_BUCKET=${12}
-REGISTRY_ACCESS_KEY=${13}
-REGISTRY_SECRET_KEY=${14}
-BUILDBOT_PATH="/data/buildbot"
-SLAVE_NAME="buildworker"
-SLAVE_SOCKET="localhost:9989"
-
-export PATH="/bin:sbin:/usr/bin:/usr/sbin:/usr/local/bin"
-
-function run { su $USER -c "$1"; }
-
-# Exit if buildbot has already been installed
-[ -d "$BUILDBOT_PATH" ] && exit 0
-
-# Setup buildbot
-run "mkdir -p $BUILDBOT_PATH"
-cd $BUILDBOT_PATH
-run "buildbot create-master master"
-run "cp $CFG_PATH/master.cfg master"
-run "sed -i -E 's#(BUILDBOT_PWD = ).+#\1\"$BUILDBOT_PWD\"#' master/master.cfg"
-run "sed -i -E 's#(IRC_PWD = ).+#\1\"$IRC_PWD\"#' master/master.cfg"
-run "sed -i -E 's#(IRC_CHANNEL = ).+#\1\"$IRC_CHANNEL\"#' master/master.cfg"
-run "sed -i -E 's#(SMTP_USER = ).+#\1\"$SMTP_USER\"#' master/master.cfg"
-run "sed -i -E 's#(SMTP_PWD = ).+#\1\"$SMTP_PWD\"#' master/master.cfg"
-run "sed -i -E 's#(EMAIL_RCP = ).+#\1\"$EMAIL_RCP\"#' master/master.cfg"
-run "buildslave create-slave slave $SLAVE_SOCKET $SLAVE_NAME $BUILDBOT_PWD"
-run "echo 'export DOCKER_CREDS=\"$REGISTRY_USER:$REGISTRY_PWD\"' > $BUILDBOT_PATH/master/credentials.cfg"
-run "echo 'export S3_BUCKET=\"$REGISTRY_BUCKET\"' >> $BUILDBOT_PATH/master/credentials.cfg"
-run "echo 'export S3_ACCESS_KEY=\"$REGISTRY_ACCESS_KEY\"' >> $BUILDBOT_PATH/master/credentials.cfg"
-run "echo 'export S3_SECRET_KEY=\"$REGISTRY_SECRET_KEY\"' >> $BUILDBOT_PATH/master/credentials.cfg"
-
-# Patch github webstatus to capture pull requests
-cp $CFG_PATH/github.py /usr/local/lib/python2.7/dist-packages/buildbot/status/web/hooks
-
-# Allow buildbot subprocesses (docker tests) to properly run in containers,
-# in particular with docker -u
-run "sed -i 's/^umask = None/umask = 000/' slave/buildbot.tac"
-
-# Setup supervisor
-cp $CFG_PATH/buildbot.conf /etc/supervisor/conf.d/buildbot.conf
-sed -i -E "s/^chmod=0700.+/chmod=0770\nchown=root:$USER/" /etc/supervisor/supervisord.conf
-kill -HUP $(pgrep -f "/usr/bin/python /usr/bin/supervisord")

hack/infrastructure/docker-ci/dcr/prod/docker-ci.yml

@@ -0,0 +1,22 @@
+docker-ci:
+  image: "docker-ci/docker-ci"
+  release_name: "docker-ci-0.5.6"
+  ports: ["80","2222:22","8011:8011"]
+  register: "80"
+  volumes: ["/run:/var/socket","/home/docker-ci:/data/docker-ci"]
+  command: []
+  env:
+    - "DEPLOYMENT=production"
+    - "IRC_CHANNEL=docker-testing"
+    - "BACKUP_BUCKET=backup-ci"
+    - "$WEB_USER"
+    - "$WEB_IRC_PWD"
+    - "$BUILDBOT_PWD"
+    - "$AWS_ACCESS_KEY"
+    - "$AWS_SECRET_KEY"
+    - "$GPG_PASSPHRASE"
+    - "$BACKUP_AWS_ID"
+    - "$BACKUP_AWS_SECRET"
+    - "$SMTP_USER"
+    - "$SMTP_PWD"
+    - "$EMAIL_RCP"

hack/infrastructure/docker-ci/dcr/prod/settings.yml

@@ -0,0 +1,5 @@
+default:
+  hipaches: ['192.168.100.67:6379']
+  daemons: ['192.168.100.67:4243']
+  use_ssh: False
+

hack/infrastructure/docker-ci/dcr/stage/docker-ci.yml

@@ -0,0 +1,22 @@
+docker-ci:
+  image: "docker-ci/docker-ci"
+  release_name: "docker-ci-stage"
+  ports: ["80","2222:22","8011:8011"]
+  register: "80"
+  volumes: ["/run:/var/socket","/home/docker-ci:/data/docker-ci"]
+  command: []
+  env:
+    - "DEPLOYMENT=staging"
+    - "IRC_CHANNEL=docker-testing-staging"
+    - "BACKUP_BUCKET=ci-backup-stage"
+    - "$BACKUP_AWS_ID"
+    - "$BACKUP_AWS_SECRET"
+    - "$WEB_USER"
+    - "$WEB_IRC_PWD"
+    - "$BUILDBOT_PWD"
+    - "$AWS_ACCESS_KEY"
+    - "$AWS_SECRET_KEY"
+    - "$GPG_PASSPHRASE"
+    - "$SMTP_USER"
+    - "$SMTP_PWD"
+    - "$EMAIL_RCP"

hack/infrastructure/docker-ci/dcr/stage/settings.yml

@@ -0,0 +1,5 @@
+default:
+  hipaches: ['192.168.100.65:6379']
+  daemons: ['192.168.100.65:4243']
+  use_ssh: False
+

hack/infrastructure/docker-ci/deployment.py

@@ -1,171 +0,0 @@
-#!/usr/bin/env python
-
-import os, sys, re, json, requests, base64
-from subprocess import call
-from fabric import api
-from fabric.api import cd, run, put, sudo
-from os import environ as env
-from datetime import datetime
-from time import sleep
-
-# Remove SSH private key as it needs more processing
-CONFIG = json.loads(re.sub(r'("DOCKER_CI_KEY".+?"(.+?)",)','',
-    env['CONFIG_JSON'], flags=re.DOTALL))
-
-# Populate environment variables
-for key in CONFIG:
-    env[key] = CONFIG[key]
-
-# Load SSH private key
-env['DOCKER_CI_KEY'] = re.sub('^.+"DOCKER_CI_KEY".+?"(.+?)".+','\\1',
-    env['CONFIG_JSON'],flags=re.DOTALL)
-
-DROPLET_NAME = env.get('DROPLET_NAME','docker-ci')
-TIMEOUT = 120            # Seconds before timeout droplet creation
-IMAGE_ID = 1004145       # Docker on Ubuntu 13.04
-REGION_ID = 4            # New York 2
-SIZE_ID = 62             # memory 2GB
-DO_IMAGE_USER = 'root'   # Image user on Digital Ocean
-API_URL = 'https://api.digitalocean.com/'
-DOCKER_PATH = '/go/src/github.com/dotcloud/docker'
-DOCKER_CI_PATH = '/docker-ci'
-CFG_PATH = '{}/buildbot'.format(DOCKER_CI_PATH)
-
-
-class DigitalOcean():
-
-    def __init__(self, key, client):
-        '''Set default API parameters'''
-        self.key = key
-        self.client = client
-        self.api_url = API_URL
-
-    def api(self, cmd_path, api_arg={}):
-        '''Make api call'''
-        api_arg.update({'api_key':self.key, 'client_id':self.client})
-        resp = requests.get(self.api_url + cmd_path, params=api_arg).text
-        resp = json.loads(resp)
-        if resp['status'] != 'OK':
-            raise Exception(resp['error_message'])
-        return resp
-
-    def droplet_data(self, name):
-        '''Get droplet data'''
-        data = self.api('droplets')
-        data = [droplet for droplet in data['droplets']
-            if droplet['name'] == name]
-        return data[0] if data else {}
-
-
-def json_fmt(data):
-    '''Format json output'''
-    return json.dumps(data, sort_keys = True, indent = 2)
-
-
-do = DigitalOcean(env['DO_API_KEY'], env['DO_CLIENT_ID'])
-
-# Get DROPLET_NAME data
-data = do.droplet_data(DROPLET_NAME)
-
-# Stop processing if DROPLET_NAME exists on Digital Ocean
-if data:
-    print ('Droplet: {} already deployed. Not further processing.'
-        .format(DROPLET_NAME))
-    exit(1)
-
-# Create droplet
-do.api('droplets/new', {'name':DROPLET_NAME, 'region_id':REGION_ID,
-    'image_id':IMAGE_ID, 'size_id':SIZE_ID,
-    'ssh_key_ids':[env['DOCKER_KEY_ID']]})
-
-# Wait for droplet to be created.
-start_time = datetime.now()
-while (data.get('status','') != 'active' and (
- datetime.now()-start_time).seconds < TIMEOUT):
-    data = do.droplet_data(DROPLET_NAME)
-    print data['status']
-    sleep(3)
-
-# Wait for the machine to boot
-sleep(15)
-
-# Get droplet IP
-ip = str(data['ip_address'])
-print 'droplet: {}    ip: {}'.format(DROPLET_NAME, ip)
-
-# Create docker-ci ssh private key so docker-ci docker container can communicate
-# with its EC2 instance
-os.makedirs('/root/.ssh')
-open('/root/.ssh/id_rsa','w').write(env['DOCKER_CI_KEY'])
-os.chmod('/root/.ssh/id_rsa',0600)
-open('/root/.ssh/config','w').write('StrictHostKeyChecking no\n')
-
-api.env.host_string = ip
-api.env.user = DO_IMAGE_USER
-api.env.key_filename = '/root/.ssh/id_rsa'
-
-# Correct timezone
-sudo('echo "America/Los_Angeles" >/etc/timezone')
-sudo('dpkg-reconfigure --frontend noninteractive tzdata')
-
-# Load public docker-ci key
-sudo("echo '{}' >> /root/.ssh/authorized_keys".format(env['DOCKER_CI_PUB']))
-
-# Create docker nightly release credentials file
-credentials = {
-    'AWS_ACCESS_KEY': env['PKG_ACCESS_KEY'],
-    'AWS_SECRET_KEY': env['PKG_SECRET_KEY'],
-    'GPG_PASSPHRASE': env['PKG_GPG_PASSPHRASE']}
-open(DOCKER_CI_PATH + '/nightlyrelease/release_credentials.json', 'w').write(
-    base64.b64encode(json.dumps(credentials)))
-
-# Transfer docker
-sudo('mkdir -p ' + DOCKER_CI_PATH)
-sudo('chown {}.{} {}'.format(DO_IMAGE_USER, DO_IMAGE_USER, DOCKER_CI_PATH))
-call('/usr/bin/rsync -aH {} {}@{}:{}'.format(DOCKER_CI_PATH, DO_IMAGE_USER, ip,
-    os.path.dirname(DOCKER_CI_PATH)), shell=True)
-
-# Install Docker and Buildbot dependencies
-sudo('mkdir /mnt/docker; ln -s /mnt/docker /var/lib/docker')
-sudo('apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9')
-sudo('echo deb https://get.docker.io/ubuntu docker main >'
-    ' /etc/apt/sources.list.d/docker.list')
-sudo('echo -e "deb http://archive.ubuntu.com/ubuntu raring main universe\n'
-    'deb http://us.archive.ubuntu.com/ubuntu/ raring-security main universe\n"'
-    ' > /etc/apt/sources.list; apt-get update')
-sudo('DEBIAN_FRONTEND=noninteractive apt-get install -q -y wget python-dev'
-    ' python-pip supervisor git mercurial linux-image-extra-$(uname -r)'
-    ' aufs-tools make libfontconfig libevent-dev libsqlite3-dev libssl-dev')
-sudo('wget -O - https://go.googlecode.com/files/go1.2.linux-amd64.tar.gz | '
-    'tar -v -C /usr/local -xz; ln -s /usr/local/go/bin/go /usr/bin/go')
-sudo('GOPATH=/go go get -d github.com/dotcloud/docker')
-sudo('pip install -r {}/requirements.txt'.format(CFG_PATH))
-
-# Install docker and testing dependencies
-sudo('apt-get install -y -q lxc-docker')
-sudo('curl -s https://phantomjs.googlecode.com/files/'
-    'phantomjs-1.9.1-linux-x86_64.tar.bz2 | tar jx -C /usr/bin'
-    ' --strip-components=2 phantomjs-1.9.1-linux-x86_64/bin/phantomjs')
-
-# Build docker-ci containers
-sudo('cd {}; docker build -t docker .'.format(DOCKER_PATH))
-sudo('cd {}; docker build -t docker-ci .'.format(DOCKER_CI_PATH))
-sudo('cd {}/nightlyrelease; docker build -t dockerbuilder .'.format(
-    DOCKER_CI_PATH))
-sudo('cd {}/registry-coverage; docker build -t registry_coverage .'.format(
-    DOCKER_CI_PATH))
-
-# Download docker-ci testing container
-sudo('docker pull mzdaniel/test_docker')
-
-# Setup buildbot
-sudo('mkdir /data')
-sudo('{0}/setup.sh root {0} {1} {2} {3} {4} {5} {6} {7} {8} {9} {10}'
-    ' {11} {12}'.format(CFG_PATH, DOCKER_PATH, env['BUILDBOT_PWD'],
-    env['IRC_PWD'], env['IRC_CHANNEL'], env['SMTP_USER'],
-    env['SMTP_PWD'], env['EMAIL_RCP'], env['REGISTRY_USER'],
-    env['REGISTRY_PWD'], env['REGISTRY_BUCKET'], env['REGISTRY_ACCESS_KEY'],
-    env['REGISTRY_SECRET_KEY']))
-
-# Preventively reboot docker-ci daily
-sudo('ln -s /sbin/reboot /etc/cron.daily')