Currently, if you have the following images:
foo/bar 1 23b27d50fb49
foo/bar 2 f2b86ec3fcc4
And you issue the following command:
docker tag foo/bar:2 foo/bar latest
docker will tag the "wrong" image, because the image id for foo/bar:1 starts
with a "2". That is, you'll end up with the following:
foo/bar 1 23b27d50fb49
foo/bar 2 f2b86ec3fcc4
foo/bar latest 23b27d50fb49
This commit reverses the priority given to tags vs. image ids in the
construction `<user>/<repo>:<tagOrId>`, meaning that if a tag that is an exact
match for the specified `tagOrId`, it will be tagged in preference to an image
with an id that happens to start with the correct character sequence.
Prior this commit, 'docker images' and other cmd's, which used utils.HumanSize(),
showed unnecessary whitespaces.
Formatting of progress has been moved to FormatProgess(), justifing the string
directly in the template.
- For the TCP test try again if socat wasn't listening yet;
- For the UDP test raise the timeout to a minute to workaround what
seems to be an issue with Linux.
API Changes
-----------
The port notation is extended to support "/udp" or "/tcp" at the *end*
of the specifier string (and defaults to tcp if "/tcp" or "/udp" are
missing)
`docker ps` now shows UDP ports as "frontend->backend/udp". Nothing
changes for TCP ports.
`docker inspect` now displays two sub-dictionaries: "Tcp" and "Udp",
under "PortMapping" in "NetworkSettings".
Theses changes stand true for the values returned by the HTTP API too.
This changeset will definitely break tools built upon the API (or upon
`docker inspect`). A less intrusive way to add UDP ports in `docker
inspect` would be to simply add "/udp" for UDP ports but it will still
break existing applications which tries to convert the whole field to an
integer. I believe that having two TCP/UDP sub-dictionaries is better
because it makes the whole thing more clear and more easy to parse right
away (i.e: you don't have to check the format of the string, split it
and convert the right part to an integer)
Code Changes
------------
Significant changes in network.go:
- A second PortAllocator is instantiated for the UDP range;
- PortMapper maintains separate mapping for TCP and UDP;
- The extPorts array in NetworkInterface is now an array of Nat objects
(so we can know on which protocol a given port was mapped when
NetworkInterface.Release() is called);
- TCP proxying on localhost has been moved away in network_proxy.go.
localhost proxy code rewrite in network_proxy.go:
We have to proxy the traffic between localhost:frontend-port and
container:backend-port because Netfilter doesn't work properly on the
loopback interface and DNAT iptable rules aren't applied there.
- Goroutines in the TCP proxying code are now explicitly stopped when
the proxy is stopped;
- UDP connection tracking using a map (more infos in [1]);
- Support for IPv6 (to be more accurate, the code is transparent to the
Go net package, so you can use, tcp/tcp4/tcp6/udp/udp4/udp6);
- Single Proxy interface for both UDP and TCP proxying;
- Full test suite.
[1] https://github.com/dotcloud/docker/issues/33#issuecomment-20010400
writing out streamed status.
This is caused by a Buffering message that is not in the correct json format:
[...]
{"status"
:"Pushing 6bba11a28f1ca247de9a47071355ce5923a45b8fea3182389f992f4
24b93edae"}Buffering to disk 244/? (n/a)..
{"status":"Pushing",[...]
The "Buffering to disk" message is originated in
srv.runtime.graph.TempLayerArchive
I am now using the StreamFormatter provided by the context from which the
method is called.
For structs protected by a single mutex, embed the mutex for more
concise usage.
Also use a sync.Mutex directly, rather than a pointer, to avoid the
need for initialization (because a Mutex's zero-value is valid and
ready to be used).
- Fix TestGetImagesJSON when there is more than one image in the test
repository;
- Remove an hardcoded constant use in TestGetImagesByName;
- Wait in a loop in TestKillDifferentUser;
- Use env instead of /usr/bin/env in TestEnv;
- Create a daemon user in contrib/mkimage-unittest.sh.
It's a fork of the mkimage-busybox.sh script and it adds socat to the
image. (socat being needed to add udp support, see #33).
This script, like mkimage-busybox.sh, probably only works on
Debian/Ubuntu.
The dockerbuilder Dockerfile was installing one package per apt-get
install operation.
This changes it so that consecutive run apt-get install operations are
batched into a single operation.
As a user who has blown $150 on VMWare Fusion and vagrant-vmware, I
would like to use my new shiny to hack on Docker. Docker already has a
multi-provider Vagrantfile, so adding another one presents little risk.
Known Issues:
- The docker install of a new kernel breaks the Vagrant shared folder.
- This seems to be because the VMWare hgfs module doesn't build
against a 3.8 kernel.
- I don't believe that shared folder support is actually in use
* replaced previously removed concepts/containers and concepts/introcution by a redirect
* moved orphan index/varable to workingwiththerepository
* added favicon to the layout.html
* added redirect_home which is a http refresh redirect. It works like a 301 for google
* fixed an issue in the layout that would make it break when on small screens
into the container instead of copying them as a regular file.
* Builder: ADD uses tar/untar for copies instead of calling 'cp -ar'.
This is more consistent, reduces the number of dependencies, and
fixe #896.
Move creating the router and populating the
routes to a separate function outside of
ListenAndServe to allow unit tests to make
assertions on the configured routes and handler
funcs.
Add the Access-Control-Allow-Methods header so that
DELETE operations are allowed.
Also move the write CORS headers method before
docker writes a 404 not found so that the client
receives the correct response and not an invalid
CORS request.
seperated the registry and index API's into their own docs and merged
in the index search api into the index api. Also renamed the original
registry api to registry_index_spec.
This updates the documentation to mention that:
1. a lot of data may get sent to the docker daemon if there is a lot of
data in the directory passed to docker build
2. ADD doesn't work in the absence of the context
3. running without a context doesn't send file data to the docker
daemon
4. explain that the data sent to the docker daemon will be used by ADD
commands
Add the -api-enable-cors flag when running docker
in daemon mode to allow CORS requests to be made to
the Remote Api. The default value is false for this
flag to not allow cross origin request to be made.
Also added a handler for OPTIONS requests the standard
for cross domain requests is to initially make an
OPTIONS request to the api.
Fall back to image-specified hostname if user doesn't
provide one, instead of only using image-specified
hostname if the user *does* try to set one.
(ditto for username)
Closes#694.
Usage: docker COMMAND [arg...]
A self-sufficient runtime for linux containers.
Commands:
attach Attach to a running container
insert Insert a file in an image
login Register or Login to the docker registry server
export Stream the contents of a container as a tar archive
diff Inspect changes on a container's filesystem
logs Fetch the logs of a container
pull Pull an image or a repository from the docker registry server
restart Restart a running container
build Build a container from Dockerfile or via stdin
history Show the history of an image
kill Kill a running container
rmi Remove an image
start Start a stopped container
tag Tag an image into a repository
commit Create a new image from a container's changes
import Create a new filesystem image from the contents of a tarball
ps List containers
rm Remove a container
run Run a command in a new container
wait Block until a container stops, then print its exit code
images List images
port Lookup the public-facing port which is NAT-ed to PRIVATE_PORT
info Display system-wide information
inspect Return low-level information on a container
push Push an image or a repository to the docker registry server
search Search for an image in the docker index
stop Stop a running container
version Show the docker version information back
* Moved the introduction page to the en/latest/index.html home of the documentation and pointed all links there
* Fixed a broken link from+to homepage
* Fixed the javascript of the documentation navigation to allow expand and collapse multiple times.
* Fixed the one typo Andy pointed out
+ Documentation: Added install instructions for RackSpace cloud using Ubuntu 12.04, 12.10, 13.04 and fixed a few minor issues with the ubuntu install docs.
This should make previewing documentation easier.
Also updated the Makefile to now copy the theme dir into the _build/website/ dir. Make connect and Make push work.
Specifically, Ubuntu Precise's cgroup-lite script uses mount -n
to mount the cgroup filesystems so they don't appear in mtab, so
detection always fails unless the admin updates mtab with /proc/mounts.
/proc/mounts is valid on just about every Linux machine in existence and
as a bonus is much easier to parse.
I also removed the regex in favor of a more accurate parser that should
also support monolitic cgroup mounts (e.g. mount -t cgroup none /cgroup).
On Gentoo, the memory cgroup is mounted at /sys/fs/cgroup/memory, but the mount line looks like the following:
memory on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
(note that the first word on the line is "memory", not "cgroup", but the other essentials are there, namely the type of cgroup and the memory mount option)
This is especially to fix the current docker on kernels such as gentoo-sources, where the "flavor" is the string "gentoo", and that obviously fails to be converted to an integer.
* Addded quick install on ubuntu as the 1st install option
* Grouped other binary installs under "binary installs"
* Removed duplicate binary ubuntu installs (linked to the docs)
* Improved "build from source" instructions
Ensure the docker daemon creates a file containing its PID under
/var/run/docker.pid.
The daemon takes care of removing the pid file when it receives either
SIGTERM, SIGINT or SIGKILL.
The daemon also refuses to start when the pidfile is found. An
explanation message is shown to the user when this happens.
This change is required to make docker easier to manage by tools like
checkproc which rely on this information.
ip from iproute2 replaces the legacy route tool which is often not
installed by default on recent Linux distributions.
The same patch has been done in network.go and is re-used here.
The raw mode is actually only needed when you attach to a container.
Having it enabled all the time can be a pain, e.g: if docker crashes
your terminal will end up in a broken state.
Since we are currently missing a real API for the docker daemon to
negotiate this kind of options, this changeset actually enable the raw
mode on the login (because it outputs a password), run and attach
commands.
This "optional raw mode" is implemented by passing a more complicated
interface than io.Writer as the stdout argument of each command. This
interface (DockerConn) exposes a method which allows the command to set
the terminal in raw mode or not.
Finally, the code added by this changeset will be deprecated by a real
API for the docker daemon.
Broken link was from python_web_app to nonexistent "base commands page"; updated to point to next item in examples menu, running_ssh_service screencast.
At least, for me, 'map' means that there are two values and one is "mapped" to
another.
In this case, just one value is provided (container's port), the other value is
automatically obtained (host's port) and the actual mapping can be seen using
``docker port`` command.
Not only is this a more common idiom, it'll make finding bugs easier,
and it'll make porting to Go 1.1 easier.
Go 1.1 will not require the final return or panic because it has a
notion of terminating statements.
Instead of allocating all possible IPs in advance, generate them as
needed.
A loop will cycle through all possible IPs in sequential order,
allocating them as needed and marking them as in use. Once the loop
exhausts all IPs, it will wrap back to the beginning. IPs that are
already in use will be skipped. When an IP is released, it will be
cleared and be available for allocation again.
Two decisions went into this design:
1) Minimize memory footprint by only allocating IPs that are actually
in use
2) Minimize reuse of released IP addresses to avoid sending traffic to
the wrong containers
As a side effect, the functions for IP/Mask<->int conversion have been
rewritten to never be able to fail in order to reduce the amount of
error returns.
Fixes gh-231
It was broken because the terminal is in raw mode.
This changeset adds code in the login commmand to do a little bit of
interpretation on the user input (something usually done by the terminal
emulator itself).
As per FIXME, CmdStream could have deadlocked if a command printed
enough on stderr. This commit fixes that, but still keeps all of
the stderr output in memory.
This sets up an idiomatic Go workspace in /opt/go with the source
shared from the host directory in
/opt/go/src/github.com/dotcloud/docker and docker installed into
/opt/go
Pull requests descriptions should be as clear as possible and include a
reference to all the issues that they address.
Code review comments may be added to your pull request. Discuss, then make the
suggested modifications and push additional commits to your feature branch. Be
sure to post a comment after pushing. The new commits will show up in the pull
request automatically, but the reviewers will not be notified unless you
comment.
Before the pull request is merged, make sure that you squash your commits into
logical units of work using `git rebase -i` and `git push -f`. After every
commit the test suite should be passing. Include documentation changes in the
same commit so that a revert would remove all traces of the feature or fix.
Commits that fix or close an issue should include a reference like `Closes #XXX`
or `Fixes #XXX`, which will automatically close the issue when merged.
Add your name to the AUTHORS file, but make sure the list is sorted and your
name and email address match your git configuration. The AUTHORS file is
regenerated occasionally from the git commit history, so a mismatch may result
in your changes being overwritten.
## Decision process
### How are decisions made?
Short answer: with pull requests to the docker repository.
Docker is an open-source project with an open design philosophy. This means that the repository is the source of truth for EVERY aspect of the project,
including its philosophy, design, roadmap and APIs. *If it's part of the project, it's in the repo. It's in the repo, it's part of the project.*
As a result, all decisions can be expressed as changes to the repository. An implementation change is a change to the source code. An API change is a change to
the API specification. A philosophy change is a change to the philosophy manifesto. And so on.
All decisions affecting docker, big and small, follow the same 3 steps:
* Step 1: Open a pull request. Anyone can do this.
* Step 2: Discuss the pull request. Anyone can do this.
* Step 3: Accept or refuse a pull request. The relevant maintainer does this (see below "Who decides what?")
### Who decides what?
So all decisions are pull requests, and the relevant maintainer makes the decision by accepting or refusing the pull request.
But how do we identify the relevant maintainer for a given pull request?
Docker follows the timeless, highly efficient and totally unfair system known as [Benevolent dictator for life](http://en.wikipedia.org/wiki/Benevolent_Dictator_for_Life),
with yours truly, Solomon Hykes, in the role of BDFL.
This means that all decisions are made by default by me. Since making every decision myself would be highly unscalable, in practice decisions are spread across multiple maintainers.
The relevant maintainer for a pull request is assigned in 3 steps:
* Step 1: Determine the subdirectory affected by the pull request. This might be src/registry, docs/source/api, or any other part of the repo.
* Step 2: Find the MAINTAINERS file which affects this directory. If the directory itself does not have a MAINTAINERS file, work your way up the the repo hierarchy until you find one.
* Step 3: The first maintainer listed is the primary maintainer. The pull request is assigned to him. He may assign it to other listed maintainers, at his discretion.
### I'm a maintainer, should I make pull requests too?
Primary maintainers are not required to create pull requests when changing their own subdirectory, but secondary maintainers are.
### Who assigns maintainers?
Solomon.
### How can I become a maintainer?
* Step 1: learn the component inside out
* Step 2: make yourself useful by contributing code, bugfixes, support etc.
* Step 3: volunteer on the irc channel (#docker@freenode)
Don't forget: being a maintainer is a time investment. Make sure you will have time to make yourself available.
You don't have to be a maintainer to make a difference on the project!
### What are a maintainer's responsibility?
It is every maintainer's responsibility to:
* 1) Expose a clear roadmap for improving their component.
* 2) Deliver prompt feedback and decisions on pull requests.
* 3) Be available to anyone with questions, bug reports, criticism etc. on their component. This includes irc, github requests and the mailing list.
* 4) Make sure their component respects the philosophy, design and roadmap of the project.
### How is this process changed?
Just like everything else: by making a pull request :)
Docker complements LXC with a high-level API which operates at the process level. It runs unix processes with strong guarantees of isolation and repeatability across servers.
Docker is an open-source engine which automates the deployment of applications as highly portable, self-sufficient containers.
Docker is a great building block for automating distributed systems: large-scale web deployments, database clusters, continuous deployment systems, private PaaS, service-oriented architectures, etc.
Docker containers are both *hardware-agnostic* and *platform-agnostic*. This means that they can run anywhere, from your
laptop to the largest EC2 compute instance and everything in between - and they don't require that you use a particular
language, framework or packaging system. That makes them great building blocks for deploying and scaling web apps, databases
and backend services without depending on a particular stack or provider.
Docker is an open-source implementation of the deployment engine which powers [dotCloud](http://dotcloud.com), a popular Platform-as-a-Service.
It benefits directly from the experience accumulated over several years of large-scale operation and support of hundreds of thousands
of applications and databases.
* *Heterogeneous payloads*: any combination of binaries, libraries, configuration files, scripts, virtualenvs, jars, gems, tarballs, you name it. No more juggling between domain-specific tools. Docker can deploy and run them all.
* *Any server*: docker can run on any x64 machine with a modern linux kernel - whether it's a laptop, a bare metal server or a VM. This makes it perfect for multi-cloud deployments.
## Better than VMs
* *Isolation*: docker isolates processes from each other and from the underlying host, using lightweight containers.
A common method for distributing applications and sandbox their execution is to use virtual machines, or VMs. Typical VM formats
are VMWare's vmdk, Oracle Virtualbox's vdi, and Amazon EC2's ami. In theory these formats should allow every developer to
automatically package their application into a "machine" for easy distribution and deployment. In practice, that almost never
happens, for a few reasons:
* *Repeatability*: because containers are isolated in their own filesystem, they behave the same regardless of where, when, and alongside what they run.
* *Size*: VMs are very large which makes them impractical to store and transfer.
* *Performance*: running VMs consumes significant CPU and memory, which makes them impractical in many scenarios, for example local development of multi-tier applications, and
large-scale deployment of cpu and memory-intensive applications on large numbers of machines.
* *Portability*: competing VM environments don't play well with each other. Although conversion tools do exist, they are limited and add even more overhead.
* *Hardware-centric*: VMs were designed with machine operators in mind, not software developers. As a result, they offer very limited tooling for what developers need most:
building, testing and running their software. For example, VMs offer no facilities for application versioning, monitoring, configuration, logging or service discovery.
By contrast, Docker relies on a different sandboxing method known as *containerization*. Unlike traditional virtualization,
containerization takes place at the kernel level. Most modern operating system kernels now support the primitives necessary
for containerization, including Linux with [openvz](http://openvz.org), [vserver](http://linux-vserver.org) and more recently [lxc](http://lxc.sourceforge.net),
Solaris with [zones](http://docs.oracle.com/cd/E26502_01/html/E29024/preface-1.html#scrolltoc) and FreeBSD with [Jails](http://www.freebsd.org/doc/handbook/jails.html).
Docker builds on top of these low-level primitives to offer developers a portable format and runtime environment that solves
all 4 problems. Docker containers are small (and their transfer can be optimized with layers), they have basically zero memory and cpu overhead,
they are completely portable and are designed from the ground up with an application-centric design.
The best part: because docker operates at the OS level, it can still be run inside a VM!
## Plays well with others
Docker does not require that you buy into a particular programming language, framework, packaging system or configuration language.
Is your application a unix process? Does it use files, tcp connections, environment variables, standard unix streams and command-line
arguments as inputs and outputs? Then docker can run it.
Can your application's build be expressed as a sequence of such commands? Then docker can build it.
Notable features
-----------------
## Escape dependency hell
* Filesystem isolation: each process container runs in a completely separate root filesystem.
A common problem for developers is the difficulty of managing all their application's dependencies in a simple and automated way.
* Resource isolation: system resources like cpu and memory can be allocated differently to each process container, using cgroups.
This is usually difficult for several reasons:
*Network isolation: each process container runs in its own network namespace, with a virtual interface and IP address of its own.
**Cross-platform dependencies*. Modern applications often depend on a combination of system libraries and binaries, language-specific packages, framework-specific modules,
internal components developed for another project, etc. These dependencies live in different "worlds" and require different tools - these tools typically don't work
well with each other, requiring awkward custom integrations.
* Copy-on-write: root filesystems are created using copy-on-write, which makes deployment extremeley fast, memory-cheap and disk-cheap.
* Logging: the standard streams (stdout/stderr/stdin) of each process container are collected and logged for real-time or batch retrieval.
* Change management: changes to a container's filesystem can be committed into a new image and re-used to create more containers. No templating or manual configuration required.
* Interactive shell: docker can allocate a pseudo-tty and attach to the standard input of any container, for example to run a throwaway interactive shell.
* Conflicting dependencies. Different applications may depend on different versions of the same dependency. Packaging tools handle these situations with various degrees of ease -
but they all handle them in different and incompatible ways, which again forces the developer to do extra work.
* Custom dependencies. A developer may need to prepare a custom version of their application's dependency. Some packaging systems can handle custom versions of a dependency,
others can't - and all of them handle it differently.
Docker solves dependency hell by giving the developer a simple way to express *all* their application's dependencies in one place,
and streamline the process of assembling them. If this makes you think of [XKCD 927](http://xkcd.com/927/), don't worry. Docker doesn't
*replace* your favorite packaging systems. It simply orchestrates their use in a simple and repeatable way. How does it do that? With layers.
Docker defines a build as running a sequence of unix commands, one after the other, in the same container. Build commands modify the contents of the container
(usually by installing new files on the filesystem), the next command modifies it some more, etc. Since each build command inherits the result of the previous
commands, the *order* in which the commands are executed expresses *dependencies*.
Here's a typical docker build process:
```bash
from ubuntu:12.10
run apt-get update
run DEBIAN_FRONTEND=noninteractive apt-get install -q -y python
run DEBIAN_FRONTEND=noninteractive apt-get install -q -y python-pip
run pip install django
run DEBIAN_FRONTEND=noninteractive apt-get install -q -y curl
run curl -L https://github.com/shykes/helloflask/archive/master.tar.gz | tar -xzv
run cd helloflask-master && pip install -r requirements.txt
```
Note that Docker doesn't care *how* dependencies are built - as long as they can be built by running a unix command in a container.
Install instructions
==================
Quick install on Ubuntu 12.04 and 12.10
---------------------------------------
```bash
curl get.docker.io | sudo sh -x
```
Binary installs
----------------
Docker supports the following binary installation methods.
Note that some methods are community contributions and not yet officially supported.
* [Ubuntu 12.04 and 12.10 (officially supported)](http://docs.docker.io/en/latest/installation/ubuntulinux/)
JOB=$(docker run -d base /bin/sh -c "while true; do echo Hello world; sleep 1; done")
# Collect the output of the job so far
docker logs $JOB
# Kill the job
docker kill $JOB
```
Listing all running containers
------------------------------
```bash
docker ps
```
Expose a service on a TCP port
------------------------------
```bash
# Expose port 4444 of this container, and tell netcat to listen on it
JOB=$(docker run -d -p 4444 base /bin/nc -l -p 4444)
# Which public port is NATed to my container?
PORT=$(docker port $JOB 4444)
# Connect to the public port via the host's public address
echo hello world | nc $(hostname) $PORT
# Verify that the network connection worked
echo "Daemon received: $(docker logs $JOB)"
```
Contributing to Docker
======================
Want to hack on Docker? Awesome! Here are instructions to get you started. They are probably not perfect, please let us know if anything feels wrong or incomplete.
Want to hack on Docker? Awesome! There are instructions to get you started on the website: http://docs.docker.io/en/latest/contributing/contributing/
Contribution guidelines
-----------------------
### Pull requests are always welcome
We are always thrilled to receive pull requests, and do our best to process them as fast as possible. Not sure if that typo is worth a pull request? Do it! We will appreciate it.
If your pull request is not accepted on the first try, don't be discouraged! If there's a problem with the implementation, hopefully you received feedback on what to improve.
We're trying very hard to keep Docker lean and focused. We don't want it to do everything for everybody. This means that we might decide against incorporating a new feature.
However, there might be a way to implement that feature *on top of* docker.
### Discuss your design on the mailing list
We recommend discussing your plans [on the mailing list](https://groups.google.com/forum/?fromgroups#!forum/docker-club) before starting to code - especially for more ambitious contributions. This gives other contributors a chance to point
you in the right direction, give feedback on your design, and maybe point out if someone else is working on the same thing.
### Create issues...
Any significant improvement should be documented as [a github issue](https://github.com/dotcloud/docker/issues) before anybody starts working on it.
### ...but check for existing issues first!
Please take a moment to check that an issue doesn't already exist documenting your bug report or improvement proposal.
If it does, it never hurts to add a quick "+1" or "I have this problem too". This will help prioritize the most common problems and requests.
They are probably not perfect, please let us know if anything feels wrong or incomplete.
### Write tests
Note
----
Golang has a great testing suite built in: use it! Take a look at existing tests for inspiration.
We also keep the documentation in this repository. The website documentation is generated using sphinx using these sources.
Please find it under docs/sources/ and read more about it https://github.com/dotcloud/docker/tree/master/docs/README.md
Please feel free to fix / update the documentation and send us pull requests. More tutorials are also welcome.
Setting up a dev environment
@@ -227,14 +263,14 @@ Setting up a dev environment
Instructions that have been verified to work on Ubuntu 12.10,
@@ -258,7 +294,7 @@ a format that is self-describing and portable, so that any compliant runtime can
The spec for Standard Containers is currently a work in progress, but it is very straightforward. It mostly defines 1) an image format, 2) a set of standard operations, and 3) an execution environment.
A great analogy for this is the shipping container. Just like Standard Containers are a fundamental unit of software delivery, shipping containers (http://bricks.argz.com/ins/7823-1/12) are a fundamental unit of physical delivery.
A great analogy for this is the shipping container. Just like how Standard Containers are a fundamental unit of software delivery, shipping containers (http://bricks.argz.com/ins/7823-1/12) are a fundamental unit of physical delivery.
### 1. STANDARD OPERATIONS
@@ -286,7 +322,7 @@ Similarly, before Standard Containers, by the time a software component ran in p
### 5. INDUSTRIAL-GRADE DELIVERY
There are 17 million shipping containers in existence, packed with every physical good imaginable. Every single one of them can be loaded on the same boats, by the same cranes, in the same facilities, and sent anywhere in the World with incredible efficiency. It is embarrassing to think that a 30 ton shipment of coffee can safely travel half-way across the World in *less time* than it takes a software team to deliver its code from one datacenter to another sitting 10 miles away.
There are 17 million shipping containers in existence, packed with every physical good imaginable. Every single one of them can be loaded onto the same boats, by the same cranes, in the same facilities, and sent anywhere in the World with incredible efficiency. It is embarrassing to think that a 30 ton shipment of coffee can safely travel half-way across the World in *less time* than it takes a software team to deliver its code from one datacenter to another sitting 10 miles away.
With Standard Containers we can put an end to that embarrassment, by making INDUSTRIAL-GRADE DELIVERY of software a reality.
@@ -336,4 +372,10 @@ Standard Container Specification
#### Security
### Legal
Transfers of Docker shall be in accordance with applicable export controls of any country and all other applicable
legal requirements. Docker shall not be distributed or downloaded to or in Cuba, Iran, North Korea, Sudan or Syria
and shall not be distributed or downloaded to any person on the Denied Persons List administered by the U.S.
An image is a root filesystem + some metadata. It is uniquely identified by a SHA256, and it can be given a symbolic name as well (so you can `docker run mystuff` instead of `docker run 819f04e5706f5...`.
The metadata is a JSON blob. It contains at least:
- the hash of the parent (only if the image is based on another image; if it was created from scratch, then the parent is `null`),
- the creation date (this is when the `docker commit` command was done).
The hash of the image is defined as:
`SHA256(SHA256(jsondata)+SHA256(tarball))`
When you run something into an image, it automatically creates a container. The container has a unique ID, and when you `docker commit <container_id> mystuff`, you are creating a new image, and giving it the nickname `mystuff`.
Repository
----------
A repository:
- belongs to a specific user,
- has a given name chosen by the user,
- is a set of tagged images.
The typical use case is to group different versions of something under a repository.
Example: you are John Doe, maintainer of a collection of PostgreSQL images based on different releases of Ubuntu. Your docker ID is `jdoe`; you decide that the repository name will by `pgsql`. You pull a bunch of base images for the different Ubuntu releases, then you setup different versions of PostgreSQL in them. You end up with the following set of images:
- a base lucid image,
- a base precise image,
- a base quantal image,
- PostgreSQL 9.1 installed on top of the lucid image,
- PostgreSQL 9.2 installed on top of the lucid image,
- PostgreSQL 9.1 installed on top of the precise image,
- PostgreSQL 9.2 installed on top of the precise image,
- PostgreSQL 9.1 installed on top of the quantal image,
- PostgreSQL 9.2 installed on top of the quantal image,
- PostgreSQL 9.3 installed on top of the quantal image.
The first three won't be in the repository, but the other ones will. You decide that the tags will be lucid9.1, lucid9.2, precise9.1, etc.
Note: those images do not have to share a common ancestor. In this case, we have three "root" images (one for each base Ubuntu release).
When someone wants to use one of your images, he will do something like:
docker run -p 5432 jdoe/pgsql@lucid9.2 postgres -D /var/lib/...
Docker will do the following:
- notice that the image name contains a slash, and is therefore a reference to a repository;
- notice that the image name contains an arroba, and is therefore a reference to a specific version;
- query the docker registry to resolve jdoe/pgsql@lucid9.2 into an image ID;
- download the image metadata+tarball from the registry (unless it already has them locally);
- recursively download all the parent images of the image (unless it already has them locally);
- run the image.
There is one special version: `latest`. When you don't request a specific version, you are implying that you want the `latest` version. When you push a version (any version!) to the repository, you are also pushing to `latest` as well.
QUESTION: do we want to update `latest` even if the commit date of the image is older than the current `latest` image?
QUESTION: who should update `latest`? Should it be done by the docker client, or automatically done server-side?
Confused?
---------
Another way to put it: a "repository" is like the "download binaries" page for a given product of a software vendor. Once version 1.42.5 is there, it probably won't be modified (they will rather release 1.42.6), unless there was something really harmful or embarrassing in 1.42.5.
The S3 storage is authoritative. I.E. the registry will very probably keep some cache of the metadata, but it will be just a cache.
Storage of repositories
-----------------------
TBD
Pull images
-----------
Pulling an image is fairly straightforward:
GET /v1/images/<id>/json
GET /v1/images/<id>/layer
GET /v1/images/<id>/history
The first two calls redirect you to their S3 counterparts. But before redirecting you, the registry checks (probably with `HEAD` requests) that both `json` and `layer` objects actually exist on S3. I.E., if there was a partial upload, when you try to `GET` the `json` or the `layer` object, the registry will give you a 404 for both objects, even if one of them does exist.
The last one sends you a JSON payload, which is a list containing all the metadata of the image and all its ancestors. The requested image comes first, then its parent, then the parent of the parent, etc.
SUGGESTION: rename `history` to `ancestry` (it sounds more hipstery, but it's actually more accurate)
SUGGESTION: add optional parameter `?length=X` to `history`, so you can limit to `X` ancestors, and avoid pulling 42000 ancestors in one go - especially if you have most of them already...
Push images
-----------
The first thing is to push the meta data:
PUT /v1/images/<id>/json
Four things can happen:
- invalid/empty JSON: the server tells you to go away (HTTP 400?)
- image already exists with the same JSON: the server tells you that it's fine (HTTP 204?)
- image already exists but is different: the server informs you that something's wrong (?)
- image doesn't exist: the server puts the JSON on S3, then generates an upload URL for the tarball, and sends you an HTTP 200 containing this upload URL
In the latter case, you want to move to the next step:
PUT the tarball to whatever-URL-you-got-on-previous-stage
SUGGESTION: consider a `PUT /v1/images/<id>/layer` with `Except: 100-continue` and honor a 301/302 redirect. This might or might not be legal HTTP.
The last thing is to try to push the parent image (unless you're sure that it is already in the registry). If the image is already there, stop. If it's not there, upload it, and recursively upload its parents in a similar fashion.
Pull repository
---------------
This:
GET /v1/users/<userid>/<reponame>
Sends back a JSON dict mapping version tags to image version, e.g.:
The request body should be the image version hash.
Example session
---------------
First idea:
# Automatically pull base, aka docker/base@latest, and run something in it
docker run base ...
(Output: 42424242)
docker commit 42424242 databeze
docker login jdoe s3s4me!
# The following two commands are equivalent
docker push jdoe/pgsql databeze
docker push jdoe/pgsql 42424242
Second idea:
docker run base ...
docker commit 42424242 jdoe/pgsql
docker login jdoe s3s4me!
docker push jdoe/pgsql
Maybe this would work too:
docker commit 42424242 pgsql
docker push pgsql
And maybe this too:
docker push -a
NOTE: when your commit overwrites an existing tag, the image should be marked "dirty" so that docker knows that it has to be pushed.
NOTE: if a pull would cause some local tag to be overwritten, docker could refuse, and ask you to rename your local tag, or ask you to specify a -f flag to overwrite. Your local changes won't be lost, but the tag will be lost, so if yon don't know the image ID it could be hard to figure out which one it was.
NOTE: we probably need some commands to move/remove tags to images.
Collaborative workflow:
alice# docker login mybigco p455w0rd
bob# docker login mybigco p455w0rd
alice# docker pull base
alice# docker run -a -t -i base /bin/sh
... hard core authoring takes place ...
alice# docker commit <container_id> wwwbigco
alice# docker push wwwbigco
... the latter actually does docker push mybigco/wwwbigco@latest ...
- It stores the images and the graph for a set of repositories
- It does not have user accounts data
- It has no notion of user accounts or authorization
- It delegates authentication and authorization to the Index Auth service using tokens
- It supports different storage backends (S3, cloud files, local FS)
- It doesn’t have a local database
- It will be open-sourced at some point
We expect that there will be multiple registries out there. To help to grasp the context, here are some examples of registries:
-**sponsor registry**: such a registry is provided by a third-party hosting infrastructure as a convenience for their customers and the docker community as a whole. Its costs are supported by the third party, but the management and operation of the registry are supported by dotCloud. It features read/write access, and delegates authentication and authorization to the Index.
-**mirror registry**: such a registry is provided by a third-party hosting infrastructure but is targeted at their customers only. Some mechanism (unspecified to date) ensures that public images are pulled from a sponsor registry to the mirror registry, to make sure that the customers of the third-party provider can “docker pull” those images locally.
-**vendor registry**: such a registry is provided by a software vendor, who wants to distribute docker images. It would be operated and managed by the vendor. Only users authorized by the vendor would be able to get write access. Some images would be public (accessible for anyone), others private (accessible only for authorized users). Authentication and authorization would be delegated to the Index. The goal of vendor registries is to let someone do “docker pull basho/riak1.3” and automatically push from the vendor registry (instead of a sponsor registry); i.e. get all the convenience of a sponsor registry, while retaining control on the asset distribution.
-**private registry**: such a registry is located behind a firewall, or protected by an additional security layer (HTTP authorization, SSL client-side certificates, IP address authorization...). The registry is operated by a private entity, outside of dotCloud’s control. It can optionally delegate additional authorization to the Index, but it is not mandatory.
..note::
Mirror registries and private registries which do not use the Index don’t even need to run the registry code. They can be implemented by any kind of transport implementing HTTP GET and PUT. Read-only registries can be powered by a simple static HTTP server.
..note::
The latter implies that while HTTP is the protocol of choice for a registry, multiple schemes are possible (and in some cases, trivial):
- HTTP with GET (and PUT for read-write registries);
- local mount point;
- remote docker addressed through SSH.
The latter would only require two new commands in docker, e.g. “registryget” and “registryput”, wrapping access to the local filesystem (and optionally doing consistency checks). Authentication and authorization are then delegated to SSH (e.g. with public keys).
:description:Documentation for docker Registry and Registry API
:keywords:docker, registry, api, index
=====================
Registry & index Spec
=====================
..contents:: Table of Contents
1. The 3 roles
===============
1.1 Index
---------
The Index is responsible for centralizing information about:
- User accounts
- Checksums of the images
- Public namespaces
The Index has different components:
- Web UI
- Meta-data store (comments, stars, list public repositories)
- Authentication service
- Tokenization
The index is authoritative for those information.
We expect that there will be only one instance of the index, run and managed by dotCloud.
1.2 Registry
------------
- It stores the images and the graph for a set of repositories
- It does not have user accounts data
- It has no notion of user accounts or authorization
- It delegates authentication and authorization to the Index Auth service using tokens
- It supports different storage backends (S3, cloud files, local FS)
- It doesn’t have a local database
- It will be open-sourced at some point
We expect that there will be multiple registries out there. To help to grasp the context, here are some examples of registries:
-**sponsor registry**: such a registry is provided by a third-party hosting infrastructure as a convenience for their customers and the docker community as a whole. Its costs are supported by the third party, but the management and operation of the registry are supported by dotCloud. It features read/write access, and delegates authentication and authorization to the Index.
-**mirror registry**: such a registry is provided by a third-party hosting infrastructure but is targeted at their customers only. Some mechanism (unspecified to date) ensures that public images are pulled from a sponsor registry to the mirror registry, to make sure that the customers of the third-party provider can “docker pull” those images locally.
-**vendor registry**: such a registry is provided by a software vendor, who wants to distribute docker images. It would be operated and managed by the vendor. Only users authorized by the vendor would be able to get write access. Some images would be public (accessible for anyone), others private (accessible only for authorized users). Authentication and authorization would be delegated to the Index. The goal of vendor registries is to let someone do “docker pull basho/riak1.3” and automatically push from the vendor registry (instead of a sponsor registry); i.e. get all the convenience of a sponsor registry, while retaining control on the asset distribution.
-**private registry**: such a registry is located behind a firewall, or protected by an additional security layer (HTTP authorization, SSL client-side certificates, IP address authorization...). The registry is operated by a private entity, outside of dotCloud’s control. It can optionally delegate additional authorization to the Index, but it is not mandatory.
..note::
Mirror registries and private registries which do not use the Index don’t even need to run the registry code. They can be implemented by any kind of transport implementing HTTP GET and PUT. Read-only registries can be powered by a simple static HTTP server.
..note::
The latter implies that while HTTP is the protocol of choice for a registry, multiple schemes are possible (and in some cases, trivial):
- HTTP with GET (and PUT for read-write registries);
- local mount point;
- remote docker addressed through SSH.
The latter would only require two new commands in docker, e.g. “registryget” and “registryput”, wrapping access to the local filesystem (and optionally doing consistency checks). Authentication and authorization are then delegated to SSH (e.g. with public keys).
1.3 Docker
----------
On top of being a runtime for LXC, Docker is the Registry client. It supports:
- Push / Pull on the registry
- Client authentication on the Index
2. Workflow
===========
2.1 Pull
--------
..image:: /static_files/docker_pull_chart.png
1. Contact the Index to know where I should download “samalba/busybox”
2. Index replies:
a. “samalba/busybox” is on Registry A
b. here are the checksums for “samalba/busybox” (for all layers)
c. token
3. Contact Registry A to receive the layers for “samalba/busybox” (all of them to the base image). Registry A is authoritative for “samalba/busybox” but keeps a copy of all inherited layers and serve them all from the same location.
4. registry contacts index to verify if token/user is allowed to download images
5. Index returns true/false lettings registry know if it should proceed or error out
6. Get the payload for all layers
It’s possible to run docker pull \https://<registry>/repositories/samalba/busybox. In this case, docker bypasses the Index. However the security is not guaranteed (in case Registry A is corrupted) because there won’t be any checksum checks.
Currently registry redirects to s3 urls for downloads, going forward all downloads need to be streamed through the registry. The Registry will then abstract the calls to S3 by a top-level class which implements sub-classes for S3 and local storage.
Token is only returned when the 'X-Docker-Token' header is sent with request.
Basic Auth is required to pull private repos. Basic auth isn't required for pulling public repos, but if one is provided, it needs to be valid and for an active account.
API (pulling repository foo/bar):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. (Docker -> Index) GET /v1/repositories/foo/bar/images
**Headers**:
Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==
X-Docker-Token: true
**Action**:
(looking up the foo/bar in db and gets images and checksums for that repo (all if no tag is specified, if tag, only checksums for those tags) see part 4.4.1)
5. (Docker -> Registry) GET /v1/images/928374982374/ancestry
**Action**:
(for each image id returned in the registry, fetch /json + /layer)
..note::
If someone makes a second request, then we will always give a new token, never reuse tokens.
2.2 Push
--------
..image:: /static_files/docker_push_chart.png
1. Contact the index to allocate the repository name “samalba/busybox” (authentication required with user credentials)
2. If authentication works and namespace available, “samalba/busybox” is allocated and a temporary token is returned (namespace is marked as initialized in index)
3. Push the image on the registry (along with the token)
4. Registry A contacts the Index to verify the token (token must corresponds to the repository name)
5. Index validates the token. Registry A starts reading the stream pushed by docker and store the repository (with its images)
6. docker contacts the index to give checksums for upload images
..note::
**It’s possible not to use the Index at all!** In this case, a deployed version of the Registry is deployed to store and serve images. Those images are not authentified and the security is not guaranteed.
..note::
**Index can be replaced!** For a private Registry deployed, a custom Index can be used to serve and validate token according to different policies.
Docker computes the checksums and submit them to the Index at the end of the push. When a repository name does not have checksums on the Index, it means that the push is in progress (since checksums are submitted at the end).
API (pushing repos foo/bar):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. (Docker -> Index) PUT /v1/repositories/foo/bar/
**Headers**:
Authorization: Basic sdkjfskdjfhsdkjfh==
X-Docker-Token: true
**Action**::
- in index, we allocated a new repository, and set to initialized
**Body**::
(The body contains the list of images that are going to be pushed, with empty checksums. The checksums will be set at the end of the push)::
If push fails and they need to start again, what happens in the index, there will already be a record for the namespace/name, but it will be initialized. Should we allow it, or mark as name already used? One edge case could be if someone pushes the same thing at the same time with two different shells.
If it's a retry on the Registry, Docker has a cookie (provided by the registry after token validation). So the Index won’t have to provide a new token.
2.3 Delete
----------
If you need to delete something from the index or registry, we need a nice clean way to do that. Here is the workflow.
1. Docker contacts the index to request a delete of a repository “samalba/busybox” (authentication required with user credentials)
2. If authentication works and repository is valid, “samalba/busybox” is marked as deleted and a temporary token is returned
3. Send a delete request to the registry for the repository (along with the token)
4. Registry A contacts the Index to verify the token (token must corresponds to the repository name)
5. Index validates the token. Registry A deletes the repository and everything associated to it.
6. docker contacts the index to let it know it was removed from the registry, the index removes all records from the database.
..note::
The Docker client should present an "Are you sure?" prompt to confirm the deletion before starting the process. Once it starts it can't be undone.
- Authenticate a user as a repos owner (for a central referenced repository)
3.1 Without an Index
--------------------
Using the Registry without the Index can be useful to store the images on a private network without having to rely on an external entity controlled by dotCloud.
In this case, the registry will be launched in a special mode (--standalone? --no-index?). In this mode, the only thing which changes is that Registry will never contact the Index to verify a token. It will be the Registry owner responsibility to authenticate the user who pushes (or even pulls) an image using any mechanism (HTTP auth, IP based, etc...).
In this scenario, the Registry is responsible for the security in case of data corruption since the checksums are not delivered by a trusted entity.
As hinted previously, a standalone registry can also be implemented by any HTTP server handling GET/PUT requests (or even only GET requests if no write access is necessary).
3.2 With an Index
-----------------
The Index data needed by the Registry are simple:
- Serve the checksums
- Provide and authorize a Token
In the scenario of a Registry running on a private network with the need of centralizing and authorizing, it’s easy to use a custom Index.
The only challenge will be to tell Docker to contact (and trust) this custom Index. Docker will be configurable at some point to use a specific Index, it’ll be the private entity responsibility (basically the organization who uses Docker in a private environment) to maintain the Index and the Docker’s configuration among its consumers.
4. The API
==========
The first version of the api is available here: https://github.com/jpetazzo/docker/blob/acd51ecea8f5d3c02b00a08176171c59442df8b3/docs/images-repositories-push-pull.md
4.1 Images
----------
The format returned in the images is not defined here (for layer and json), basically because Registry stores exactly the same kind of information as Docker uses to manage them.
The format of ancestry is a line-separated list of image ids, in age order. I.e. the image’s parent is on the last line, the parent of the parent on the next-to-last line, etc.; if the image has no parent, the file is empty.
- **username** : min 4 character, max 30 characters, must match the regular expression [a-z0-9_].
- **password**: min 5 characters
**Valid**: return HTTP 200
Errors: HTTP 400 (we should create error codes for possible errors)
- invalid json
- missing field
- wrong format (username, password, email, etc)
- forbidden name
- name already exists
..note::
A user account will be valid only if the email has been validated (a validation link is sent to the email address).
4.2.2 Update a user (Index)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
PUT /v1/users/<username>
**Body**:
{"password": "toto"}
..note::
We can also update email address, if they do, they will need to reverify their new email address.
4.2.3 Login (Index)
^^^^^^^^^^^^^^^^^^^
Does nothing else but asking for a user authentication. Can be used to validate credentials. HTTP Basic Auth for now, maybe change in future.
GET /v1/users
**Return**:
- Valid: HTTP 200
- Invalid login: HTTP 401
- Account inactive: HTTP 403 Account is not Active
4.3 Tags (Registry)
-------------------
The Registry does not know anything about users. Even though repositories are under usernames, it’s just a namespace for the registry. Allowing us to implement organizations or different namespaces per user later, without modifying the Registry’s API.
The following naming restrictions apply:
- Namespaces must match the same regular expression as usernames (See 4.2.1.)
- Repository names must match the regular expression [a-zA-Z0-9-_.]
4.3.1 Get all tags
^^^^^^^^^^^^^^^^^^
GET /v1/repositories/<namespace>/<repository_name>/tags
For the Index to “resolve” the repository name to a Registry location, it uses the X-Docker-Endpoints header. In other terms, this requests always add a “X-Docker-Endpoints” to indicate the location of the registry which hosts this repository.
4.4.1 Get the images
^^^^^^^^^^^^^^^^^^^^^
GET /v1/repositories/<namespace>/<repo_name>/images
Usually, the Registry provides a Cookie when a Token verification succeeded. Every time the Registry passes a Cookie, you have to pass it back the same cookie.::
:description:Build a new image from the Dockerfile passed via stdin
:keywords:build, docker, container, documentation
================================================
``build`` -- Build a container from a Dockerfile
================================================
::
Usage: docker build [OPTIONS] PATH | URL | -
Build a new container image from the source code at PATH
-t="": Tag to be applied to the resulting image in case of success.
-q=false: Suppress verbose build output.
When a single Dockerfile is given as URL, then no context is set. When a git repository is set as URL, the repository is used as context
Examples
--------
..code-block::bash
docker build .
| This will read the Dockerfile from the current directory. It will also send any other files and directories found in the current directory to the docker daemon.
| The contents of this directory would be used by ADD commands found within the Dockerfile.
| This will send a lot of data to the docker daemon if the current directory contains a lot of data.
| If the absolute path is provided instead of '.', only the files and directories required by the ADD commands from the Dockerfile will be added to the context and transferred to the docker daemon.
|
..code-block::bash
docker build - < Dockerfile
| This will read a Dockerfile from Stdin without context. Due to the lack of a context, no contents of any local directory will be sent to the docker daemon.
| ADD doesn't work when running in this mode due to the absence of the context, thus having no source files to copy to the container.
..code-block::bash
docker build github.com/creack/docker-firefox
| This will clone the github repository and use it as context. The Dockerfile at the root of the repository is used as Dockerfile.
| Note that you can specify an arbitrary git repository by using the 'git://' schema.
Want to hack on Docker? Awesome! The repository includes `all the instructions you need to get started <https://github.com/dotcloud/docker/blob/master/CONTRIBUTING.md>`_.
:description:Sharing data between 2 couchdb databases
:keywords:docker, example, package installation, networking, couchdb, data volumes
.._running_couchdb_service:
CouchDB Service
===============
..include:: example_header.inc
Here's an example of using data volumes to share the same data between 2 couchdb containers.
This could be used for hot upgrades, testing different versions of couchdb on the same data, etc.
Create first database
---------------------
Note that we're marking /var/lib/couchdb as a data volume.
..code-block::bash
COUCH1=$(docker run -d -v /var/lib/couchdb shykes/couchdb:2013-05-03)
Add data to the first database
------------------------------
We're assuming your docker host is reachable at `localhost`. If not, replace `localhost` with the public IP of your docker host.
..code-block::bash
HOST=localhost
URL="http://$HOST:$(docker port $COUCH1 5984)/_utils/"
echo"Navigate to $URL in your browser, and use the couch interface to add data"
Create second database
----------------------
This time, we're requesting shared access to $COUCH1's volumes.
..code-block::bash
COUCH2=$(docker run -d -volumes-from $COUCH1) shykes/couchdb:2013-05-03)
Browse data on the second database
----------------------------------
..code-block::bash
HOST=localhost
URL="http://$HOST:$(docker port $COUCH2 5984)/_utils/"
echo"Navigate to $URL in your browser. You should see the same data as in the first database!"
Congratulations, you are running 2 Couchdb containers, completely isolated from each other *except* for their data.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Victor Vieux
