diff --git a/source/_static/images/bulk-provision.png b/source/_static/images/bulk-provision.png new file mode 100644 index 00000000..23042be9 Binary files /dev/null and b/source/_static/images/bulk-provision.png differ diff --git a/source/_static/images/bulk-provision.vsdx b/source/_static/images/bulk-provision.vsdx new file mode 100644 index 00000000..eca3e871 Binary files /dev/null and b/source/_static/images/bulk-provision.vsdx differ diff --git a/source/_static/images/icis_installer_workflow.png b/source/_static/images/icis_installer_workflow.png deleted file mode 100644 index e0015399..00000000 Binary files a/source/_static/images/icis_installer_workflow.png and /dev/null differ diff --git a/source/_static/images/icis_pxe_diagram.png b/source/_static/images/icis_pxe_diagram.png deleted file mode 100644 index cb788ecc..00000000 Binary files a/source/_static/images/icis_pxe_diagram.png and /dev/null differ diff --git a/source/bulk_provisioning.rst b/source/bulk_provisioning.rst index dccdfbc0..81bf01f8 100644 --- a/source/bulk_provisioning.rst +++ b/source/bulk_provisioning.rst @@ -1,146 +1,164 @@ .. _bulk_provisioning: Bulk Provisioning -================= +***************** -Data centers tend to land and install new machines in bulk. When dealing with -large quantities of new machines or hardware upgrades, data center admins may need -a way to install and configure with a minimal amount of effort. To support this -use case, the Clear Linux team has developed some tooling and instructions for -how to bulk-load machines with Clear Linux using iPXE. +Data centers have a need to install and configure new instances of operating +systems in bulk. When managing new computers or hardware upgrades in bulk, data +center administrators need tooling for making changes with minimal effort. -This bulk provisioning scenario is a variation on the :ref:`network_boot` scenario, -with some alterations to facilitate installing and configuring many hosts. The Clear -Linux installer (``Ister``) was augmented to detect and find ``cloud-init`` configuration -data during the install. When this data is found, a :file:`cloud-init` file is fetched, -and the host, after being rebooted by ``ister``, uses micro-config drive (``ucf``) to -configure the host as directed by the :file:`cloud-init` configuration file. +The bulk provisioning scenario is a continuation of the :ref:`network_boot` +scenario. After a network boot occurs, ``ister`` provides automation for this +task using `cloud-init`_ files by performing the following steps: -In our pilot of this workflow, we used ``clear-cloud-init`` (which will become ucf - micro config drive) -to install the trust relationships for the host to be further managed by Ansible. The -:file:`cloud-init` configuration file was served by a simple little web application -that ``Ister`` knows how to query for host configuration data. This allowed us to -simultaneously power-on one hundred machines, each of which -* picked up the Clear Linux installer via PXE, -* installed Clear Linux, -* rebooted, and -* entered into a mode that can be managed through Ansible. +#. Finding cloud-init files +#. Creating a `micro-config-drive`_ (ucf) to store the cloud-init files +#. Creating a systemd service which runs at first reboot to apply the + configurations defined by the cloud-init files +#. Reboots the machine to run the systemd service -The web application that serves cloud-init configurations to ``Ister`` is uninspiringly named -"Ister Cloud Init Service" (ICIS), and is available at github - https://github.com/clearlinux/ister-cloud-init-svc +At this point, the machine has been rebooted twice, once to perform a network +boot and once again to apply cloud-init configurations. Afte the second reboot, +the machine can enter into a mode ready to be managed with Ansible. -One nice attribute of this system is that once the iPXE bits are created, many installer -behaviors can be configured without having to regenerate the iPXE bits. +The ister cloud init service (`icis`_) is used by ister to host the cloud-init +configurations. Ister can be directed to look for cloud-init configurations +hosted by icis. + +One nice attribute of this system is that once the iPXE bits are created, many +installer behaviors can be configured without having to regenerate the iPXE +bits. System Architecture -~~~~~~~~~~~~~~~~~~~ +=================== -The following diagrams illustrate the system architecture. +The following diagram illustrates the flow of information between the pxe +server and a host booting into the installer via pxe. This diagram is intended +to show the logical flow of information, and is not a literal depiction of the +protocol-level exchanges. -.. image:: _static/images/icis_pxe_diagram.png - :align: center - :alt: pxe-server-overview - -As can be seen, ICIS is a web application (specifically a Flask application, hence managed by ``uwsgi`` -under nginx) that can run **anywhere** visible to hosts booting from PXE. In our example, we simply -hosted it on the same system serving the pxe content. - -The following diagram illustrates the flow of information between the pxe server and a host booting -into the installer via pxe. This diagram is intended to show the logical flow of information, and -is not a literal depiction of the protocol-level exchanges. - -.. image:: _static/images/icis_installer_workflow.png +.. image:: _static/images/bulk-provision.png :align: center :alt: pxe-icis-install-workflow -Configuration Overview -~~~~~~~~~~~~~~~~~~~~~~ +Application server, web server, dhcp server, tftp server are hosted on a +single machine, but can be hosted anywhere visible to PXE clients. -Here is a high-level overview describing set up and configuration of a bulk-provisioning solution. +Configuration +============= -* First, stand up an iPXE server as described in the :ref:`network_boot` docs. Note that you will - be generating your own artifacts to be served by PXE in a later step. +Step 1 +------ -* Land `Ister Cloud Init Service `_ on the - pxe server. The README in the ICIS github repo has directions on how to install and configure. +First, stand up an iPXE server as described in the :ref:`network_boot` docs. +Note that you will be generating your own artifacts to be served by PXE in a +later step. -* Generate the installer that will load and boot over PXE. Copy the relevant files into the - appropriate location on the iPXE server. +Step 2 +------ - A pxe installer is generated for every release of Clear Linux OS. It can be found alongside the - published images in https://download.clearlinux.org/releases/XXXXX/clear/clear-XXXXX-pxe.tar.xz +Land `Ister Cloud Init Service `_ on the pxe server. The README in the ICIS github repo has directions on +how to install and configure. - Alternatively, use the `create_pxe.sh `_ - script to roll your own. Note this creates the pxe-installer from the "provisioning" installer in - Clear Linux. This is a touchless installer that takes all installation information from a config - file. You can get this image from a given Clear Linux release by looking - in https://download.clearlinux.org/releases/XXXX/clear/ where XXXX is a Clear Linux release number. - Just download the image alongside the ``create_pxe.sh`` script and name it ``provision.img``. +Step 3 +------ -* Stage config files for ``ister`` that will govern Ister's behavior. This includes modifying the - script for ipxe boot, getting it to pass an additional parameter to the kernel. +Generate the installer that will load and boot over PXE. Copy the relevant files into the +appropriate location on the iPXE server. - The magic that ties all of this together is that the pxe script conveys to ister the location of its configuration files via the kernel command line of the installer it kicks off. The kernel preserves its command line precisely, and ister inspects it via ``/proc/cmdline``. +A pxe installer is generated for every release of Clear Linux OS. It can be found alongside the +published images in https://download.clearlinux.org/releases/XXXXX/clear/clear-XXXXX-pxe.tar.xz - Here is an example pxe script: +Alternatively, use the `create_pxe.sh `_ +script to roll your own. Note this creates the pxe-installer from the "provisioning" installer in +Clear Linux. This is a touchless installer that takes all installation information from a config +file. You can get this image from a given Clear Linux release by looking +in https://download.clearlinux.org/releases/XXXX/clear/ where XXXX is a Clear Linux release number. +Just download the image alongside the ``create_pxe.sh`` script and name it ``provision.img``. - .. code-block:: console +Step 4 +------ - #!ipxe - kernel linux quiet rdinit=/usr/lib/systemd/systemd-bootchart initcall_debug tsc=reliable - no_timer_check noreplace-smp rw initrd=initrd isterconf=http://192.168.1.1/icis/static/ister/ister.conf - initrd initrd - boot +Stage config files for ``ister`` that will govern Ister's behavior. This includes modifying the +script for ipxe boot, getting it to pass an additional parameter to the kernel. - When the pxe installer kicks off ``ister``, it will make note of the location of the ``conf`` file - that was given on the kernel command line, and fetch the file. This file then tells ``ister`` where - to get the json template file that describes partition schemes, and which version of Clear Linux to - install. This means that so long as the contents of a release are compatible with the version of - software update (``swupd``) in the installer, this pxe installer can be told to install a newer - version of Clear Linux simply by tweaking the json on the web server, rather than rolling an - entirely new installer. +The magic that ties all of this together is that the pxe script conveys to ister the location of its configuration files via the kernel command line of the installer it kicks off. The kernel preserves its command line precisely, and ister inspects it via ``/proc/cmdline``. - One other important piece of configuration data in the json configuration file is the location of - an ICIS configuration service. Ister will query ICIS for a role using the MAC address of the network - interface being used to communicate with the ICIS service. Ister will then fetch that specific - :file:`cloud-init` file and ``configure ucd`` to run on first-boot against that config file. +Here is an example pxe script: - The `Ister Cloud Init Service `_ github repo - has example ister configuration files under ``static/ister``. +.. code-block:: console - Here is an example ister-template.json file. + #!ipxe + kernel linux quiet rdinit=/usr/lib/systemd/systemd-bootchart initcall_debug tsc=reliable + no_timer_check noreplace-smp rw initrd=initrd isterconf=http://192.168.1.1/icis/static/ister/ister.conf + initrd initrd + boot - .. code-block:: console +When the pxe installer kicks off ``ister``, it will make note of the location of the ``conf`` file +that was given on the kernel command line, and fetch the file. This file then tells ``ister`` where +to get the json template file that describes partition schemes, and which version of Clear Linux to +install. This means that so long as the contents of a release are compatible with the version of +software update (``swupd``) in the installer, this pxe installer can be told to install a newer +version of Clear Linux simply by tweaking the json on the web server, rather than rolling an +entirely new installer. - { - "DestinationType" : "phyiscal", - "PartitionLayout" : [ { "disk" : "/dev/sda", "partition" : 1, - "size" : "64M", "type" : "EFI" }, - { "disk" : "/dev/sda", "partition" : 2, - "size" : "2G", "type" : "linux" } ], - "FilesystemTypes" : [ { "disk" : "/dev/sda", "partition" : 1, "type" : "vfat" }, - { "disk" : "/dev/sda", "partition" : 2, "type" : "ext4" } ], - "PartitionMountPoints" : [ { "disk" : "/dev/sda", "partition" : 1, - "mount" : "/boot" }, - { "disk" : "/dev/sda", "partition" : 2, - "mount" : "/" } ], - "Version": 6580, - "Bundles": ["kernel-native", "os-core-update", "os-core", - "bootloader", "sysadmin-hostmgmt", "openssh-server"], - "PostNonChroot": ["./installation-image-post-update-version.py"], - "IsterCloudInitSvc": ["http://192.168.1.1/icis/"] - } +One other important piece of configuration data in the json configuration file is the location of +an ICIS configuration service. Ister will query ICIS for a role using the MAC address of the network +interface being used to communicate with the ICIS service. Ister will then fetch that specific +:file:`cloud-init` file and ``configure ucd`` to run on first-boot against that config file. -* Configure ICIS to map MAC addresses to role files appropriately. Then create the role files, which - are ``cloud-init`` configuration files. Note, it is possible to simply specify a "default" role for - any unmatched MAC address; this may be handy when all install targets are to be configured identically. +The `Ister Cloud Init Service `_ github repo +has example ister configuration files under ``static/ister``. -* Final pre-flight check. Assuming your iPXE server is at 192.168.1.1, all of the following urls need to be working. +Here is an example ister-template.json file. - - http://192.168.1.1/icis/static/ister/ister.conf - - http://192.168.1.1/icis/static/ister/ister_config.json - - http://192.168.1.1/icis/get_config/ - - http://192.168.1.1/icis/get_role/ - - http://192.168.1.1/ipxe_boot_script.txt +.. code-block:: console -* Boot an iPXE client and watch Clear Linux install. + { + "DestinationType" : "phyiscal", + "PartitionLayout" : [ { "disk" : "/dev/sda", "partition" : 1, + "size" : "64M", "type" : "EFI" }, + { "disk" : "/dev/sda", "partition" : 2, + "size" : "2G", "type" : "linux" } ], + "FilesystemTypes" : [ { "disk" : "/dev/sda", "partition" : 1, "type" : "vfat" }, + { "disk" : "/dev/sda", "partition" : 2, "type" : "ext4" } ], + "PartitionMountPoints" : [ { "disk" : "/dev/sda", "partition" : 1, + "mount" : "/boot" }, + { "disk" : "/dev/sda", "partition" : 2, + "mount" : "/" } ], + "Version": 6580, + "Bundles": ["kernel-native", "os-core-update", "os-core", + "bootloader", "sysadmin-hostmgmt", "openssh-server"], + "PostNonChroot": ["./installation-image-post-update-version.py"], + "IsterCloudInitSvc": ["http://192.168.1.1/icis/"] + } + +Step 5 +------ + +Configure ICIS to map MAC addresses to role files appropriately. Then create the role files, which +are ``cloud-init`` configuration files. Note, it is possible to simply specify a "default" role for +any unmatched MAC address; this may be handy when all install targets are to be configured identically. + +Step 6 +------ + +Final pre-flight check. Assuming your iPXE server is at 192.168.1.1, all of the +following urls need to be working: + +* http://192.168.1.1/icis/static/ister/ister.conf +* http://192.168.1.1/icis/static/ister/ister_config.json +* http://192.168.1.1/icis/get_config/ +* http://192.168.1.1/icis/get_role/ +* http://192.168.1.1/ipxe_boot_script.txt + +Step 7 +------ + +Boot an iPXE client and watch Clear Linux install. + + +.. _icis: https://github.com/clearlinux/ister-cloud-init-svc +.. _cloud-init: https://cloudinit.readthedocs.io +.. _micro-config-drive: https://github.com/clearlinux/micro-config-drive \ No newline at end of file