Merge pull request #400 from krisdale/kd-autospec-apply-guide-template

Apply guide template to autospec guide + update examples
This commit is contained in:
michael vincerra
2019-03-04 13:48:45 -08:00
committed by GitHub
+293 -146
View File
@@ -1,182 +1,167 @@
.. _autospec:
Build RPMs with autospec
########################
autospec
########
This guide shows you how to create RPMs with :ref:`autospec <autospec-about>`,
a tool that assists in automated creation and maintenance of RPM packaging
on |CL-ATTR|.
**autospec** is a tool to assist in the automated creation and maintenance of
RPM packaging in |CL-ATTR|. Where a standard RPM build process using
:command:`rpmbuild` requires a tarball and :file:`.spec` file to start, autospec
requires only a tarball and package name to start.
See our :ref:`autospec concept page <autospec-about>` for a detailed explaination
of how ``autospec`` works on |CL|. For a general understanding of how RPMs work,
we recommend visiting the `rpm website`_ or the `RPM Packaging Guide`_ .
.. contents::
:local:
:depth: 1
Description
***********
The autospec tool attempts to infer the requirements of the :file:`.spec` file
by analyzing the source code and :file:`Makefile` information. It will
continuously run updated builds based on new information discovered from build
failures until it has a complete and valid :file:`.spec` file. If needed, you
can influence the behavior of autospec and customize the build by providing
optional `control files`_ to the autospec tool.
autospec uses mock as a sandbox to run the builds. Visit the `mock wiki`_ for
additional information on using mock.
For a general understanding of how RPMs work, visit the `rpm website`_ or the
`RPM Packaging Guide`_ .
How it works
************
Learn the autospec tool set up and process.
.. contents::
:local:
:depth: 1
Prerequisites
*************
=============
This guide requires that you:
The setup for building source in |CL| must be completed before using the
autospec tool.
* Have installed |CL| on a host machine or virtual environment. For detailed
instructions on installing |CL|, visit the :ref:`get-started` section.
Refer to `Setup environment to build source`_ for instructions on completing
setup.
* :ref:`install-tooling`
Create an RPM
=============
.. _install-tooling:
The basic autospec process is described in the following steps:
Install the |CL| development tooling framework
==============================================
#. The :command:`make autospec` command generates a :file:`.spec` file based on
analysis of code and existing control files.
.. _install-tooling-after-header:
Any control files should be located in the same directory as the resulting
:file:`.spec` file.
#. Install the `os-clr-on-clr` developer bundle on your host system.
View the `autospec README`_ for more information on `control files`_.
.. code-block:: bash
#. autospec creates a build root with mock config.
sudo swupd bundle-add os-clr-on-clr
#. autospec attempts to build an RPM from the generated :file:`.spec`.
#. Download the :file:`user-setup.sh` script.
#. autospec detects any missed declarations in the :file:`.spec`.
.. code-block:: bash
#. If build errors occur, autospec will scan the build log to try and detect
the root cause.
curl -O https://raw.githubusercontent.com/clearlinux/common/master/user-setup.sh
#. If autospec detects the root cause and knows how to continue, it will restart
the build automatically at step 1 with updated build instructions.
#. Make :file:`user-setup.sh` executable.
#. Otherwise, autospec will stop the build for user inspection to resolve the
errors. Respond to the build process output by fixing source code issues
and/or editing control files to resolve issues, which may include
dependencies or exclusions. See `autospec README`_ for more information on
control files.
.. code-block:: bash
The user resumes the process at step 1 after errors are resolved.
chmod +x user-setup.sh
If a binary dependency doesn't exist in |CL|, you will need to build it
before running autospec again.
#. Run the script as an unprivileged user.
Following these steps, autospec continues to rebuild the package, based on
new information discovered from build failures, until it has a valid
:file:`.spec`. If no build errors occur, RPM packages are successfully built.
.. code-block:: bash
Examples
********
./user-setup.sh
Complete `Setup environment to build source`_ before using these examples.
#. After the script completes, log out and log in again to complete
the setup process.
.. contents::
:local:
:depth: 1
The `user-setup script`_ creates a folder called :file:`clearlinux`, which
contains the :file:`Makefile`, :file:`packages`, and :file:`projects`
subfolders.
Example 1: Build RPM with existing spec file
============================================
The :file:`projects` folder contains the main tools, `autospec`
and `common`, used for making packages in |CL|.
This example shows how to build a RPM from a pre-packaged upstream package, with
an existing spec file. The example uses the ``dmidecode`` package.
.. _install-tooling-end:
Create a RPM with autospec
**************************
Choose one of the following options to build RPMs and manage source
code:
* :ref:`build-a-new-rpm` and spec file using ``make autospecnew``.
* :ref:`build-source-code-with-existing-spec-file` using ``make build``, without changing the
spec file.
* :ref:`generate-a-new-spec-file` using ``make autospec``, based on changes in the control files.
.. _build-a-new-rpm:
Option 1: Build a new RPM
=========================
Use this method to build a new RPM with no spec file. In this example,
we build a new helloclear RPM.
#. Navigate to the autospec workspace.
#. Navigate to the autospec workspace and clone the ``dmidecode`` package:
.. code-block:: bash
cd ~/clearlinux
#. Enter the command:
.. code-block:: bash
make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz"
NAME="helloclear"
make clone_dmidecode
.. note::
For a local tarball, use this type of *URL*: \file://<absolute-path-to-tarball>
#. If build failures or dependency issues occur, continue below.
Otherwise, skip directly to `Next steps`_.
#. Navigate to the specific package.
You can clone all package repos at once using:
.. code-block:: bash
cd ~/clearlinux/packages/[package-name]
make [-j NUM] clone-packages
#. Respond to the build process output by editing control files to resolve
issues, which may include dependencies or exclusions.
See `autospec readme`_
The optional NUM is the number of threads to use.
#. Run this command:
For a list of available packages, view the
:file:`~/clearlinux/projects/common/packages` file.
.. code-block:: bash
make autospec
Repeat the last two steps above until all errors are resolved and you
complete a successful build.
**Congratulations!**
You've successfully created a RPM.
Skip to `Next steps`_.
.. _build-source-code-with-existing-spec-file:
Option 2: Build source code with an existing spec file
======================================================
Use this method if you only want to build the RPM using the spec file. This
method assumes that a spec file already exists. In this example, we run a
``make build`` on the ``dmidecode`` package.
#. Navigate to the ``dmidecode`` package in clearlinux:
#. Navigate to the local copy of the ``dmidecode`` package and build it:
.. code-block:: bash
cd ~/clearlinux/packages/dmidecode/
#. To download the tarball and build, run the command:
.. code-block:: bash
make build
**Congratulations!**
#. The resulting RPMs are in :file:`./rpms`. Build logs and additional RPMs are
in :file:`./results`.
You've successfully created a RPM.
Example 2: Build a new RPM
==========================
Skip to `Next steps`_.
This example shows how to build a new RPM with no spec file. The example will
create a simple helloclear RPM.
.. _generate-a-new-spec-file:
Option 3: Generate a new spec file with a pre-defined package
=============================================================
Use this method to modify an existing package. In this example, you will
modify an existing |CL| package called ``dmidecode`` to create a custom
RPM. You will make a simple change to this package, change the revision to
a new number that is higher than the |CL| OS version, and rebuild the package.
#. Navigate to clearlinux:
#. Navigate to the autospec workspace and build the helloclear RPM. The
:file:`Makefile` provides a :command:`make autospecnew` that can
automatically generate an RPM package using the autospec tool. You must pass
the URL to the source tarball and the NAME of the RPM you wish to create:
.. code-block:: bash
cd ~/clearlinux
make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear"
#. Copy the ``dmidecode`` package.
The resulting RPMs are in :file:`./packages/helloclear/rpms`. Builde logs and
additional RPMs are in :file:`./packages/helloclear/results`.
Example 3: Generate a new spec file with a pre-defined package
==============================================================
This example shows how to modify an existing package to create a custom RPM. In
this example you will make a simple change to the ``dmidecode`` package and
rebuild the package.
#. Navigate to the autospec workspace and clone the ``dmidecode`` package:
.. code-block:: bash
cd ~/clearlinux
make clone_dmidecode
#. Navigate into the *dmidecode* directory:
@@ -185,9 +170,9 @@ a new number that is higher than the |CL| OS version, and rebuild the package.
cd packages/dmidecode
#. With an editor, open the :file:`excludes` file and add these lines:
#. Open the :file:`excludes` file with an editor and add these lines:
.. code-block:: bash
.. code-block:: console
/usr/bin/biosdecode
/usr/bin/ownership
@@ -201,49 +186,211 @@ a new number that is higher than the |CL| OS version, and rebuild the package.
These files aren't needed by dmidecode, so we can remove them without
any issues.
#. Save the file and exit.
#. At :file:`~/clearlinux/packages/dmidecode`, build the modified
``dmidecode`` package:
#. In the :file:`dmidecode` directory, build the modified ``dmidecode`` package:
.. code-block:: bash
make autospec
When the process completes, you will see new RPM packages in the
:file:`results/` folder.
#. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`.
#. To view the new RPM packages, enter:
Example 4: Provide control files to autospec
============================================
This example shows how to modify control files to correct build failures that
autospec is unable to resolve. In this example you will add a missing license
and dependencies in order for autospec to complete a successful build.
#. Navigate to the autospec workspace:
.. code-block:: bash
ls /clearlinux/packages/dmidecode/results/
cd ~/clearlinux
**Congratulations!**
#. If you have not already, clone all upstream package repos:
You've successfully created a RPM.
.. code-block:: bash
Next steps
make [-j NUM] clone-packages
The optional NUM is the number of threads to use.
.. note::
In a later step of this example, we will search the cloned package repos
for a missing dependency.
#. Build the opae-sdk RPM:
.. code-block:: bash
make autospecnew URL="https://github.com/OPAE/opae-sdk/archive/0.13.0.tar.gz" NAME="opae-sdk"
This will give an error for a missing license file:
.. code-block:: console
[FATAL] Cannot find any license or opae-sdk.license file!
#. Navigate to the package with build failures:
.. code-block:: bash
cd packages/opae-sdk
#. Add a license:
.. code-block:: bash
echo "Intel Corporation" > opae-sdk.license
#. Run autospec again:
.. code-block:: bash
make autospec
This will result in a generic error:
.. code-block:: console
[FATAL] Build failed, aborting
#. Open the build log to view the error details:
.. code-block:: bash
cat ./results/build.log
In the build log, you will find details for the specific failures. In this
instance, there are missing dependencies:
.. code-block:: console
CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files:
CJSON_LIBRARY
linked by target "opae-c++-utils" in directory /builddir/build/BUILD/opae-sdk-0.13.0/tools/c++utilslib
json-c_LIBRARIES
linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae
libuuid_LIBRARIES
linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae
#. Search the spec files of upstream |CL| packages to see if the json-c library
is availabe. In this case, it does exist and we'll add the json-c 'dev'
package into the buildreq_add:
.. code-block:: bash
grep 'json-c\.so$' ~/clearlinux/packages/*/*.spec
echo "json-c-dev" >> buildreq_add
.. note::
This search step works only if the user cloned all of the upstream package
repos. In this example, upstream package repos were cloned in a previous
step.
#. Search the spec files of upstream |CL| packages to see if the libuuid library
is available. In this case, it exists in the util-linux package, so we'll add
util-linux-dev package into the buildreq_add:
.. code-block:: bash
grep 'libuuid\.so$' ~/clearlinux/packages/*/*.spec
echo "util-linux-dev" >> buildreq_add
#. Run autospec again and find the successfully-generated RPMs in the rpms
directory:
.. code-block:: bash
make autospec
.. note::
If you need a dependency that does not exist in the |CL| repo, you must first
build it manually (see `Example 2: Build a new RPM`_), then add the repo so
that autospec knows the package exists. For example:
.. code-block:: bash
cd ~/clearlinux/packages/<package-name>
make repoadd
make repostatus
You only need to add the dependency to the :file:`buildreq_add` control file
if autospec is not able to automatically find the correct dependency on its
own.
References
**********
Now you can create a custom bundle with your new RPM and use it with |CL|:
Reference the `autospec README`_ for details regarding autospec commands and
options.
* Use the :ref:`Mixer tool <mixer>` to add a new bundle to your derivative of |CL|.
* Use the :ref:`Mixin tool <mixin>` to customize your upstream |CL| installation with a new bundle.
Setup environment to build source
=================================
.. _install-tooling-after-header:
Setup of the workspace and tooling used for building source in |CL| is mostly
automated for you with a setup script. It uses tools from the
:command:`os-clr-on-clr` bundle.
The setup script creates a workspace in the :file:`clearlinux` folder, with the
subfolders :file:`Makefile`, :file:`packages`, and :file:`projects`. The
:file:`projects` folder contains the main tools used for making packages in
|CL|: `autospec` and `common`.
Follow these steps to setup the workspace and tooling for building source:
#. Install the :command:`os-clr-on-clr` bundle:
.. code-block:: bash
sudo swupd bundle-add os-clr-on-clr
#. Download the :file:`user-setup.sh` script:
.. code-block:: bash
curl -O https://raw.githubusercontent.com/clearlinux/common/master/user-setup.sh
#. Make :file:`user-setup.sh` executable:
.. code-block:: bash
chmod +x user-setup.sh
#. Run the script as an unprivileged user:
.. code-block:: bash
./user-setup.sh
#. After the script completes, log out and log in again to complete the setup
process.
#. Set your Git user email and username for the repos on your system:
.. code-block:: bash
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
This global setting is used by |CL| tools that make use of Git.
.. _install-tooling-end:
Related topics
**************
* :ref:`Mixer tool <mixer>`
* :ref:`Mixin tool <mixin>`
* :ref:`autospec <autospec-about>`
* :ref:`Bundles <bundles-about>`
.. _rpm website: http://rpm.org
.. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/
.. _user-setup script: https://github.com/clearlinux/common/blob/master/user-setup.sh
.. _autospec readme: https://github.com/clearlinux/autospec
.. _autospec README: https://github.com/clearlinux/autospec
.. _control files: https://github.com/clearlinux/autospec#control-files
.. _mock wiki: https://github.com/rpm-software-management/mock/wiki
.. _rpm website: http://rpm.org
.. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/