diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index b5b083a6..0b955d3d 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -1,182 +1,167 @@ .. _autospec: -Build RPMs with autospec -######################## +autospec +######## -This guide shows you how to create RPMs with :ref:`autospec `, -a tool that assists in automated creation and maintenance of RPM packaging -on |CL-ATTR|. +**autospec** is a tool to assist in the automated creation and maintenance of +RPM packaging in |CL-ATTR|. Where a standard RPM build process using +:command:`rpmbuild` requires a tarball and :file:`.spec` file to start, autospec +requires only a tarball and package name to start. -See our :ref:`autospec concept page ` for a detailed explaination -of how ``autospec`` works on |CL|. For a general understanding of how RPMs work, -we recommend visiting the `rpm website`_ or the `RPM Packaging Guide`_ . +.. contents:: + :local: + :depth: 1 + +Description +*********** + +The autospec tool attempts to infer the requirements of the :file:`.spec` file +by analyzing the source code and :file:`Makefile` information. It will +continuously run updated builds based on new information discovered from build +failures until it has a complete and valid :file:`.spec` file. If needed, you +can influence the behavior of autospec and customize the build by providing +optional `control files`_ to the autospec tool. + +autospec uses mock as a sandbox to run the builds. Visit the `mock wiki`_ for +additional information on using mock. + +For a general understanding of how RPMs work, visit the `rpm website`_ or the +`RPM Packaging Guide`_ . + +How it works +************ + +Learn the autospec tool set up and process. + +.. contents:: + :local: + :depth: 1 Prerequisites -************* +============= -This guide requires that you: +The setup for building source in |CL| must be completed before using the +autospec tool. -* Have installed |CL| on a host machine or virtual environment. For detailed - instructions on installing |CL|, visit the :ref:`get-started` section. +Refer to `Setup environment to build source`_ for instructions on completing +setup. -* :ref:`install-tooling` +Create an RPM +============= -.. _install-tooling: +The basic autospec process is described in the following steps: -Install the |CL| development tooling framework -============================================== +#. The :command:`make autospec` command generates a :file:`.spec` file based on + analysis of code and existing control files. -.. _install-tooling-after-header: + Any control files should be located in the same directory as the resulting + :file:`.spec` file. -#. Install the `os-clr-on-clr` developer bundle on your host system. + View the `autospec README`_ for more information on `control files`_. - .. code-block:: bash +#. autospec creates a build root with mock config. - sudo swupd bundle-add os-clr-on-clr +#. autospec attempts to build an RPM from the generated :file:`.spec`. -#. Download the :file:`user-setup.sh` script. +#. autospec detects any missed declarations in the :file:`.spec`. - .. code-block:: bash +#. If build errors occur, autospec will scan the build log to try and detect + the root cause. - curl -O https://raw.githubusercontent.com/clearlinux/common/master/user-setup.sh +#. If autospec detects the root cause and knows how to continue, it will restart + the build automatically at step 1 with updated build instructions. -#. Make :file:`user-setup.sh` executable. +#. Otherwise, autospec will stop the build for user inspection to resolve the + errors. Respond to the build process output by fixing source code issues + and/or editing control files to resolve issues, which may include + dependencies or exclusions. See `autospec README`_ for more information on + control files. - .. code-block:: bash + The user resumes the process at step 1 after errors are resolved. - chmod +x user-setup.sh + If a binary dependency doesn't exist in |CL|, you will need to build it + before running autospec again. -#. Run the script as an unprivileged user. +Following these steps, autospec continues to rebuild the package, based on +new information discovered from build failures, until it has a valid +:file:`.spec`. If no build errors occur, RPM packages are successfully built. - .. code-block:: bash +Examples +******** - ./user-setup.sh +Complete `Setup environment to build source`_ before using these examples. -#. After the script completes, log out and log in again to complete - the setup process. +.. contents:: + :local: + :depth: 1 - The `user-setup script`_ creates a folder called :file:`clearlinux`, which - contains the :file:`Makefile`, :file:`packages`, and :file:`projects` - subfolders. +Example 1: Build RPM with existing spec file +============================================ - The :file:`projects` folder contains the main tools, `autospec` - and `common`, used for making packages in |CL|. +This example shows how to build a RPM from a pre-packaged upstream package, with +an existing spec file. The example uses the ``dmidecode`` package. -.. _install-tooling-end: - -Create a RPM with autospec -************************** - -Choose one of the following options to build RPMs and manage source -code: - -* :ref:`build-a-new-rpm` and spec file using ``make autospecnew``. - -* :ref:`build-source-code-with-existing-spec-file` using ``make build``, without changing the - spec file. - -* :ref:`generate-a-new-spec-file` using ``make autospec``, based on changes in the control files. - -.. _build-a-new-rpm: - -Option 1: Build a new RPM -========================= - -Use this method to build a new RPM with no spec file. In this example, -we build a new helloclear RPM. - -#. Navigate to the autospec workspace. +#. Navigate to the autospec workspace and clone the ``dmidecode`` package: .. code-block:: bash cd ~/clearlinux - -#. Enter the command: - - .. code-block:: bash - - make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" - NAME="helloclear" + make clone_dmidecode .. note:: - For a local tarball, use this type of *URL*: \file:// - -#. If build failures or dependency issues occur, continue below. - Otherwise, skip directly to `Next steps`_. - - #. Navigate to the specific package. + You can clone all package repos at once using: .. code-block:: bash - cd ~/clearlinux/packages/[package-name] + make [-j NUM] clone-packages - #. Respond to the build process output by editing control files to resolve - issues, which may include dependencies or exclusions. - See `autospec readme`_ + The optional NUM is the number of threads to use. - #. Run this command: + For a list of available packages, view the + :file:`~/clearlinux/projects/common/packages` file. - .. code-block:: bash - - make autospec - - Repeat the last two steps above until all errors are resolved and you - complete a successful build. - -**Congratulations!** - -You've successfully created a RPM. - -Skip to `Next steps`_. - -.. _build-source-code-with-existing-spec-file: - -Option 2: Build source code with an existing spec file -====================================================== - -Use this method if you only want to build the RPM using the spec file. This -method assumes that a spec file already exists. In this example, we run a -``make build`` on the ``dmidecode`` package. - -#. Navigate to the ``dmidecode`` package in clearlinux: +#. Navigate to the local copy of the ``dmidecode`` package and build it: .. code-block:: bash cd ~/clearlinux/packages/dmidecode/ - -#. To download the tarball and build, run the command: - - .. code-block:: bash - make build -**Congratulations!** +#. The resulting RPMs are in :file:`./rpms`. Build logs and additional RPMs are + in :file:`./results`. -You've successfully created a RPM. +Example 2: Build a new RPM +========================== -Skip to `Next steps`_. +This example shows how to build a new RPM with no spec file. The example will +create a simple helloclear RPM. -.. _generate-a-new-spec-file: - -Option 3: Generate a new spec file with a pre-defined package -============================================================= - -Use this method to modify an existing package. In this example, you will -modify an existing |CL| package called ``dmidecode`` to create a custom -RPM. You will make a simple change to this package, change the revision to -a new number that is higher than the |CL| OS version, and rebuild the package. - -#. Navigate to clearlinux: +#. Navigate to the autospec workspace and build the helloclear RPM. The + :file:`Makefile` provides a :command:`make autospecnew` that can + automatically generate an RPM package using the autospec tool. You must pass + the URL to the source tarball and the NAME of the RPM you wish to create: .. code-block:: bash cd ~/clearlinux + make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear" -#. Copy the ``dmidecode`` package. + The resulting RPMs are in :file:`./packages/helloclear/rpms`. Builde logs and + additional RPMs are in :file:`./packages/helloclear/results`. + +Example 3: Generate a new spec file with a pre-defined package +============================================================== + +This example shows how to modify an existing package to create a custom RPM. In +this example you will make a simple change to the ``dmidecode`` package and +rebuild the package. + +#. Navigate to the autospec workspace and clone the ``dmidecode`` package: .. code-block:: bash + cd ~/clearlinux make clone_dmidecode #. Navigate into the *dmidecode* directory: @@ -185,9 +170,9 @@ a new number that is higher than the |CL| OS version, and rebuild the package. cd packages/dmidecode -#. With an editor, open the :file:`excludes` file and add these lines: +#. Open the :file:`excludes` file with an editor and add these lines: - .. code-block:: bash + .. code-block:: console /usr/bin/biosdecode /usr/bin/ownership @@ -201,49 +186,211 @@ a new number that is higher than the |CL| OS version, and rebuild the package. These files aren't needed by dmidecode, so we can remove them without any issues. -#. Save the file and exit. - -#. At :file:`~/clearlinux/packages/dmidecode`, build the modified - ``dmidecode`` package: +#. In the :file:`dmidecode` directory, build the modified ``dmidecode`` package: .. code-block:: bash make autospec - When the process completes, you will see new RPM packages in the - :file:`results/` folder. +#. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. -#. To view the new RPM packages, enter: +Example 4: Provide control files to autospec +============================================ + +This example shows how to modify control files to correct build failures that +autospec is unable to resolve. In this example you will add a missing license +and dependencies in order for autospec to complete a successful build. + +#. Navigate to the autospec workspace: .. code-block:: bash - ls /clearlinux/packages/dmidecode/results/ + cd ~/clearlinux -**Congratulations!** +#. If you have not already, clone all upstream package repos: -You've successfully created a RPM. + .. code-block:: bash -Next steps + make [-j NUM] clone-packages + + The optional NUM is the number of threads to use. + + .. note:: + + In a later step of this example, we will search the cloned package repos + for a missing dependency. + +#. Build the opae-sdk RPM: + + .. code-block:: bash + + make autospecnew URL="https://github.com/OPAE/opae-sdk/archive/0.13.0.tar.gz" NAME="opae-sdk" + + This will give an error for a missing license file: + + .. code-block:: console + + [FATAL] Cannot find any license or opae-sdk.license file! + +#. Navigate to the package with build failures: + + .. code-block:: bash + + cd packages/opae-sdk + +#. Add a license: + + .. code-block:: bash + + echo "Intel Corporation" > opae-sdk.license + +#. Run autospec again: + + .. code-block:: bash + + make autospec + + This will result in a generic error: + + .. code-block:: console + + [FATAL] Build failed, aborting + +#. Open the build log to view the error details: + + .. code-block:: bash + + cat ./results/build.log + + In the build log, you will find details for the specific failures. In this + instance, there are missing dependencies: + + .. code-block:: console + + CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: + CJSON_LIBRARY + linked by target "opae-c++-utils" in directory /builddir/build/BUILD/opae-sdk-0.13.0/tools/c++utilslib + json-c_LIBRARIES + linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae + libuuid_LIBRARIES + linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae + +#. Search the spec files of upstream |CL| packages to see if the json-c library + is availabe. In this case, it does exist and we'll add the json-c 'dev' + package into the buildreq_add: + + .. code-block:: bash + + grep 'json-c\.so$' ~/clearlinux/packages/*/*.spec + echo "json-c-dev" >> buildreq_add + + .. note:: + + This search step works only if the user cloned all of the upstream package + repos. In this example, upstream package repos were cloned in a previous + step. + +#. Search the spec files of upstream |CL| packages to see if the libuuid library + is available. In this case, it exists in the util-linux package, so we'll add + util-linux-dev package into the buildreq_add: + + .. code-block:: bash + + grep 'libuuid\.so$' ~/clearlinux/packages/*/*.spec + echo "util-linux-dev" >> buildreq_add + +#. Run autospec again and find the successfully-generated RPMs in the rpms + directory: + + .. code-block:: bash + + make autospec + +.. note:: + + If you need a dependency that does not exist in the |CL| repo, you must first + build it manually (see `Example 2: Build a new RPM`_), then add the repo so + that autospec knows the package exists. For example: + + .. code-block:: bash + + cd ~/clearlinux/packages/ + make repoadd + make repostatus + + You only need to add the dependency to the :file:`buildreq_add` control file + if autospec is not able to automatically find the correct dependency on its + own. + +References ********** -Now you can create a custom bundle with your new RPM and use it with |CL|: +Reference the `autospec README`_ for details regarding autospec commands and +options. -* Use the :ref:`Mixer tool ` to add a new bundle to your derivative of |CL|. -* Use the :ref:`Mixin tool ` to customize your upstream |CL| installation with a new bundle. +Setup environment to build source +================================= + +.. _install-tooling-after-header: + +Setup of the workspace and tooling used for building source in |CL| is mostly +automated for you with a setup script. It uses tools from the +:command:`os-clr-on-clr` bundle. + +The setup script creates a workspace in the :file:`clearlinux` folder, with the +subfolders :file:`Makefile`, :file:`packages`, and :file:`projects`. The +:file:`projects` folder contains the main tools used for making packages in +|CL|: `autospec` and `common`. + +Follow these steps to setup the workspace and tooling for building source: + +#. Install the :command:`os-clr-on-clr` bundle: + + .. code-block:: bash + + sudo swupd bundle-add os-clr-on-clr + +#. Download the :file:`user-setup.sh` script: + + .. code-block:: bash + + curl -O https://raw.githubusercontent.com/clearlinux/common/master/user-setup.sh + +#. Make :file:`user-setup.sh` executable: + + .. code-block:: bash + + chmod +x user-setup.sh + +#. Run the script as an unprivileged user: + + .. code-block:: bash + + ./user-setup.sh + +#. After the script completes, log out and log in again to complete the setup + process. + +#. Set your Git user email and username for the repos on your system: + + .. code-block:: bash + + git config --global user.email "you@example.com" + git config --global user.name "Your Name" + + This global setting is used by |CL| tools that make use of Git. + +.. _install-tooling-end: Related topics ************** * :ref:`Mixer tool ` * :ref:`Mixin tool ` -* :ref:`autospec ` -* :ref:`Bundles ` - - -.. _rpm website: http://rpm.org - -.. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/ .. _user-setup script: https://github.com/clearlinux/common/blob/master/user-setup.sh - -.. _autospec readme: https://github.com/clearlinux/autospec +.. _autospec README: https://github.com/clearlinux/autospec +.. _control files: https://github.com/clearlinux/autospec#control-files +.. _mock wiki: https://github.com/rpm-software-management/mock/wiki +.. _rpm website: http://rpm.org +.. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/