From 32a703810b6333b52ab2ecc4d40d74a878843242 Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Tue, 12 Feb 2019 11:13:14 -0800 Subject: [PATCH 01/13] Update autospec guide - Apply guide template - Update examples per latest Signed-off-by: Kristal Dale --- .../guides/maintenance/autospec.rst | 258 +++++++++--------- 1 file changed, 134 insertions(+), 124 deletions(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index 257f8e2c..af8cac90 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -1,36 +1,91 @@ .. _autospec: -Build RPMs with autospec -######################## +autospec +######## -This guide shows you how to create RPMs with :ref:`autospec `, -a tool that assists in automated creation and maintenance of RPM packaging -on |CL-ATTR|. +autospec is a tool to assist in the automated creation and maintenance of RPM +packaging in |CL-ATTR|. Where a standard RPM build process using rpmbuild +requires a tarball and :file:`.spec` file to start, autospec requires only a +tarball and package name to start. -See our :ref:`autospec concept page ` for a detailed explaination -of how ``autospec`` works on |CL|. For a general understanding of how RPMs work, -we recommend visiting the `rpm website`_ or the `RPM Packaging Guide`_ . +.. contents:: + :local: + :depth: 1 + +.. Todo: this should be the concept content + +.. This guide shows you how to create RPMs with :ref:`autospec `, a tool that assists in automated creation and maintenance of RPM packaging on |CL-ATTR|. + +.. See our :ref:`autospec concept page ` for a detailed explaination of how ``autospec`` works on |CL|. For a general understanding of how RPMs work, we recommend visiting the `rpm website`_ or the `RPM Packaging Guide`_ . + +Description +*********** + +.. TODO + +How to use autospec +******************* + +Learn the autospec tool set up and workflow. + +.. contents:: + :local: + :depth: 1 Prerequisites -************* +============= -This guide requires that you: +#. **OS installed** -* Have installed |CL| on a host machine or virtual environment. For detailed - instructions on installing |CL|, visit the :ref:`get-started` section. + The |CL| must be installed to use the autospec tool. -* :ref:`install-tooling` +#. **Required bundles** -.. _install-tooling: + The autospec tool requires that the :command:`os-clr-on-clr` and bundle is + installed. -Install the |CL| tooling framework -================================== +Workflow +======== -#. Install the `os-clr-on-clr` developer bundle on your host system. +First time setup +---------------- - .. code-block:: bash +Before you use autospec, you will need to set up the autospec environment and +tools. This is mostly automated for you by using the provided +:file:`user_setup.sh` script. - sudo swupd bundle-add os-clr-on-clr +The `user-setup script`_ creates the :file:`clearlinux` folder, which contains +the :file:`Makefile`, :file:`packages`, and :file:`projects` subfolders. The +:file:`projects` folder contains the main tools, `autospec` and `common`, used +for making packages in |CL|. + +Create a RPM +------------ + +autospec helps build RPMs following these basic steps: + +#. Run autospec, passing in the tarball URL and package name. + +#. If there are build failures or dependency issues, supply the necessary + dependency, ban, or exclusion information via control files to autospec. + + View the `autospec README`_ for more information on control files. + + If a binary dependency doesn't exist in |CL|, you will need to build it + before running autospec again. + +#. Run autospec again. + +#. Repeat steps 2-3 until all errors are resolved, resulting in a successful + build. + +Examples +******** + +Example 1: First time setup +=========================== + +This example shows the basic steps for first time setup of autospec. #. Download the :file:`user-setup.sh` script. @@ -50,56 +105,62 @@ Install the |CL| tooling framework ./user-setup.sh -#. After the script completes, log out and log in again to complete - the setup process. +#. After the script completes, log out and log in again to complete the setup + process. - The `user-setup script`_ creates a folder called :file:`clearlinux`, which - contains the :file:`Makefile`, :file:`packages`, and :file:`projects` - subfolders. +#. Set your Git user email and username for the repos on your system - The :file:`projects` folder contains the main tools, `autospec` - and `common`, used for making packages in |CL|. + .. code-block:: bash -Create a RPM with autospec -************************** + git config --global user.email ”you@example.com" + git config --global user.name "Your Name" -Choose one of the following options to build RPMs and manage source -code: +.. TODO this last step (for GIT) - required? what is it doing? -* :ref:`build-a-new-rpm` and spec file using ``make autospecnew``. +Example 2: Build RPM with existing spec file +============================================ -* :ref:`build-source-code-with-existing-spec-file` using ``make build``, without changing the - spec file. +This example shows how to build a RPM from a pre-packaged upstream package with +an existing spec file. -* :ref:`generate-a-new-spec-file` using ``make autospec``, based on changes in the control files. +#. If you do not already have them locally, clone the |CL| package definitions: -.. _build-a-new-rpm: + .. code-block:: bash -Option 1: Build a new RPM -========================= + make -j clone-packages -Use this method to build a new RPM with no spec file. In this example, -we build a new helloclear RPM. + Alternately, you can clone a single package using: -#. Navigate to the autospec workspace. + .. code-block:: bash + + make clone_ + +#. Navigate to the ``dmidecode`` package and build it: + + .. code-block:: bash + + cd ~/clearlinux/packages/dmidecode/ + make build + +#. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. + +Example 3: Build a new package +============================== + +This example shows how to build a new RPM with no spec file. + +#. Navigate to the autospec workspace and build the helloclear RPM: .. code-block:: bash cd ~/clearlinux + make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear" -#. Enter the command: +#. The resulting RPMs are in :file:`~/clearlinux/packages/helloclear/rpms`. + Logs are in :file:`~/clearlinux/packages/helloclear/results`. - .. code-block:: bash - - make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" - NAME="helloclear" - - .. note:: - - For a local tarball, use this type of *URL*: \file:// - -#. If build failures or dependency issues occur, continue below. - Otherwise, skip directly to `Next steps`_. +#. If build failures or dependency issues occur, provide the necessary + dependency, ban, or exclusion information via control files to autospec: #. Navigate to the specific package. @@ -108,10 +169,10 @@ we build a new helloclear RPM. cd ~/clearlinux/packages/[package-name] #. Respond to the build process output by editing control files to resolve - issues, which may include dependencies or exclusions. - See `autospec readme`_ + issues, which may include dependencies or exclusions. See + `autospec README`_ for more information on control files. - #. Run this command: + #. Run autospec again: .. code-block:: bash @@ -120,59 +181,20 @@ we build a new helloclear RPM. Repeat the last two steps above until all errors are resolved and you complete a successful build. -**Congratulations!** -You've successfully created a RPM. +Example 4: Generate a new spec file with a pre-defined package +============================================================== -Skip to `Next steps`_. +This example shows how to modify an existing package to create a custom RPM. In +this example you will make a simple change to the ``dmidecode`` package, change +the revision to a new number higher than the |CL| OS version, and rebuild the +package. -.. _build-source-code-with-existing-spec-file: - -Option 2: Build source code with an existing spec file -====================================================== - -Use this method if you only want to build the RPM using the spec file. This -method assumes that a spec file already exists. In this example, we run a -``make build`` on the ``dmidecode`` package. - -#. Navigate to the ``dmidecode`` package in clearlinux: - - .. code-block:: bash - - cd ~/clearlinux/packages/dmidecode/ - -#. To download the tarball and build, run the command: - - .. code-block:: bash - - make build - -**Congratulations!** - -You've successfully created a RPM. - -Skip to `Next steps`_. - -.. _generate-a-new-spec-file: - -Option 3: Generate a new spec file with a pre-defined package -============================================================= - -Use this method to modify an existing package. In this example, you will -modify an existing |CL| package called ``dmidecode`` to create a custom -RPM. You will make a simple change to this package, change the revision to -a new number that is higher than the |CL| OS version, and rebuild the package. - -#. Navigate to clearlinux: +#. Navigate to the autospec workspace and copy the ``dmidecode`` package: .. code-block:: bash cd ~/clearlinux - -#. Copy the ``dmidecode`` package. - - .. code-block:: bash - make clone_dmidecode #. Navigate into the *dmidecode* directory: @@ -181,9 +203,9 @@ a new number that is higher than the |CL| OS version, and rebuild the package. cd packages/dmidecode -#. With an editor, open the :file:`excludes` file and add these lines: +#. Open the :file:`excludes` file with an editor and add these lines: - .. code-block:: bash + .. code-block:: console /usr/bin/biosdecode /usr/bin/ownership @@ -197,10 +219,7 @@ a new number that is higher than the |CL| OS version, and rebuild the package. These files aren't needed by dmidecode, so we can remove them without any issues. -#. Save the file and exit. - -#. At :file:`~/clearlinux/packages/dmidecode`, build the modified - ``dmidecode`` package: +#. In the :file:`dmidecode` directory, build the modified ``dmidecode`` package: .. code-block:: bash @@ -209,23 +228,17 @@ a new number that is higher than the |CL| OS version, and rebuild the package. When the process completes, you will see new RPM packages in the :file:`results/` folder. -#. To view the new RPM packages, enter: - - .. code-block:: bash - - ls /clearlinux/packages/dmidecode/results/ - -**Congratulations!** - -You've successfully created a RPM. +#. View the new RPM packages in :file:`/clearlinux/packages/dmidecode/results/` Next steps ********** -Now you can create a custom bundle with your new RPM and use it with |CL|: +Create a custom bundle and use it with |CL|: -* Use the :ref:`Mixer tool ` to add a new bundle to your derivative of |CL|. -* Use the :ref:`Mixin tool ` to customize your upstream |CL| installation with a new bundle. +* Use the :ref:`Mixer tool ` to add a new bundle to your derivative of + |CL|. +* Use the :ref:`Mixin tool ` to customize your upstream |CL| installation + with a new bundle. Related topics ************** @@ -235,11 +248,8 @@ Related topics * :ref:`autospec ` * :ref:`Bundles ` +.. _user-setup script: https://github.com/clearlinux/common/blob/master/user-setup.sh +.. _autospec README: https://github.com/clearlinux/autospec .. _rpm website: http://rpm.org - .. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/ - -.. _user-setup script: https://github.com/clearlinux/common/blob/master/user-setup.sh - -.. _autospec readme: https://github.com/clearlinux/autospec From 4102f8c8a55e04122dde70a494d9627ab05f6410 Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Thu, 14 Feb 2019 11:16:08 -0800 Subject: [PATCH 02/13] Correct substitution annotation in command examples. Misc minor edits. Signed-off-by: Kristal Dale --- .../guides/maintenance/autospec.rst | 20 ++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index af8cac90..b08a2f5c 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -41,7 +41,7 @@ Prerequisites #. **Required bundles** - The autospec tool requires that the :command:`os-clr-on-clr` and bundle is + The autospec tool requires that the :command:`os-clr-on-clr` bundle is installed. Workflow @@ -54,10 +54,9 @@ Before you use autospec, you will need to set up the autospec environment and tools. This is mostly automated for you by using the provided :file:`user_setup.sh` script. -The `user-setup script`_ creates the :file:`clearlinux` folder, which contains -the :file:`Makefile`, :file:`packages`, and :file:`projects` subfolders. The -:file:`projects` folder contains the main tools, `autospec` and `common`, used -for making packages in |CL|. +The `user-setup script`_ creates the autospec workspace in the +:file:`clearlinux` folder. The workspace contains the :file:`Makefile`, :file:`packages`, and :file:`projects` subfolders. The :file:`projects` folder contains +the main tools, `autospec` and `common`, used for making packages in |CL|. Create a RPM ------------ @@ -112,7 +111,7 @@ This example shows the basic steps for first time setup of autospec. .. code-block:: bash - git config --global user.email ”you@example.com" + git config --global user.email "you@example.com" git config --global user.name "Your Name" .. TODO this last step (for GIT) - required? what is it doing? @@ -123,11 +122,12 @@ Example 2: Build RPM with existing spec file This example shows how to build a RPM from a pre-packaged upstream package with an existing spec file. -#. If you do not already have them locally, clone the |CL| package definitions: +#. If you do not already have them locally, clone the |CL| package repos: .. code-block:: bash - make -j clone-packages + cd clearlinux + make [-j NUM] clone-packages Alternately, you can clone a single package using: @@ -144,6 +144,8 @@ an existing spec file. #. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. +.. _autospec-build-new-pack: + Example 3: Build a new package ============================== @@ -166,7 +168,7 @@ This example shows how to build a new RPM with no spec file. .. code-block:: bash - cd ~/clearlinux/packages/[package-name] + cd ~/clearlinux/packages/ #. Respond to the build process output by editing control files to resolve issues, which may include dependencies or exclusions. See From f5fc3113cf57c89ce9dfb553a3fa5d4af368b6d7 Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Thu, 14 Feb 2019 11:20:06 -0800 Subject: [PATCH 03/13] Correct example title Signed-off-by: Kristal Dale --- source/clear-linux/guides/maintenance/autospec.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index b08a2f5c..8767bd95 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -144,10 +144,10 @@ an existing spec file. #. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. -.. _autospec-build-new-pack: +.. _autospec-build-new-rpm: -Example 3: Build a new package -============================== +Example 3: Build a new RPM +========================== This example shows how to build a new RPM with no spec file. From 01622aa3b65334e06336ae893135c0eabfc64297 Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Fri, 15 Feb 2019 10:42:13 -0800 Subject: [PATCH 04/13] Add concept info to explanation section Signed-off-by: Kristal Dale --- .../guides/maintenance/autospec.rst | 98 ++++++++++--------- 1 file changed, 53 insertions(+), 45 deletions(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index 8767bd95..fbd02c59 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -3,30 +3,29 @@ autospec ######## -autospec is a tool to assist in the automated creation and maintenance of RPM -packaging in |CL-ATTR|. Where a standard RPM build process using rpmbuild -requires a tarball and :file:`.spec` file to start, autospec requires only a -tarball and package name to start. +**autospec** is a tool to assist in the automated creation and maintenance of +RPM packaging in |CL-ATTR|. Where a standard RPM build process using +:command:`rpmbuild` requires a tarball and :file:`.spec` file to start, autospec requires only a tarball and package name to start. .. contents:: :local: :depth: 1 -.. Todo: this should be the concept content - -.. This guide shows you how to create RPMs with :ref:`autospec `, a tool that assists in automated creation and maintenance of RPM packaging on |CL-ATTR|. - -.. See our :ref:`autospec concept page ` for a detailed explaination of how ``autospec`` works on |CL|. For a general understanding of how RPMs work, we recommend visiting the `rpm website`_ or the `RPM Packaging Guide`_ . - Description *********** -.. TODO +The autospec tool attempts to infer the requirements of the :file:`.spec` file +by analyzing the source code and :file:`Makefile` information. It will +continuously run updated builds based on new information discovered from build failures until it has a complete and valid :file:`.spec` file. The autospec tool makes use of mock to achieve this. It is possible to influence the exact +behavior of autospec by providing `control files`_. -How to use autospec -******************* +For a general understanding of how RPMs work, we recommend visiting the +`rpm website`_ or the `RPM Packaging Guide`_ . -Learn the autospec tool set up and workflow. +How to use +********** + +Learn the autospec tool set up and process. .. contents:: :local: @@ -61,26 +60,41 @@ the main tools, `autospec` and `common`, used for making packages in |CL|. Create a RPM ------------ -autospec helps build RPMs following these basic steps: +The basic autospec process is described in the following steps: -#. Run autospec, passing in the tarball URL and package name. +#. The :command:`make autospec` command generates a :file:`.spec` file based on + analysis of code and control files, if present. -#. If there are build failures or dependency issues, supply the necessary - dependency, ban, or exclusion information via control files to autospec. +#. autospec creates a ``build root`` with mock config. - View the `autospec README`_ for more information on control files. +#. autospec attempts to build an RPM from the generated :file:`.spec`. + +#. autospec detects any missed declarations in the :file:`.spec`. + +#. If build errors occur, autospec will scan the build log to try and detect + the root cause. + +#. If autospec detects the root cause and knows how to continue, it will restart + the build automatically at step 1 with updated build instructions. + +#. Otherwise, autospec will stop the build for user inspection and editing of + control files to resolve the errors. The user resumes the process at step 1 + after errors are resolved. If a binary dependency doesn't exist in |CL|, you will need to build it before running autospec again. -#. Run autospec again. + View the `autospec README`_ for more information on control files. -#. Repeat steps 2-3 until all errors are resolved, resulting in a successful - build. +Following these steps, autospec continues to rebuild the package, based on +new information discovered from build failures, until it has a valid +:file:`.spec`. If no build errors occur, RPM packages are successfully built. Examples ******** +Make sure all `Prerequisites`_ are fulfilled before using these examples. + Example 1: First time setup =========================== @@ -114,26 +128,27 @@ This example shows the basic steps for first time setup of autospec. git config --global user.email "you@example.com" git config --global user.name "Your Name" -.. TODO this last step (for GIT) - required? what is it doing? - Example 2: Build RPM with existing spec file ============================================ -This example shows how to build a RPM from a pre-packaged upstream package with -an existing spec file. +This example shows how to build a RPM from a pre-packaged upstream package, with +an existing spec file. The example uses the ``dmidecode`` package. -#. If you do not already have them locally, clone the |CL| package repos: +#. Make sure you have completed the + `First time set up for autospec `_. + +#. Clone the ``dmidecode`` package. You can clone all |CL| package repos: .. code-block:: bash cd clearlinux make [-j NUM] clone-packages - Alternately, you can clone a single package using: + Or you can clone a single package using: .. code-block:: bash - make clone_ + make clone_dmidecode #. Navigate to the ``dmidecode`` package and build it: @@ -144,14 +159,18 @@ an existing spec file. #. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. -.. _autospec-build-new-rpm: - Example 3: Build a new RPM ========================== -This example shows how to build a new RPM with no spec file. +This example shows how to build a new RPM with no spec file. The example will +create a simple helloclear RPM. -#. Navigate to the autospec workspace and build the helloclear RPM: +#. Navigate to the autospec workspace. The :file:`Makefile` provides a + :command:`make autospecnew` that can automatically generate an RPM package + using the autospec tool. You must pass the URL to the source tarball and the + NAME of the RPM you wish to create. + + Build the helloclear RPM: .. code-block:: bash @@ -183,7 +202,6 @@ This example shows how to build a new RPM with no spec file. Repeat the last two steps above until all errors are resolved and you complete a successful build. - Example 4: Generate a new spec file with a pre-defined package ============================================================== @@ -232,16 +250,6 @@ package. #. View the new RPM packages in :file:`/clearlinux/packages/dmidecode/results/` -Next steps -********** - -Create a custom bundle and use it with |CL|: - -* Use the :ref:`Mixer tool ` to add a new bundle to your derivative of - |CL|. -* Use the :ref:`Mixin tool ` to customize your upstream |CL| installation - with a new bundle. - Related topics ************** @@ -252,6 +260,6 @@ Related topics .. _user-setup script: https://github.com/clearlinux/common/blob/master/user-setup.sh .. _autospec README: https://github.com/clearlinux/autospec - +.. _control files: https://github.com/clearlinux/autospec#control-files .. _rpm website: http://rpm.org .. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/ From d1c921abf95d4fb5653d9d5495b9617f2e58fd85 Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Fri, 15 Feb 2019 13:47:15 -0800 Subject: [PATCH 05/13] Make wording across examples more consistent, remove redundant steps. Signed-off-by: Kristal Dale --- .../guides/maintenance/autospec.rst | 79 +++++++++++-------- 1 file changed, 46 insertions(+), 33 deletions(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index fbd02c59..6e7aa873 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -5,7 +5,8 @@ autospec **autospec** is a tool to assist in the automated creation and maintenance of RPM packaging in |CL-ATTR|. Where a standard RPM build process using -:command:`rpmbuild` requires a tarball and :file:`.spec` file to start, autospec requires only a tarball and package name to start. +:command:`rpmbuild` requires a tarball and :file:`.spec` file to start, autospec +requires only a tarball and package name to start. .. contents:: :local: @@ -16,11 +17,13 @@ Description The autospec tool attempts to infer the requirements of the :file:`.spec` file by analyzing the source code and :file:`Makefile` information. It will -continuously run updated builds based on new information discovered from build failures until it has a complete and valid :file:`.spec` file. The autospec tool makes use of mock to achieve this. It is possible to influence the exact +continuously run updated builds based on new information discovered from build +failures until it has a complete and valid :file:`.spec` file. The autospec tool +makes use of mock to achieve this. It is possible to influence the exact behavior of autospec by providing `control files`_. -For a general understanding of how RPMs work, we recommend visiting the -`rpm website`_ or the `RPM Packaging Guide`_ . +For a general understanding of how RPMs work, visit the `rpm website`_ or the +`RPM Packaging Guide`_ . How to use ********** @@ -54,8 +57,10 @@ tools. This is mostly automated for you by using the provided :file:`user_setup.sh` script. The `user-setup script`_ creates the autospec workspace in the -:file:`clearlinux` folder. The workspace contains the :file:`Makefile`, :file:`packages`, and :file:`projects` subfolders. The :file:`projects` folder contains -the main tools, `autospec` and `common`, used for making packages in |CL|. +:file:`clearlinux` folder. The workspace contains the :file:`Makefile`, +:file:`packages`, and :file:`projects` subfolders. The :file:`projects` folder +contains the main tools, `autospec` and `common`, used for making packages in +|CL|. Create a RPM ------------ @@ -95,6 +100,10 @@ Examples Make sure all `Prerequisites`_ are fulfilled before using these examples. +.. contents:: + :local: + :depth: 1 + Example 1: First time setup =========================== @@ -134,23 +143,23 @@ Example 2: Build RPM with existing spec file This example shows how to build a RPM from a pre-packaged upstream package, with an existing spec file. The example uses the ``dmidecode`` package. -#. Make sure you have completed the - `First time set up for autospec `_. +Make sure you have completed `Example 1: First time setup`_ before starting +this example. -#. Clone the ``dmidecode`` package. You can clone all |CL| package repos: - - .. code-block:: bash - - cd clearlinux - make [-j NUM] clone-packages - - Or you can clone a single package using: +#. Navigate to the autospec workspace and clone the ``dmidecode`` package. .. code-block:: bash + cd ~/clearlinux make clone_dmidecode -#. Navigate to the ``dmidecode`` package and build it: + Or you can clone all package repos at once using: + + .. code-block:: bash + + make [-j NUM] clone-packages + +#. Navigate to the local copy of the ``dmidecode`` package and build it: .. code-block:: bash @@ -165,20 +174,20 @@ Example 3: Build a new RPM This example shows how to build a new RPM with no spec file. The example will create a simple helloclear RPM. -#. Navigate to the autospec workspace. The :file:`Makefile` provides a - :command:`make autospecnew` that can automatically generate an RPM package - using the autospec tool. You must pass the URL to the source tarball and the - NAME of the RPM you wish to create. +Make sure you have completed `Example 1: First time setup`_ before starting +this example. - Build the helloclear RPM: +#. Navigate to the autospec workspace and build the helloclear RPM. The + :file:`Makefile` provides a :command:`make autospecnew` that can + automatically generate an RPM package using the autospec tool. You must pass + the URL to the source tarball and the NAME of the RPM you wish to create: .. code-block:: bash cd ~/clearlinux make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear" -#. The resulting RPMs are in :file:`~/clearlinux/packages/helloclear/rpms`. - Logs are in :file:`~/clearlinux/packages/helloclear/results`. +#. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. #. If build failures or dependency issues occur, provide the necessary dependency, ban, or exclusion information via control files to autospec: @@ -199,8 +208,8 @@ create a simple helloclear RPM. make autospec - Repeat the last two steps above until all errors are resolved and you - complete a successful build. + #. Repeat the last two steps above until all errors are resolved and you + complete a successful build. Example 4: Generate a new spec file with a pre-defined package ============================================================== @@ -210,13 +219,22 @@ this example you will make a simple change to the ``dmidecode`` package, change the revision to a new number higher than the |CL| OS version, and rebuild the package. -#. Navigate to the autospec workspace and copy the ``dmidecode`` package: +Make sure you have completed `Example 1: First time setup`_ before starting +this example. + +#. Navigate to the autospec workspace and clone the ``dmidecode`` package. .. code-block:: bash cd ~/clearlinux make clone_dmidecode + Or you can clone all package repos at once using: + + .. code-block:: bash + + make [-j NUM] clone-packages + #. Navigate into the *dmidecode* directory: .. code-block:: bash @@ -245,18 +263,13 @@ package. make autospec - When the process completes, you will see new RPM packages in the - :file:`results/` folder. - -#. View the new RPM packages in :file:`/clearlinux/packages/dmidecode/results/` +#. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. Related topics ************** * :ref:`Mixer tool ` * :ref:`Mixin tool ` -* :ref:`autospec ` -* :ref:`Bundles ` .. _user-setup script: https://github.com/clearlinux/common/blob/master/user-setup.sh .. _autospec README: https://github.com/clearlinux/autospec From 8f089a86e5adaa2278caca828e6048ea2fadc421 Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Tue, 19 Feb 2019 13:46:23 -0800 Subject: [PATCH 06/13] Revisions based on feedback from @bktan8. Added a fifth example, showing how to provide control files to autospec when there is a build failure. Signed-off-by: Kristal Dale --- .../guides/maintenance/autospec.rst | 214 +++++++++++++----- 1 file changed, 159 insertions(+), 55 deletions(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index 6e7aa873..cc22ad33 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -18,9 +18,12 @@ Description The autospec tool attempts to infer the requirements of the :file:`.spec` file by analyzing the source code and :file:`Makefile` information. It will continuously run updated builds based on new information discovered from build -failures until it has a complete and valid :file:`.spec` file. The autospec tool -makes use of mock to achieve this. It is possible to influence the exact -behavior of autospec by providing `control files`_. +failures until it has a complete and valid :file:`.spec` file. You can influence +the behavior of autospec and customize the build by providing `control files`_ +to the autospec tool. + +autospec uses mock as a sandbox to run the builds. Visit the `mock wiki`_ for +additional information on using mock. For a general understanding of how RPMs work, visit the `rpm website`_ or the `RPM Packaging Guide`_ . @@ -49,7 +52,7 @@ Prerequisites Workflow ======== -First time setup +First-time setup ---------------- Before you use autospec, you will need to set up the autospec environment and @@ -62,15 +65,20 @@ The `user-setup script`_ creates the autospec workspace in the contains the main tools, `autospec` and `common`, used for making packages in |CL|. -Create a RPM ------------- +Create an RPM +------------- The basic autospec process is described in the following steps: #. The :command:`make autospec` command generates a :file:`.spec` file based on - analysis of code and control files, if present. + analysis of code and existing control files. -#. autospec creates a ``build root`` with mock config. + Control files should be located in the same directory as the resulting + :file:`.spec` file. + + View the `autospec README`_ for more information on `control files`_. + +#. autospec creates a build root with mock config. #. autospec attempts to build an RPM from the generated :file:`.spec`. @@ -82,15 +90,17 @@ The basic autospec process is described in the following steps: #. If autospec detects the root cause and knows how to continue, it will restart the build automatically at step 1 with updated build instructions. -#. Otherwise, autospec will stop the build for user inspection and editing of - control files to resolve the errors. The user resumes the process at step 1 - after errors are resolved. +#. Otherwise, autospec will stop the build for user inspection to resolve the + errors. Respond to the build process output by fixing source code issues + and/or editing control files to resolve issues, which may include + dependencies or exclusions. See `autospec README`_ for more information on + control files. + + The user resumes the process at step 1 after errors are resolved. If a binary dependency doesn't exist in |CL|, you will need to build it before running autospec again. - View the `autospec README`_ for more information on control files. - Following these steps, autospec continues to rebuild the package, based on new information discovered from build failures, until it has a valid :file:`.spec`. If no build errors occur, RPM packages are successfully built. @@ -109,19 +119,19 @@ Example 1: First time setup This example shows the basic steps for first time setup of autospec. -#. Download the :file:`user-setup.sh` script. +#. Download the :file:`user-setup.sh` script: .. code-block:: bash curl -O https://raw.githubusercontent.com/clearlinux/common/master/user-setup.sh -#. Make :file:`user-setup.sh` executable. +#. Make :file:`user-setup.sh` executable: .. code-block:: bash chmod +x user-setup.sh -#. Run the script as an unprivileged user. +#. Run the script as an unprivileged user: .. code-block:: bash @@ -130,7 +140,7 @@ This example shows the basic steps for first time setup of autospec. #. After the script completes, log out and log in again to complete the setup process. -#. Set your Git user email and username for the repos on your system +#. Set your Git user email and username for the repos on your system: .. code-block:: bash @@ -146,18 +156,25 @@ an existing spec file. The example uses the ``dmidecode`` package. Make sure you have completed `Example 1: First time setup`_ before starting this example. -#. Navigate to the autospec workspace and clone the ``dmidecode`` package. +#. Navigate to the autospec workspace and clone the ``dmidecode`` package: .. code-block:: bash cd ~/clearlinux make clone_dmidecode - Or you can clone all package repos at once using: + .. note:: - .. code-block:: bash + You can clone all package repos at once using: - make [-j NUM] clone-packages + .. code-block:: bash + + make [-j NUM] clone-packages + + The optional NUM is the number of threads to use. + + For a list of available packages, view the + :file:`~/clearlinux/projects/common/packages` file. #. Navigate to the local copy of the ``dmidecode`` package and build it: @@ -166,7 +183,8 @@ this example. cd ~/clearlinux/packages/dmidecode/ make build -#. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. +#. The resulting RPMs are in :file:`./rpms`. Build logs and additional RPMs are + in :file:`./results`. Example 3: Build a new RPM ========================== @@ -187,54 +205,26 @@ this example. cd ~/clearlinux make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear" -#. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. - -#. If build failures or dependency issues occur, provide the necessary - dependency, ban, or exclusion information via control files to autospec: - - #. Navigate to the specific package. - - .. code-block:: bash - - cd ~/clearlinux/packages/ - - #. Respond to the build process output by editing control files to resolve - issues, which may include dependencies or exclusions. See - `autospec README`_ for more information on control files. - - #. Run autospec again: - - .. code-block:: bash - - make autospec - - #. Repeat the last two steps above until all errors are resolved and you - complete a successful build. + The resulting RPMs are in :file:`./packages/helloclear/rpms`. Builde logs and + additional RPMs are in :file:`./packages/helloclear/results`. Example 4: Generate a new spec file with a pre-defined package ============================================================== This example shows how to modify an existing package to create a custom RPM. In -this example you will make a simple change to the ``dmidecode`` package, change -the revision to a new number higher than the |CL| OS version, and rebuild the -package. +this example you will make a simple change to the ``dmidecode`` package and +rebuild the package. Make sure you have completed `Example 1: First time setup`_ before starting this example. -#. Navigate to the autospec workspace and clone the ``dmidecode`` package. +#. Navigate to the autospec workspace and clone the ``dmidecode`` package: .. code-block:: bash cd ~/clearlinux make clone_dmidecode - Or you can clone all package repos at once using: - - .. code-block:: bash - - make [-j NUM] clone-packages - #. Navigate into the *dmidecode* directory: .. code-block:: bash @@ -265,6 +255,119 @@ this example. #. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. +Example 5: Provide control files to autospec +============================================ + +This example shows how to modify control files to correct build failures that +autospec is unable to resolve. In this example you will add a missing license +and dependencies in order for autospec to complete a successful build. + +Make sure you have completed `Example 1: First time setup`_ before starting +this example. + +#. Navigate to the autospec workspace: + + .. code-block:: bash + + cd ~/clearlinux + +#. Build the opae-sdk RPM: + + .. code-block:: bash + + make autospecnew URL="https://github.com/OPAE/opae-sdk/archive/0.13.0.tar.gz" NAME="opae-sdk" + + This will give an error for a missing license file: + + .. code-block:: console + + [FATAL] Cannot find any license or opae-sdk.license file! + +#. Navigate to the package with build failures: + + .. code-block:: bash + + cd packages/opae-sdk + +#. Add a license: + + .. code-block:: bash + + echo "MIT" > opae-sdk.license + +#. Run autospec again: + + .. code-block:: bash + + make autospec + + This will result in a generic error: + + .. code-block:: console + + [FATAL] Build failed, aborting + +#. Open the build log to view the error details: + + .. code-block:: bash + + cat ./results/build.log + + In the build log, you will find details for the specific failures. In this + instance, there are missing dependencies: + + .. code-block:: console + + CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: + CJSON_LIBRARY + linked by target "opae-c++-utils" in directory /builddir/build/BUILD/opae-sdk-0.13.0/tools/c++utilslib + json-c_LIBRARIES + linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae + libuuid_LIBRARIES + linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae + +#. Search for the json-c package in |CL|: + + .. code-block:: bash + + grep 'json-c\.so$' ~/clearlinux/packages/*/*.spec + +#. Add the json-c-dev dependency: + + .. code-block:: bash + + echo "json-c-dev" >> buildreq_add + +#. Search for the libuuid package in |CL| and add the util-linux-dev dependency: + + .. code-block:: bash + + grep 'libuuid\.so$' ~/clearlinux/packages/*/*.spec + echo "util-linux-dev" >> buildreq_add + +#. Run autospec again and find the successfully-generated RPMs in the rpms + directory: + + .. code-block:: bash + + make autospec + +.. note:: + + If you need a dependency that does not exist in the |CL| repo, you must first + build it manually, then add the repo so that autospec knows the package + exists. For example: + + .. code-block:: bash + + cd ~/clearlinux/packages/ + make repoadd + make repostatus + + You only need to add the dependency to the :file:`buildreq_add` control file + if autospec is not able to automatically guess the correct dependency on its + own. + Related topics ************** @@ -274,5 +377,6 @@ Related topics .. _user-setup script: https://github.com/clearlinux/common/blob/master/user-setup.sh .. _autospec README: https://github.com/clearlinux/autospec .. _control files: https://github.com/clearlinux/autospec#control-files +.. _mock wiki: https://github.com/rpm-software-management/mock/wiki .. _rpm website: http://rpm.org .. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/ From fa70e17e544cea18eee21a19039d0b13bdc4c7d7 Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Wed, 20 Feb 2019 15:23:27 -0800 Subject: [PATCH 07/13] Clean up prereq list formatting per template. Signed-off-by: Kristal Dale --- source/clear-linux/guides/maintenance/autospec.rst | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index cc22ad33..a6fb9362 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -40,14 +40,11 @@ Learn the autospec tool set up and process. Prerequisites ============= -#. **OS installed** +* |CL| - The |CL| must be installed to use the autospec tool. + :ref:`bare-metal-install`. -#. **Required bundles** - - The autospec tool requires that the :command:`os-clr-on-clr` bundle is - installed. +* :command:`os-clr-on-clr` bundle. Workflow ======== From 745723c17e3e8c146094ce0e90764c77fe84c1ce Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Wed, 20 Feb 2019 16:03:06 -0800 Subject: [PATCH 08/13] Added link to add bundle. Signed-off-by: Kristal Dale --- source/clear-linux/guides/maintenance/autospec.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index a6fb9362..ca0cac19 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -44,7 +44,9 @@ Prerequisites :ref:`bare-metal-install`. -* :command:`os-clr-on-clr` bundle. +* :command:`os-clr-on-clr` bundle + + `Install a bundle`_. Workflow ======== @@ -377,3 +379,4 @@ Related topics .. _mock wiki: https://github.com/rpm-software-management/mock/wiki .. _rpm website: http://rpm.org .. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/ +.. _Install a bundle: https://clearlinux.org/documentation/clear-linux/guides/maintenance/swupd-guide#adding-a-bundle From 8b81e3061530f1c58fe82dd81be71fe33186033a Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Thu, 21 Feb 2019 13:54:50 -0800 Subject: [PATCH 09/13] Minor update around prereq format to bring inline with template. Signed-off-by: Kristal Dale --- source/clear-linux/guides/maintenance/autospec.rst | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index ca0cac19..38bdc418 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -40,13 +40,8 @@ Learn the autospec tool set up and process. Prerequisites ============= -* |CL| - - :ref:`bare-metal-install`. - -* :command:`os-clr-on-clr` bundle - - `Install a bundle`_. +The autospec tool requires that the :command:`os-clr-on-clr` bundle is +installed. Refer to `Install a bundle`_ for more details. Workflow ======== From b51dae4a5a8386c0405a1597f01b7a2ef700b152 Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Wed, 27 Feb 2019 11:22:30 -0800 Subject: [PATCH 10/13] Minor edits for consistency with CL tooling guide layout. Signed-off-by: Kristal Dale --- .../clear-linux/guides/maintenance/autospec.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index 38bdc418..95e29a5c 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -28,8 +28,8 @@ additional information on using mock. For a general understanding of how RPMs work, visit the `rpm website`_ or the `RPM Packaging Guide`_ . -How to use -********** +How it works +************ Learn the autospec tool set up and process. @@ -43,11 +43,8 @@ Prerequisites The autospec tool requires that the :command:`os-clr-on-clr` bundle is installed. Refer to `Install a bundle`_ for more details. -Workflow -======== - First-time setup ----------------- +================ Before you use autospec, you will need to set up the autospec environment and tools. This is mostly automated for you by using the provided @@ -60,7 +57,7 @@ contains the main tools, `autospec` and `common`, used for making packages in |CL|. Create an RPM -------------- +============= The basic autospec process is described in the following steps: @@ -102,7 +99,10 @@ new information discovered from build failures, until it has a valid Examples ******** -Make sure all `Prerequisites`_ are fulfilled before using these examples. +Complete all `Prerequisites`_ before using these examples. + +Reference the `autospec README`_ for details regarding autospec commands and +options. .. contents:: :local: From c1485ce26414c85a82dccee0a80d33dd86c1c005 Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Fri, 1 Mar 2019 14:17:18 -0800 Subject: [PATCH 11/13] Update wording around prerequisites and example of first time setup to be presented more as the setup for building source in CL (rather than autospec specific). So that the setup instructions may be reused. Added section labels back in for external reference. Signed-off-by: Kristal Dale --- .../guides/maintenance/autospec.rst | 60 +++++++++++-------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index 95e29a5c..597b363a 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -40,21 +40,11 @@ Learn the autospec tool set up and process. Prerequisites ============= -The autospec tool requires that the :command:`os-clr-on-clr` bundle is -installed. Refer to `Install a bundle`_ for more details. +The setup for building source in |CL| must be completed before using the +autospec tool. -First-time setup -================ - -Before you use autospec, you will need to set up the autospec environment and -tools. This is mostly automated for you by using the provided -:file:`user_setup.sh` script. - -The `user-setup script`_ creates the autospec workspace in the -:file:`clearlinux` folder. The workspace contains the :file:`Makefile`, -:file:`packages`, and :file:`projects` subfolders. The :file:`projects` folder -contains the main tools, `autospec` and `common`, used for making packages in -|CL|. +Refer to `Example 1: Complete setup for building source`_ for +instructions on completing setup. Create an RPM ============= @@ -108,10 +98,26 @@ options. :local: :depth: 1 -Example 1: First time setup -=========================== +Example 1: Complete setup for building source +============================================= -This example shows the basic steps for first time setup of autospec. +.. _install-tooling-after-header: + +Setup of the workspace and tooling used for building source in |CL| is mostly +automated for you with a setup script. It uses tools from the +:command:`os-clr-on-clr` bundle. + +The setup script creates a workspace in the :file:`clearlinux` folder, with the +subfolders :file:`Makefile`, :file:`packages`, and :file:`projects`. The :file:`projects` folder contains the main tools used for making packages in +|CL|: `autospec` and `common`. + +Follow these steps to setup the workspace and tooling for building source: + +#. Install the :command:`os-clr-on-clr` bundle: + + .. code-block:: bash + + sudo swupd bundle-add os-clr-on-clr #. Download the :file:`user-setup.sh` script: @@ -141,14 +147,18 @@ This example shows the basic steps for first time setup of autospec. git config --global user.email "you@example.com" git config --global user.name "Your Name" + This global setting is used by |CL| tools that make use of Git. + +.. _install-tooling-end: + Example 2: Build RPM with existing spec file ============================================ This example shows how to build a RPM from a pre-packaged upstream package, with an existing spec file. The example uses the ``dmidecode`` package. -Make sure you have completed `Example 1: First time setup`_ before starting -this example. +Complete `Example 1: Complete setup for building source`_ before starting this +example. #. Navigate to the autospec workspace and clone the ``dmidecode`` package: @@ -186,8 +196,8 @@ Example 3: Build a new RPM This example shows how to build a new RPM with no spec file. The example will create a simple helloclear RPM. -Make sure you have completed `Example 1: First time setup`_ before starting -this example. +Complete `Example 1: Complete setup for building source`_ before starting this +example. #. Navigate to the autospec workspace and build the helloclear RPM. The :file:`Makefile` provides a :command:`make autospecnew` that can @@ -209,8 +219,8 @@ This example shows how to modify an existing package to create a custom RPM. In this example you will make a simple change to the ``dmidecode`` package and rebuild the package. -Make sure you have completed `Example 1: First time setup`_ before starting -this example. +Complete `Example 1: Complete setup for building source`_ before starting this +example. #. Navigate to the autospec workspace and clone the ``dmidecode`` package: @@ -256,8 +266,8 @@ This example shows how to modify control files to correct build failures that autospec is unable to resolve. In this example you will add a missing license and dependencies in order for autospec to complete a successful build. -Make sure you have completed `Example 1: First time setup`_ before starting -this example. +Complete `Example 1: Complete setup for building source`_ before starting this +example. #. Navigate to the autospec workspace: From 553f4f359cb864c6cf3d24498837f0d0805a8ae6 Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Fri, 1 Mar 2019 16:26:08 -0800 Subject: [PATCH 12/13] Revisions based on feedback from @bktan8. - Wording improvments - Move instructions for required setup for building source to References section. This follows similar org to the mixer guide, keeping emphasis in 'how it works' on the description of what is required and how it works, and keeping the examples section as just examples of autospec. Idea is this setup is a supporting reference for using/enabling the tool. Signed-off-by: Kristal Dale --- .../guides/maintenance/autospec.rst | 471 +++++++++--------- 1 file changed, 240 insertions(+), 231 deletions(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index 597b363a..f53a7564 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -43,8 +43,8 @@ Prerequisites The setup for building source in |CL| must be completed before using the autospec tool. -Refer to `Example 1: Complete setup for building source`_ for -instructions on completing setup. +Refer to `Complete setup for building source`_ for instructions on completing +setup. Create an RPM ============= @@ -89,17 +89,247 @@ new information discovered from build failures, until it has a valid Examples ******** -Complete all `Prerequisites`_ before using these examples. - -Reference the `autospec README`_ for details regarding autospec commands and -options. +`Complete setup for building source`_ before using these examples. .. contents:: :local: :depth: 1 -Example 1: Complete setup for building source -============================================= +Example 1: Build RPM with existing spec file +============================================ + +This example shows how to build a RPM from a pre-packaged upstream package, with +an existing spec file. The example uses the ``dmidecode`` package. + +#. Navigate to the autospec workspace and clone the ``dmidecode`` package: + + .. code-block:: bash + + cd ~/clearlinux + make clone_dmidecode + + .. note:: + + You can clone all package repos at once using: + + .. code-block:: bash + + make [-j NUM] clone-packages + + The optional NUM is the number of threads to use. + + For a list of available packages, view the + :file:`~/clearlinux/projects/common/packages` file. + +#. Navigate to the local copy of the ``dmidecode`` package and build it: + + .. code-block:: bash + + cd ~/clearlinux/packages/dmidecode/ + make build + +#. The resulting RPMs are in :file:`./rpms`. Build logs and additional RPMs are + in :file:`./results`. + +Example 2: Build a new RPM +========================== + +This example shows how to build a new RPM with no spec file. The example will +create a simple helloclear RPM. + +#. Navigate to the autospec workspace and build the helloclear RPM. The + :file:`Makefile` provides a :command:`make autospecnew` that can + automatically generate an RPM package using the autospec tool. You must pass + the URL to the source tarball and the NAME of the RPM you wish to create: + + .. code-block:: bash + + cd ~/clearlinux + make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear" + + The resulting RPMs are in :file:`./packages/helloclear/rpms`. Builde logs and + additional RPMs are in :file:`./packages/helloclear/results`. + +Example 3: Generate a new spec file with a pre-defined package +============================================================== + +This example shows how to modify an existing package to create a custom RPM. In +this example you will make a simple change to the ``dmidecode`` package and +rebuild the package. + +#. Navigate to the autospec workspace and clone the ``dmidecode`` package: + + .. code-block:: bash + + cd ~/clearlinux + make clone_dmidecode + +#. Navigate into the *dmidecode* directory: + + .. code-block:: bash + + cd packages/dmidecode + +#. Open the :file:`excludes` file with an editor and add these lines: + + .. code-block:: console + + /usr/bin/biosdecode + /usr/bin/ownership + /usr/bin/vpddecode + /usr/share/man/man8/biosdecode.8 + /usr/share/man/man8/ownership.8 + /usr/share/man/man8/vpddecode.8 + + .. note:: + + These files aren't needed by dmidecode, so we can remove them without + any issues. + +#. In the :file:`dmidecode` directory, build the modified ``dmidecode`` package: + + .. code-block:: bash + + make autospec + +#. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. + +Example 4: Provide control files to autospec +============================================ + +This example shows how to modify control files to correct build failures that +autospec is unable to resolve. In this example you will add a missing license +and dependencies in order for autospec to complete a successful build. + +#. Navigate to the autospec workspace: + + .. code-block:: bash + + cd ~/clearlinux + +#. If you have not already, clone all upstream package repos: + + .. code-block:: bash + + make [-j NUM] clone-packages + + The optional NUM is the number of threads to use. + + .. note:: + + In a later step of this example, we will search the cloned package repos + for a missing dependency. + +#. Build the opae-sdk RPM: + + .. code-block:: bash + + make autospecnew URL="https://github.com/OPAE/opae-sdk/archive/0.13.0.tar.gz" NAME="opae-sdk" + + This will give an error for a missing license file: + + .. code-block:: console + + [FATAL] Cannot find any license or opae-sdk.license file! + +#. Navigate to the package with build failures: + + .. code-block:: bash + + cd packages/opae-sdk + +#. Add a license: + + .. code-block:: bash + + echo "Intel Corporation" > opae-sdk.license + +#. Run autospec again: + + .. code-block:: bash + + make autospec + + This will result in a generic error: + + .. code-block:: console + + [FATAL] Build failed, aborting + +#. Open the build log to view the error details: + + .. code-block:: bash + + cat ./results/build.log + + In the build log, you will find details for the specific failures. In this + instance, there are missing dependencies: + + .. code-block:: console + + CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: + CJSON_LIBRARY + linked by target "opae-c++-utils" in directory /builddir/build/BUILD/opae-sdk-0.13.0/tools/c++utilslib + json-c_LIBRARIES + linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae + libuuid_LIBRARIES + linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae + +#. Search the spec files of upstream |CL| packages to see if the json-c library + is availabe. In this case, it does exist and we'll add the json-c 'dev' + package into the buildreq_add: + + .. code-block:: bash + + grep 'json-c\.so$' ~/clearlinux/packages/*/*.spec + echo "json-c-dev" >> buildreq_add + + .. note:: + + This search step works only if the user cloned all of the upstream package + repos. In this example, upstream package repos were cloned in a previous + step. + +#. Search the spec files of upstream |CL| packages to see if the libuuid library + is available. In this case, it exists in the util-linux package, so we'll add + util-linux-dev package into the buildreq_add: + + .. code-block:: bash + + grep 'libuuid\.so$' ~/clearlinux/packages/*/*.spec + echo "util-linux-dev" >> buildreq_add + +#. Run autospec again and find the successfully-generated RPMs in the rpms + directory: + + .. code-block:: bash + + make autospec + +.. note:: + + If you need a dependency that does not exist in the |CL| repo, you must first + build it manually (see `Example 2: Build a new RPM`_), then add the repo so + that autospec knows the package exists. For example: + + .. code-block:: bash + + cd ~/clearlinux/packages/ + make repoadd + make repostatus + + You only need to add the dependency to the :file:`buildreq_add` control file + if autospec is not able to automatically find the correct dependency on its + own. + +References +********** + +Reference the `autospec README`_ for details regarding autospec commands and +options. + +Complete setup for building source +================================== .. _install-tooling-after-header: @@ -108,7 +338,8 @@ automated for you with a setup script. It uses tools from the :command:`os-clr-on-clr` bundle. The setup script creates a workspace in the :file:`clearlinux` folder, with the -subfolders :file:`Makefile`, :file:`packages`, and :file:`projects`. The :file:`projects` folder contains the main tools used for making packages in +subfolders :file:`Makefile`, :file:`packages`, and :file:`projects`. The +:file:`projects` folder contains the main tools used for making packages in |CL|: `autospec` and `common`. Follow these steps to setup the workspace and tooling for building source: @@ -151,227 +382,6 @@ Follow these steps to setup the workspace and tooling for building source: .. _install-tooling-end: -Example 2: Build RPM with existing spec file -============================================ - -This example shows how to build a RPM from a pre-packaged upstream package, with -an existing spec file. The example uses the ``dmidecode`` package. - -Complete `Example 1: Complete setup for building source`_ before starting this -example. - -#. Navigate to the autospec workspace and clone the ``dmidecode`` package: - - .. code-block:: bash - - cd ~/clearlinux - make clone_dmidecode - - .. note:: - - You can clone all package repos at once using: - - .. code-block:: bash - - make [-j NUM] clone-packages - - The optional NUM is the number of threads to use. - - For a list of available packages, view the - :file:`~/clearlinux/projects/common/packages` file. - -#. Navigate to the local copy of the ``dmidecode`` package and build it: - - .. code-block:: bash - - cd ~/clearlinux/packages/dmidecode/ - make build - -#. The resulting RPMs are in :file:`./rpms`. Build logs and additional RPMs are - in :file:`./results`. - -Example 3: Build a new RPM -========================== - -This example shows how to build a new RPM with no spec file. The example will -create a simple helloclear RPM. - -Complete `Example 1: Complete setup for building source`_ before starting this -example. - -#. Navigate to the autospec workspace and build the helloclear RPM. The - :file:`Makefile` provides a :command:`make autospecnew` that can - automatically generate an RPM package using the autospec tool. You must pass - the URL to the source tarball and the NAME of the RPM you wish to create: - - .. code-block:: bash - - cd ~/clearlinux - make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear" - - The resulting RPMs are in :file:`./packages/helloclear/rpms`. Builde logs and - additional RPMs are in :file:`./packages/helloclear/results`. - -Example 4: Generate a new spec file with a pre-defined package -============================================================== - -This example shows how to modify an existing package to create a custom RPM. In -this example you will make a simple change to the ``dmidecode`` package and -rebuild the package. - -Complete `Example 1: Complete setup for building source`_ before starting this -example. - -#. Navigate to the autospec workspace and clone the ``dmidecode`` package: - - .. code-block:: bash - - cd ~/clearlinux - make clone_dmidecode - -#. Navigate into the *dmidecode* directory: - - .. code-block:: bash - - cd packages/dmidecode - -#. Open the :file:`excludes` file with an editor and add these lines: - - .. code-block:: console - - /usr/bin/biosdecode - /usr/bin/ownership - /usr/bin/vpddecode - /usr/share/man/man8/biosdecode.8 - /usr/share/man/man8/ownership.8 - /usr/share/man/man8/vpddecode.8 - - .. note:: - - These files aren't needed by dmidecode, so we can remove them without - any issues. - -#. In the :file:`dmidecode` directory, build the modified ``dmidecode`` package: - - .. code-block:: bash - - make autospec - -#. The resulting RPMs are in :file:`./rpms`. Logs are in :file:`./results`. - -Example 5: Provide control files to autospec -============================================ - -This example shows how to modify control files to correct build failures that -autospec is unable to resolve. In this example you will add a missing license -and dependencies in order for autospec to complete a successful build. - -Complete `Example 1: Complete setup for building source`_ before starting this -example. - -#. Navigate to the autospec workspace: - - .. code-block:: bash - - cd ~/clearlinux - -#. Build the opae-sdk RPM: - - .. code-block:: bash - - make autospecnew URL="https://github.com/OPAE/opae-sdk/archive/0.13.0.tar.gz" NAME="opae-sdk" - - This will give an error for a missing license file: - - .. code-block:: console - - [FATAL] Cannot find any license or opae-sdk.license file! - -#. Navigate to the package with build failures: - - .. code-block:: bash - - cd packages/opae-sdk - -#. Add a license: - - .. code-block:: bash - - echo "MIT" > opae-sdk.license - -#. Run autospec again: - - .. code-block:: bash - - make autospec - - This will result in a generic error: - - .. code-block:: console - - [FATAL] Build failed, aborting - -#. Open the build log to view the error details: - - .. code-block:: bash - - cat ./results/build.log - - In the build log, you will find details for the specific failures. In this - instance, there are missing dependencies: - - .. code-block:: console - - CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: - CJSON_LIBRARY - linked by target "opae-c++-utils" in directory /builddir/build/BUILD/opae-sdk-0.13.0/tools/c++utilslib - json-c_LIBRARIES - linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae - libuuid_LIBRARIES - linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae - -#. Search for the json-c package in |CL|: - - .. code-block:: bash - - grep 'json-c\.so$' ~/clearlinux/packages/*/*.spec - -#. Add the json-c-dev dependency: - - .. code-block:: bash - - echo "json-c-dev" >> buildreq_add - -#. Search for the libuuid package in |CL| and add the util-linux-dev dependency: - - .. code-block:: bash - - grep 'libuuid\.so$' ~/clearlinux/packages/*/*.spec - echo "util-linux-dev" >> buildreq_add - -#. Run autospec again and find the successfully-generated RPMs in the rpms - directory: - - .. code-block:: bash - - make autospec - -.. note:: - - If you need a dependency that does not exist in the |CL| repo, you must first - build it manually, then add the repo so that autospec knows the package - exists. For example: - - .. code-block:: bash - - cd ~/clearlinux/packages/ - make repoadd - make repostatus - - You only need to add the dependency to the :file:`buildreq_add` control file - if autospec is not able to automatically guess the correct dependency on its - own. - Related topics ************** @@ -384,4 +394,3 @@ Related topics .. _mock wiki: https://github.com/rpm-software-management/mock/wiki .. _rpm website: http://rpm.org .. _RPM Packaging Guide: https://rpm-packaging-guide.github.io/ -.. _Install a bundle: https://clearlinux.org/documentation/clear-linux/guides/maintenance/swupd-guide#adding-a-bundle From dbb8ba03bd361b48167541da5ba3d54851aea80e Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Mon, 4 Mar 2019 12:46:42 -0800 Subject: [PATCH 13/13] Wording clarification in response to feedback from @mvincerx Signed-off-by: Kristal Dale --- .../clear-linux/guides/maintenance/autospec.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/maintenance/autospec.rst index f53a7564..0b955d3d 100644 --- a/source/clear-linux/guides/maintenance/autospec.rst +++ b/source/clear-linux/guides/maintenance/autospec.rst @@ -18,9 +18,9 @@ Description The autospec tool attempts to infer the requirements of the :file:`.spec` file by analyzing the source code and :file:`Makefile` information. It will continuously run updated builds based on new information discovered from build -failures until it has a complete and valid :file:`.spec` file. You can influence -the behavior of autospec and customize the build by providing `control files`_ -to the autospec tool. +failures until it has a complete and valid :file:`.spec` file. If needed, you +can influence the behavior of autospec and customize the build by providing +optional `control files`_ to the autospec tool. autospec uses mock as a sandbox to run the builds. Visit the `mock wiki`_ for additional information on using mock. @@ -43,7 +43,7 @@ Prerequisites The setup for building source in |CL| must be completed before using the autospec tool. -Refer to `Complete setup for building source`_ for instructions on completing +Refer to `Setup environment to build source`_ for instructions on completing setup. Create an RPM @@ -54,7 +54,7 @@ The basic autospec process is described in the following steps: #. The :command:`make autospec` command generates a :file:`.spec` file based on analysis of code and existing control files. - Control files should be located in the same directory as the resulting + Any control files should be located in the same directory as the resulting :file:`.spec` file. View the `autospec README`_ for more information on `control files`_. @@ -89,7 +89,7 @@ new information discovered from build failures, until it has a valid Examples ******** -`Complete setup for building source`_ before using these examples. +Complete `Setup environment to build source`_ before using these examples. .. contents:: :local: @@ -328,8 +328,8 @@ References Reference the `autospec README`_ for details regarding autospec commands and options. -Complete setup for building source -================================== +Setup environment to build source +================================= .. _install-tooling-after-header: