diff --git a/source/guides/clear/telemetrics.rst b/source/guides/clear/telemetrics.rst index 6e0fa4ab..42d152b8 100644 --- a/source/guides/clear/telemetrics.rst +++ b/source/guides/clear/telemetrics.rst @@ -7,12 +7,13 @@ This guide describes the |CL-ATTR| telemetry solution. .. important:: - Telemetry in |CL| is **opt-in**. The telemetry client is **not** active + Telemetry in |CL| is **opt-in**. The telemetry client is **not** active and sends **no** data until you explicitly enable it. .. note:: - The telemetry functionality adheres to `Intel privacy policies `_ + The telemetry functionality adheres to + `Intel privacy policies `_ regarding the collection and use of :abbr:`PII (Personally Identifiable Information)` and is open source. No intentionally identifiable information about the user or system owner is @@ -33,7 +34,8 @@ client side for instrumenting your code for debug and analysis. Telemetry, one of the key features of |CL|, enables developers to observe and proactively address issues in the OS before end users are impacted. -Telemetrics is a combination word made from: +Telemetrics is a +`portmanteau word `_ made from: * Telemetry, which is sensing and reporting data. * Analytics, which is using visualization and statistical inferencing to make @@ -45,11 +47,12 @@ error checks, and the BIOS error report table for unhandled hardware failures. Telemetry enables real-time issue reporting to allow system developers to focus quickly on an issue and monitor corrective actions. -|CL| telemetry is fully customizable and can be used during software +|CL| telemetry is fully customizable and can also be used during software development for debugging purposes. You can use the libtelemetry library in your code to create custom telemetry records. You can also use the telem-record-gen utility in script files for light-touch record creation -where instrumenting code files doesn't make sense. +where instrumenting code files doesn't make sense. For more information on +configuring the telemetry client, refer to section `Client Configuration`_. The |CL| telemetrics solution is an **opt-in** choice on the client side. By default, the telemetry client is disabled until you choose to enable it. @@ -60,10 +63,11 @@ Architecture |CL| telemetry has two fundamental components, which are shown in Figure 1: -* Client: generates and delivers records to the backend server via the network. +* Client, which generates and delivers records to the backend server via the + network. -* Backend: receives records sent from the client and displays the cumulative - content through a specialized web interface. +* Backend, which receives records sent from the client and displays the + cumulative content through a specialized web interface. .. figure:: /_figures/telemetrics/telemetry-e2e.png :alt: Figure 1, Telemetry Architecture @@ -78,7 +82,8 @@ includes the following components: * telempostd, which is a daemon that manages spooled telemetry records and delivers these records according to configurable settings. * probes, which collect specific types of data from the operating system. -* libtelemetry, which is the API that telemetrics probes use to create records. +* libtelemetry, which is the API that telemetrics probes use to create + records. The telemetry backend provides the server-side component of the telemetrics solution and consists of: @@ -94,9 +99,10 @@ solution and consists of: .. note:: - The default telemetry backend server is hosted by the Intel |CL| development - team and is not viewable outside the Intel firewall. To collect your own - records, you must set up your own telemetry backend server. + The default telemetry backend server is hosted by the Intel |CL| + development team and is not viewable outside the Intel firewall. To + collect your own records, you must set up your own telemetry backend + server. How to use ********** @@ -104,33 +110,36 @@ How to use From a workflow perspective, the |CL| telemetrics system is straightforward. On the client side, the main decisions after installation and enabling telemetry involve what to do with the record data generated by the probes. -You can send the data to the default or a custom backend server, keep the data -local to the system, or both. The backend server has a more complex setup, but -once it's running, it is simple to use and configure. +You can send the data to the default telemetry server or a custom backend +server, keep the data local to the system, or both. The backend server has a +more complex setup, but once it's running, it is simple to configure and use. This section describes some of the possible scenarios for configuring the |CL| telemetrics system, and suggests which ones make sense according to your needs. +For more information on configuring the telemetry client, refer to section +`Client Configuration`_. + Scenarios ========= #. Enable telemetry: - Before probes can generate records, the telemetry client daemons must be - enabled. You can configure the client before enabling by creating a custom - :file:`telemetrics.conf` file that you place in the :file:`/etc/telemetrics` - directory. If you choose to use the default settings, records will be sent - to the telemetrics backend server managed by the |CL| development team at - Intel. + You must opt-in and start telemetry before probes can generate records. + You can configure the client before starting telemetry by creating a + custom :file:`telemetrics.conf` file that you place in the + :file:`/etc/telemetrics` directory. If you choose to use the built-in + default settings, records will be sent to the telemetrics backend server + managed by the |CL| development team at Intel. #. Save record data locally: You can configure the telemetry client to save records locally. This is - convenient when you want instant feedback during a development cycle, or to - track system issues if you believe there is a machine specific problem. The - client can be set not to send records at all, or to both keep the records - locally and send to the backend server. + convenient when you want instant feedback during a development cycle, or + to track system issues if you believe there is a machine-specific problem. + The client can be set not to send records at all or to both keep the + records locally and send to the backend server. #. Set up a server to collect data: @@ -142,10 +151,11 @@ Scenarios #. Instrument your code with the libtelemetry API: - The :command:`telemetrics` bundle includes the libtelemetry C library, which - exposes an API used by the telemprobd and telempostd daemons. You can use - these in your applications as well. The API documentation is found in the - :file:`telemetry.h` file in `Telemetrics client`_ repository. + The :command:`telemetrics` bundle includes the libtelemetry C library, + which exposes an API used by the telemprobd and telempostd daemons. You + can use these in your applications as well. The API documentation is + located in the :file:`telemetry.h` file in `Telemetrics client`_ + repository. Examples @@ -161,25 +171,29 @@ Enable or disable telemetry #. Enabling during installation: During the initial installation of |CL|, you are requested to join the - stability enhancement program and allow |CL| to collect anonymous reports to - improve system stability. If you choose not to join this program, then the - telemetry software bundle is not added to your system. Choosing to join will - automatically enable telemetry on your system after installation is - complete. + stability enhancement program and allow |CL| to collect anonymous reports + to improve system stability. If you choose not to join this program, then + the telemetry software bundle is not added to your system. If you do + choose to join the program, the installer will automatically enable + telemetry on your system by installing the telemetrics bundle, creating + the file :file:`/etc/telemetrics/opt-in`, and enabling the telemetrics + systemd services to run after installation is complete and the system is + restarted. #. Enabling after install: - To start telemetry on your system, run the following command: + To install telemetry on your system, run the following commands: .. code-block:: bash + sudo swupd bundle-add telemetrics + sudo telemctl opt-in sudo telemctl start - This enables and starts the :command:`telemprobd` and :command:`telempostd` - daemons. Your system will begin to send telemetry data to the server defined - in the file :file:`/etc/telemetrics/telemetrics.conf`. If this file does not - exist, the :command:`telemprobd` and :command:`telempostd` daemons will use - the file :file:`/usr/share/defaults/telemetrics/telemetrics.conf`. + This installs the necessary software, enables telemetry by creating the + file :file:`/etc/telemetrics/opt-in`, and starts the :command:`telemprobd` + and :command:`telempostd` daemons. Your system will begin to send + telemetry data to the backend server. #. Disabling after install: @@ -191,33 +205,27 @@ Enable or disable telemetry #. Opt in to telemetry: - To opt-in to the telemetry services, simply enter the opt-in command, which - also starts the service: + To opt-in to the telemetry services, simply enter the opt-in command: .. code-block:: bash sudo telemctl opt-in + sudo telemctl start - This removes the :file:`/etc/telemetrics/opt-out` file, if it exists, and - starts the telemetry services. - - .. note:: - - To opt-in but not immediately start telemetry services, you must - run the command :command:`sudo telemctl stop` after the :command:`opt-in` - command is entered. Once you are ready to start the service, enter the - command :command:`sudo telemctl start`. + This creates the :file:`/etc/telemetrics/opt-in` file, if it doesn't + already exist. You will need to explicitly start the telemetry services + after you have opted in. #. Opt out of telemetry: - To stop sending telemetrics data from your system, opt out of the telemetry - service: + To stop sending telemetrics data from your system, opt out of the + telemetry service: .. code-block:: bash sudo telemctl opt-out - This creates the file :file:`/etc/telemetrics/opt-out` and stops the + This removes the file :file:`/etc/telemetrics/opt-in` and stops the telemetry services. @@ -230,11 +238,12 @@ system. To change how records are managed, copy the default :file:`/usr/share/defaults/telemetrics/telemetrics.conf` file to :file:`/etc/telemetrics/telemetrics.conf` and edit it. The changes in the -:file:`/etc/telemetrics/telemetrics.conf` file will override the defaults in -the :file:`/usr/share/defaults/telemetrics/telemetrics.conf` file. You may need -root permissions to create and edit files in :file:`/etc`. For each -example, and for any time you make changes to the configuration file, you must -restart the client daemons to pick up the changes: +:file:`/etc/telemetrics/telemetrics.conf` file will override the built-in +defaults referenced in the +:file:`/usr/share/defaults/telemetrics/telemetrics.conf` file. +You will need root permissions to create and edit files in :file:`/etc`. For +each example, and for any time you make changes to the configuration file, +you must restart the client daemons to pick up the changes: .. code-block:: bash @@ -272,8 +281,8 @@ Use :command:`-h` or :command:`--help` to view usage options. this, you can use an IP address or fully qualified domain name. -Set up a back-end server to collect telemetry records -===================================================== +Set up a backend server to collect telemetry records +==================================================== For this example, start with a clean installation of |CL| on a new system using the :ref:`bare-metal-install-server` getting started guide and: @@ -287,7 +296,7 @@ using the :ref:`bare-metal-install-server` getting started guide and: * Create an administrative user named :guilabel:`clear` and add this user to sudoers -#. Log in with your administrative user, from your :file:`$HOME` directory, +#. Log in with your administrative user, from your :file:`$HOME` directory, run :command:`git` to clone the :guilabel:`telemetrics-backend` repository into the :file:`$HOME/telemetrics-backend` directory: @@ -300,9 +309,12 @@ using the :ref:`bare-metal-install-server` getting started guide and: You may need to set up the :envvar:`https_proxy` environment variable if you have issues reaching github.com. -#. Change your current working directory to :file:`telemetrics-backend/scripts`. -#. Before you install the telemetrics backend with the :file:`deploy.sh` script - file in the next step, here is an explanation of the options to be specified: +#. Change your current working directory to + :file:`telemetrics-backend/scripts`. + +#. Before you install the telemetrics backend with the :file:`deploy.sh` + script file in the next step, here is an explanation of the options to be + specified: * :command:`-a install` to perform an install * :command:`-d clr` to install to a |CL| distro @@ -320,8 +332,6 @@ using the :ref:`bare-metal-install-server` getting started guide and: ./deploy.sh -H localhost -a install -d clr - - The script starts and lists all the defined options and prompts you for the :guilabel:`PostgreSQL` database password. @@ -341,10 +351,10 @@ using the :ref:`bare-metal-install-server` getting started guide and: .. note:: - The :file:`deploy.sh` script uses :command:`sudo` to run commands and you - may be prompted to enter your user password at any time while the script - is executing. If this occurs, enter your user password to execute the - :command:`sudo` command. + The :file:`deploy.sh` script uses :command:`sudo` to run commands and + you may be prompted to enter your user password at any time while the + script is executing. If this occurs, enter your user password to + execute the :command:`sudo` command. #. After all the server components have been installed, you are prompted to @@ -359,13 +369,13 @@ using the :ref:`bare-metal-install-server` getting started guide and: passwd: password updated successfully Enter `postgres` for the current value of the password and then enter a new - password, retype it to verify the new password and the :guilabel:`PostgreSQL` - database password will be updated. + password. Retype it to verify the new password and the + :guilabel:`PostgreSQL` database password will be updated. -#. After the installation is complete, you can use your web browser to view the - new server by opening the browser on the system and typing in localhost - in the address bar. You should see a web page similar to the one shown in - Figure 2 below. +#. After the installation is complete, you can use your web browser to view + the new server by opening the browser on the system and typing in + :command:`localhost` in the address bar. You should see a web page similar + to the one shown in Figure 2 below. .. figure:: /_figures/telemetrics/telemetry-backend-1.png :alt: Telemetry UI @@ -375,12 +385,12 @@ using the :ref:`bare-metal-install-server` getting started guide and: Create records with telem-record-gen ==================================== -The telemetrics bundle provides a record generator tool called +The :command:`telemetrics` bundle provides a record generator tool called `telem-record-gen`. This tool can be used to create records from shell -scripts or the command line when writing a probe in C is not desirable. +scripts or the command line when it is not desirable to write a probe in C. Records are sent to the backend server, and can also be echoed to stdout. -There are three ways to supply the payload to the record. +There are three ways to supply the payload to the record: #. On the command line, use the :command:`-p ` option: @@ -464,15 +474,16 @@ There are three ways to supply the payload to the record. payload read from stdin + Set a static machine id ======================= The machine id reported by the telemetry client is rotated every three days for privacy reasons. If you wish to have a static machine id for testing -purposes, you can opt in by creating a file named `opt-in-static-machine-id` -in the directory :file:`/etc/telemetrics/`. +purposes, you can opt in by creating a file named +:file:`opt-in-static-machine-id` in the directory :file:`/etc/telemetrics/`. -#. Create a directory `telemetrics`. +#. Create a directory :file:`telemetrics`. .. code-block:: bash @@ -488,7 +499,7 @@ in the directory :file:`/etc/telemetrics/`. .. note:: - The machine ID is different than the system hostname. + The machine ID is different from the system hostname. Instrument your code with the libtelemetry API ============================================== @@ -497,9 +508,9 @@ Prerequisites ------------- Confirm that the telemetrics header file is located on the system at -:file:`usr/include/telemetry.h` The `latest version`_ of the file can also be -found on github for reference, but installing the :command:`telemetry` bundle -will install the header file that matches your |CL| version. +:file:`usr/include/telemetry.h`. The `latest version`_ of the file can also +be found on github for reference, but installing the :command:`telemetrics` +bundle will install the header file that matches your |CL| version. #. Includes and variables: @@ -531,33 +542,33 @@ will install the header file that matches your |CL| version. Severity: Type: uint32_t Value: Severity field value. Accepted values are in the range 1-4, with - 1 being the lowest severity, and 4 being the highest severity. Values - provided outside of this range are clamped to 1 or 4. [low, med, high, - crit] + 1 being the lowest severity and 4 being the highest severity. Values + provided outside of this range are clamped to 1 or 4 [low, med, high, + crit]. Payload_version: Type: uint32_t - Value: Payload format version. The only supported value right now is 1, + Value: Payload format version. The only currently supported value is 1, which indicates that the payload is a freely-formatted (unstructured) string. Values greater than 1 are reserved for future use. Classification: Type: char array - Value: It should have the form, DOMAIN/PROBENAME/REST: DOMAIN is the + Value: It should have the form, DOMAIN/PROBENAME/REST: DOMAIN is the reverse domain to use as a namespace for the probe (e.g. org.clearlinux), PROBENAME is the name of the probe, and REST is an arbitrary value that the probe should use to classify the record. The maximum length for the classification string is 122 bytes. Each sub-category may be no longer - than 40 bytes long. Two / delimiters are required. + than 40 bytes long. Two \'/\' delimiters are required. Tm_handle: Type: Telem_ref struct pointer - Value: Struct pointer declared by the caller, The struct is initialized + Value: Struct pointer declared by the caller. The struct is initialized if the function returns success. Payload: Type: char pointer - Value: The payload to set + Value: The payload to set. #. For this example, we'll set the payload to “hello” by using :command:`asprintf()`: @@ -568,19 +579,20 @@ will install the header file that matches your |CL| version. exit(EXIT_FAILURE); } - The functions :command:`asprintf()` and :command:`vasprintf()` are analogs of - :command:`sprintf(3)` and :command:`vsprintf(3)`, except that they allocate a - string large enough to hold the output including the terminating null byte - ('\0'), and return a pointer to it via the first argument. This pointer - should be passed to :command:`free(3)` to release the allocated storage when - it is no longer needed. + The functions :command:`asprintf()` and :command:`vasprintf()` are analogs + of :command:`sprintf(3)` and :command:`vsprintf(3)`, except that they + allocate a string large enough to hold the output including the + terminating null byte ('\0'), and return a pointer to it via the first + argument. This pointer should be passed to :command:`free(3)` to release + the allocated storage when it is no longer needed. #. Create the new telemetry record: - The function :command:`tm_create_record()` initializes a telemetry record and - sets the severity and classification of that record, as well as the payload - version number. The memory needed to store the telemetry record is allocated - and should be freed with :command:`tm_free_record()` when no longer needed. + The function :command:`tm_create_record()` initializes a telemetry + record and sets the severity and classification of that record, as well as + the payload version number. The memory needed to store the telemetry + record is allocated and should be freed with :command:`tm_free_record()` + when no longer needed. .. code-block:: console @@ -592,8 +604,9 @@ will install the header file that matches your |CL| version. #. Set the payload field of a telemetrics record: - The function :command:`tm_set_payload()` attaches the provided telemetry record - data to the telemetry record. The current maximum payload size is 8192b. + The function :command:`tm_set_payload()` attaches the provided telemetry + record data to the telemetry record. The current maximum payload size is + 8192b. .. code-block:: console @@ -604,8 +617,8 @@ will install the header file that matches your |CL| version. } free(payload); - The :command:`free()` function frees the memory space pointed to by `ptr`, which - must have been returned by a previous call to :command:`malloc()`, + The :command:`free()` function frees the memory space pointed to by `ptr`, + which must have been returned by a previous call to :command:`malloc()`, :command:`calloc()`, or :command:`realloc()`. Otherwise, or if :command:`free(ptr)` has already been called before, undefined behavior occurs. If `ptr` is NULL, no operation is performed. @@ -613,9 +626,9 @@ will install the header file that matches your |CL| version. #. Send a record to the telemetrics daemon: The function :command:`tm_send_record()` delivers the record to the local - :command:`telemprobd(1)` service. Since the telemetry record was allocated by - the program it should be freed with :command:`tm_free_record()` when it is no - longer needed. + :command:`telemprobd(1)` service. Since the telemetry record was allocated + by the program it should be freed with :command:`tm_free_record()` when it + is no longer needed. .. code-block:: console @@ -720,88 +733,109 @@ Reference The telemetry API ================= -Installing the :command:`telemetrics` bundle includes the libtelemetry C library, -which exposes an API used by the telemprobd and telempostd daemons. You can -use these in your applications as well. The API documentation is found in the -:file:`telemetry.h` file in `Telemetrics client`_ repository. +Installing the :command:`telemetrics` bundle includes the libtelemetry C +library, which exposes an API used by the telemprobd and telempostd daemons. +You can use these in your applications as well. The API documentation is found +in the :file:`telemetry.h` file in `Telemetrics client`_ repository. Client configuration ==================== The telemetry client will look for the configuration file located at :file:`/etc/telemetrics/telemetrics.conf` and use it if it exists. If the -file does not exist, the client will use the default configuration located -at :file:`/usr/share/defaults telemetrics/telemetrics.conf`. To modify or +file does not exist, the client will use the default configuration defined +at build time. There is a sample configuration file located at +:file:`/usr/share/defaults/telemetrics/telemetrics.conf` and represents the +default values that are used when the programs are built. To modify or customize the configuration, copy the file from -:file:`/usr/share/defaults/telemetrics` to :file:`/etc/telemetrics` and edit it. +:file:`/usr/share/defaults/telemetrics/telemetrics.conf` to the file +:file:`/etc/telemetrics/telemetrics.conf` and edit it to add your +customizations. + +.. code-block:: bash + + sudo mkdir -p /etc/telemetrics + cp /usr/share/defaults/telemetrics/telemetrics.conf /etc/telemetrics/telemetrics.conf + +.. note:: + + Telemetrics configuration is a layered mechanism since the defaults are + defined at build time and each field can be overwritten individually. + Therefore you only need to add the specific field that you want to change + from the default value to your customized value in the + :file:`/etc/telemetrics/telemetrics.conf` file. Configuration options --------------------- -The client uses the following configuration options from the config file: +The client can use the following configuration options from the config file: server - This specifies the web server to which telempostd sends the telemetry records. + This specifies the web server to which telempostd sends the telemetry + records. socket_path - This specifies the path of the unix domain socket on which the telemprobd + This specifies the path of the unix domain socket on which telemprobd listens for connections from the probes. spool_dir - This configuration option is related to spooling. If the daemon is not able - to send the telemetry records to the backend server due to reasons such as - the network availability, then it stores the records in a spool directory. - This option specifies the path of the spool directory. This directory should - be owned by the same user as the daemon. + This configuration option is related to spooling. If the daemon is not + able to send the telemetry records to the backend server due to reasons + such as the network availability, then it stores the records in a spool + directory. This option specifies the path of the spool directory. This + directory should be owned by the same user as the daemon. record_expiry - This is the time, in minutes, after which the records in the spool directory - are deleted by the daemon. + This is the time, in minutes, after which the records in the spool + directory are deleted by the daemon. spool_process_time - This specifies the time interval, in seconds, that the daemon waits for + This specifies the time interval, in seconds, that the daemon waits before checking the spool directory for records. The daemon picks up the - records in the order of modification date and tries to send the record to the - server. It sends a maximum of 10 records at a time. If it was able to send a - record successfully, it deletes the record from the spool. If the daemon - finds a record older than the "record_expiry" time, then it deletes that - record. The daemon looks at a maximum of 20 records in a single spool run loop. + records in the order of modification date and tries to send the record to + the server. It sends a maximum of 10 records at a time. If it was able to + send a record successfully, it deletes the record from the spool. If the + daemon finds a record older than the "record_expiry" time, then it deletes + that record. The daemon looks at a maximum of 20 records in a single spool + run loop. rate_limit_enabled - This determines whether rate-limiting is enabled or disabled. When enabled, - there is a threshold on both records sent within a window of time, and record - bytes sent within a window a time. + This determines whether rate-limiting is enabled or disabled. When + enabled, there is a threshold on both records sent within a window of + time, and record bytes sent within a window a time. record_burst_limit This is the maximum amount of records allowed to be passed by the daemon - within the record_window_length of time. If set to -1, the rate-limiting for - record bursts is disabled. + within the record_window_length of time. If set to -1, the rate-limiting + for record bursts is disabled. record_window_length The time, in minutes (0-59), that establishes the window length for the record_burst_limit. For example, if record_burst_window=1000 and - record_window_length=15, then no more than 1000 records can be passed within - any given fifteen-minute window. + record_window_length=15, then no more than 1000 records can be passed + within any given fifteen-minute window. byte_burst_limit - This is the maximum amount of bytes that can be passed by the daemon within - the byte_window_length of time. If set to -1, the rate-limiting for byte - bursts is disabled. + This is the maximum amount of bytes that can be passed by the daemon + within the byte_window_length of time. If set to -1, the rate-limiting + for byte bursts is disabled. byte_window_length - This is the time, in minutes (0-59), that establishes the window length for - the byte_burst_limit. + This is the time, in minutes (0-59), that establishes the window length + for the byte_burst_limit. rate_limit_strategy This is the strategy chosen once the rate-limiting threshold has been reached. Currently the options are 'drop' or 'spool', with spool being the - default. If spool is chosen, records will be spooled and sent at a later time. + default. If spool is chosen, records will be spooled and sent at a later + time. record_retention_enabled - When this key is enabled (true) the daemon saves a copy of the payload on - disk from all valid records. To avoid the excessive use of disk space only - the latest 100 records are kept. The default value for this configuration key - is false. + When this key is enabled (true), the daemon saves a copy of the payload on + disk from all valid records. To avoid the excessive use of disk space, + only the latest 100 records are kept. The default value for this + configuration key is false. record_server_delivery_enabled - This key controls the delivery of records to server; when enabled (default - value), the record will be posted to the address in the configuration file. - If this configuration key is disabled (false), records will not be spooled or - posted to backend. This configuration key can be used in combination with - record_retention_enabled to keep copies of telemetry records locally only. + This key controls the delivery of records to the server; when enabled + (default value), the record will be posted to the address in the + configuration file. If this configuration key is disabled (false), + records will not be spooled or posted to backend. This configuration key + can be used in combination with record_retention_enabled to keep copies + of telemetry records locally only. .. note:: Configuration options may change as the telemetry client evolves. - Please use the comments in the file itself as the most accurate + Please use the comments in the default file itself as the most accurate reference for configuration. @@ -831,7 +865,7 @@ for managing the telemetry services and probes. The tool is located in start/stop/restart ------------------ -The commands to start, stop and restart the telemetry services manage all +The commands to start, stop, and restart the telemetry services manage all required services and probes on the system. There is no need to separately start/stop/restart the two client daemons telemprobd and telempostd. The :command:`restart` command option will call :command:`telemctl stop`