clearlinux/clear-linux-documentation
2
0
Fork 0
mirror of https://github.com/clearlinux/clear-linux-documentation.git synced 2026-08-26 02:18:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #276 from mvincerx/add-telemetry-guides

Browse Source 
Adds telemetry.rst landing page and 5 new guides:
This commit is contained in:
michael vincerra
2018-11-06 17:09:41 -08:00
committed by GitHub
parent 98f6c7c5d5 af933879ab
commit 4f288d0239
8 changed files with 577 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/clear-linux/guides/guides.rst
+1
View File
@@ -19,3 +19,4 @@ after completing the |CL| :ref:`installation <get-started>`.
maintenance/maintenance
network/network
deploy-at-scale
telemetrics/telemetrics
source/clear-linux/guides/maintenance/maintenance.rst
-1
View File
@@ -18,7 +18,6 @@ completed.
mixer
mixin
validate-signatures
telemetry-enable
time
hostname
increase-virtual-disk-size
source/clear-linux/guides/telemetrics/telemctl.rst
+64
View File
@@ -0,0 +1,64 @@
.. _telemctl:
telemctl options
################
The |CL-ATTR| telemetry client provides an admin tool called telemctl for
managing the telemetry services and probes. The tool is located in
:file:` /usr/bin`. Running it with no argument results in the following:
.. code-block:: bash
sudo telemctl
.. code-block:: console
/usr/bin/telemctl - Control actions for telemetry services
stop Stops all running telemetry services
start Starts all telemetry services
restart Restarts all telemetry services
is-active Checks if telemprobd and telempostd are active
opt-in Opts in to telemetry, and starts telemetry services
opt-out Opts out of telemetry, and stops telemetry services
journal Prints telemetry journal contents. Use -h argument for more
options
telemctl commands:
******************
start/stop/restart
==================
The commands to start, stop and restart the telemetry services manage all
required services and probes on the system. There is no need to separately
start/stop/restart the two client daemons **telemprobd** and **telempostd**.
The **restart** command option will call **telemctl stop** followed by
**telemctl start** .
is-active
=========
The `is-active` option reports whether the two client daemons are active.
This is useful to verify that the **opt-in** and **opt-out** options have
taken effect, or to ensure that telemetry is functioning on the system.
Note that both daemons are verified.
.. code-block:: bash
sudo telemctl is-active
.. code-block:: console
telemprobd : active
telempostd : active
.. include:: ./telemetry-enable.rst
:start-after: incl-opt-in-out-telemetry:
:end-before: incl-opt-in-out-telemetry-end:
Next steps
==========
Learn to read records:
* :ref:`telemetry-journal`
source/clear-linux/guides/telemetrics/telemetrics.rst
+19
View File
@@ -0,0 +1,19 @@
.. _telemetrics:
Telemetry
#########
The |CL-ATTR| telemetrics solution collects data from running |CL| systems
and helps to quickly identify and fix bugs in the OS. These guides will walk
you through setup, configuration, and customization of the telemetry client. The data collected from the client system is analyzed and presented by the
telemetry backend solution. For more details, learn how to
:ref:`telemetry-backend`.
.. toctree::
:maxdepth: 1
telemetry-enable
telemetry-config
telemctl
telemetry-journal
telemetry-api
source/clear-linux/guides/telemetrics/telemetry-api.rst
+170
View File
@@ -0,0 +1,170 @@
.. _telemetry-api:
Telemetry API
#############
Installing the ``telemetrics`` bundle includes the libtelemetry C library,
which exposes an API used by the telemprobd and telempostd daemons. You
can use these in your applications as well. The API documentation is found
in the :file:`telemetry.h` file in `Telemetrics client`_ repository.
Creating records with telem-record-gen
**************************************
The telemetrics bundle also provides a record generator tool called
``telem-record-gen``. This tool can be used to create records from shell
scripts, etc., when writing a probe in C is not desirable. Records are sent
to the backend server, and can also be echoed to stdout.
telem-record-gen usage
======================
.. code-block:: bash
telem-record-gen [OPTIONS] - create and send a custom telemetry record
.. code-block:: console
Help Options:
-h, --help Show help options
Application Options:
-V, --version Print the program version
-s, --severity Severity level (1-4) - (default 1)
-c, --class Classification level_1/level_2/level_3 (required)
-p, --payload Record body (max size = 8k) (required)
-P, --payload-file File to read payload from
-R, --record-version Version number for format of payload (default 1)
-e, --event-id Event id to use in the record
-o, --echo Echo record to stdout
-n, --no-post Do not post record just print
The :command:`-c` and :command:`-p` options are required; defaults are
supplied for most other options. The maximum payload size is 8k
(8192 bytes). Excess is ignored, regardless of source (file/commandline/
stdin). An empty payload is allowed, but even an empty payload must be
specified in one of the three ways shown below.
telem-record-gen examples
=========================
There are three ways to supply the payload to the record.
#. On the command line, use the :command:`-p <string>` option:
.. code-block:: bash
telem-record-gen -c a/b/c -n -o -p 'payload goes here'
.. code-block:: console
record_format_version: 4
classification: a/b/c
severity: 1
machine_id: FFFFFFFF
creation_timestamp: 1539023189
arch: x86_64
host_type: innotek GmbH|VirtualBox|1.2
build: 25180
kernel_version: 4.14.71-404.lts
payload_format_version: 1
system_name: clear-linux-os
board_name: VirtualBox|Oracle Corporation
cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
bios_version: VirtualBox
event_id: 2236710e4fc11e4a646ce956c7802788
payload goes here
#. Specify a file that contains the payload with the option
:command:'-P path/to/file'.
.. code-block:: bash
telem-record-gen -c a/b/c -n -o -P ./payload_file.txt
.. code-block:: console
record_format_version: 4
classification: a/b/c
severity: 1
machine_id: FFFFFFFF
creation_timestamp: 1539023621
arch: x86_64
host_type: innotek GmbH|VirtualBox|1.2
build: 25180
kernel_version: 4.14.71-404.lts
payload_format_version: 1
system_name: clear-linux-os
board_name: VirtualBox|Oracle Corporation
cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
bios_version: VirtualBox
event_id: d73d6040afd7693cccdfece479df9795
payload read from file
#. If the :command:`-p` or :command:`-P` options are absent, the tool reads
from stdin so you can use it in a :file:`heredoc` in scripts.
.. code-block:: bash
telem-record-gen -c a/b/c -n -o << HEOF
payload read from stdin
HEOF
.. code-block:: console
record_format_version: 4
classification: a/b/c
severity: 1
machine_id: FFFFFFFF
creation_timestamp: 1539023621
arch: x86_64
host_type: innotek GmbH|VirtualBox|1.2
build: 25180
kernel_version: 4.14.71-404.lts
payload_format_version: 1
system_name: clear-linux-os
board_name: VirtualBox|Oracle Corporation
cpu_model: Intel(R) Core(TM) i7-4650U CPU @ 1.70GHz
bios_version: VirtualBox
event_id: 2f070e8e71679f2b1f28794e3a6c42ee
payload read from stdin
.. note::
Although only the classification and payload are specified, the tool supplies values for the remaining values.
Telemetry records and the REST API
==================================
If you have not configured the telemetry client to keep records locally, you
can view them using the Web UI of the server, or you can query them from the
server using the REST API provided by |CL| telemetrics. The API is
available at :file:`<server>/api/records`, and when queried, returns a JSON
response that contains a list of records. There are several parameters for
filtering queries, similar to the filters available through the telemetryui Records view.
* classification: The classification of the record
* severity: The severity of the record. Restricted to integer value
* machine_id: The id of the machine where this record was generated on
* build: The build on which the record was generated. Restricted to 256
characters.
* created_in_days: causes the query to return records created after the last
given days
* created_in_sec: returns the records created after the last given seconds
* limit: The maximum number of records to be returned.
Next Steps
==========
* :ref:`telemetry-backend`
* `Telemetrics client`_
Related topics
==============
* :ref:`telemetry-about`
.. _Telemetrics client: https://github.com/clearlinux/telemetrics-client/
source/clear-linux/guides/telemetrics/telemetry-config.rst
+101
View File
@@ -0,0 +1,101 @@
.. _telemetry-config:
Telemetry client configuration
##############################
The telemetry client will look for the configuration file located at
:file:`/etc/telemetrics/telemetrics.conf` and use it if it exists. If the
file does not exist, the client will use the default configuration located
at :file:`/usr/share/defaults/telemetrics/telemetrics.conf`. To modify or
customize the configuration, copy the file from
:file:`/usr/share/defaults/telemetrics` to
:file:`/etc/telemetrics` and edit it.
Configuration Options
*********************
The client uses the following configuration options from the config file:
* **server**: This specifies the web server to which telempostd sends the
telemetry records.
* **socket_path**: This specifies the path of the unix domain socket that
the telemprobd listens on for connections from the probes.
* **spool_dir**: This configuration option is related to spooling. If the
daemon is not able to send the telemetry records to the backend server due
to reasons such as the network availability, then it stores the records in
a spool directory. This option specifies that path of the spool directory.
This directory should be owned by the same user as the daemon.
- mkdir -p /var/spool/telemetry
- chown -R telemetry:telemetry /var/spool/telemetry
- systemctl restart telemprobd.service
* **record_expiry**: This is the time in minutes after which the records in
the spool directory are deleted by the daemon.
* **spool_process_time**: This specifies the time interval in seconds that
the daemon waits for before checking the spool directory for records. The
daemon picks up the records in the order of modification date and tries to
send the record to the server. It sends a maximum of 10 records at a time.
If it was able to send a record successfully, it deletes the record from
the spool. If the daemon finds a record older than the "record_expiry"
time, then it deletes that record. The daemon looks at a maximum of 20
records in a single spool run loop.
* **rate_limit_enabled**: This determines whether rate-limiting is enabled or
disabled. When enabled, there is a threshold on both records sent within a
window of time, and record bytes sent within a window a time.
* **record_burst_limit**: This is the maximum amount of records allowed to be
passed by the daemon within the record_window_length of time. If set to
-1, the rate-limiting for record bursts is disabled.
* **record_window_length**: The time in minutes (0-59) that
establishes the window length for the record_burst_limit. EX: if
record_burst_window=1000 and record_window_length=15, then no more than
1000 records can be passed within any given fifteen minute window.
* **byte_burst_limit**: This is the maximum amount of bytes that can be
passed by the daemon within the byte_window_length of time. If set to -1, the rate-limiting for byte bursts is disabled.
* **byte_window_length**: This is the time, in minutes (0-59), that
establishes the window length for the byte_burst_limit.
* **rate_limit_strategy**: This is the strategy chosen once the rate-limiting
threshold has been reached. Currently the options are 'drop' or 'spool',
with spool being the default. If spool is chosen, records will be spooled
and sent at a later time.
* **record_retention_enabled**: When this key is enabled (true) the daemon
saves a copy of the payload on disk from all valid records. To avoid the
excessive use of disk space only the latest 100 records are kept. The
default value for this configuration key is false.
* **record_server_delivery_enabled**: This key controls the delivery of
records to server; when enabled (default value), the record will be posted
to the address in the configuration file. If this configuration key is
disabled (false), records will not be spooled or posted to backend. This
configuration key can be used in combination with record_retention_enabled
to keep copies of telemetry records locally only.
.. note::
Configuration options may change as the telemetry client evolves.
Please use the comments in the file itself as the most accurate
reference for configuration.
Setting a static machine id
===========================
The machine id reported by the telemetry client is rotated every 3 days for
privacy reasons. If you wish to have a static machine id for testing
purposes, you can opt in by creating a static machine id file named
"opt-in-static-machine-id" under the directory :file:`/etc/telemetrics/`.
Where "unique machine ID" is your desired static machine ID.
.. code-block:: bash
sudo mkdir -p /etc/telemetrics
.. code-block:: bash
sudo echo "unique machine ID" > /etc/telemetrics/opt-in-static-machine-id
.. note::
The machine id mentioned here is not the same as the system hostname. Learn how to :ref:`hostname`:
Next steps
==========
* :ref:`telemctl`
source/clear-linux/guides/maintenance/telemetry-enable.rst → source/clear-linux/guides/telemetrics/telemetry-enable.rst
+114 -116
View File
@@ -1,116 +1,114 @@
.. _telemetry-enable:
Enable and disable telemetry in |CL-ATTR|
#########################################
|CL-ATTR| includes a telemetry solution as part of the OS that records events
of interest and reports them back to the development team via the telemetrics
client daemons, **telempobd** and **telempostmd**. This functionality is
maintained in the **telemetrics** software bundle.
.. note::
The telemetry functionality adheres to `Intel privacy policies`_
regarding the collection and use of :abbr:`PII (Personally Identifiable
Information)` and is open source. Specifically, no intentionally
identifiable information about the user or system owner is collected.
End users may enable or disable the telemetry component of |CL| or even
redirect where the records go if they wish to collect records for themselves.
Install the telemetry software bundle
*************************************
During the initial installation of |CL|, you are requested to join the
stability enhancement program and allow |CL| to collect anonymous reports
to improve system stability. If you choose not to join this program, then the
telemetry software bundle is not added to your system.
To install the telemetry bundle, enter the following command as either the
root user or with :command:`sudo` privileges:
.. code-block:: bash
sudo swupd bundle-add telemetrics
This adds the telemetrics-client to your system and you will automatically
opt-in for the service.
Enable telemetry
*****************
To start telemetry on your system, run the following command:
.. code-block:: bash
sudo telemctl start
This enables and starts the :command:`telemprobd` and :command:`telempostd` daemons and your system will
begin to send telemetry data to the server defined in the file
:file:`/etc/telemetrics/telemetrics.conf`. If this file does not exist, the
:command:`telemprobd` and :command:`telempostd` daemons will use the file
:file:`/usr/share/defaults/telemetrics/telemetrics.conf`.
Disable telemetry
*****************
To disable both of the telemetry daemons, run the following command:
.. code-block:: bash
sudo telemctl stop
Opt-out of telemetry
********************
To stop sending telemetrics data from your system, opt out of the
telemetry service:
.. code-block:: bash
sudo telemctl opt-out
This creates the file :file:`/etc/telemetrics/opt-out` and stops the
telemetry services.
Opt-in to telemetry
*******************
Conversely, to opt-in to the telemetry services, simply enter the opt-in
command and start the service:
.. code-block:: bash
sudo telemctl opt-in
This removes the file :file:`/etc/telemetrics/opt-out` file, if it exists,
and starts the telemetry services.
.. note::
To opt-in but not immediately start telemetry services, you will need to
run the command :command:`sudo telemctl stop` after the :command:`opt-in`
command is entered. Once you are ready to start the service, enter the
command :command:`sudo telemctl start`.
Remove the telemetry software bundle
************************************
To completely remove telemetrics from your system, use the command
:command:`swupd` to remove the telemetry software bundle:
.. code-block:: bash
sudo swupd bundle-remove telemetrics
Additional resources
********************
* `Telemetry feature description`_
* :ref:`Telemetry architecture<telemetry-about>`
* :ref:`telemetry-backend`
* https://github.com/clearlinux/telemetrics-client
.. _Intel privacy policies: https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html
.. _`Telemetry feature description`:
https://clearlinux.org/features/telemetry
.. _telemetry-enable:
Enable telemetry
################
Telemetry enables developers to observe and proactively address issues on
|CL-ATTR| before end users are impacted. The telemetry functionality
is maintained in the ``telemetrics`` software bundle.
.. note::
The telemetry functionality adheres to `Intel privacy policies`_
regarding the collection and use of :abbr:`PII (Personally Identifiable
Information)` and is open source. Specifically, no intentionally
identifiable information about the user or system owner is collected.
End users may enable or disable the telemetry component of |CL| or even
redirect where the records go if they wish to collect records for themselves.
Install the telemetry software bundle
*************************************
During the initial installation of |CL|, you are requested to join the
stability enhancement program and allow |CL| to collect anonymous reports
to improve system stability. If you choose not to join this program, then the
telemetry software bundle is not added to your system.
To install the telemetry bundle, enter the following command as either the
root user or with :command:`sudo` privileges:
.. code-block:: bash
sudo swupd bundle-add telemetrics
This adds the telemetrics-client to your system, and you will automatically
opt-in for the service.
Enable telemetry
================
To start telemetry on your system, run the following command:
.. code-block:: bash
sudo telemctl start
This enables and starts the :command:`telemprobd` and :command:`telempostd`
daemons. Your system will begin to send telemetry data to the server defined
in the file :file:`/etc/telemetrics/telemetrics.conf`. If this file does not
exist, the :command:`telemprobd` and :command:`telempostd` daemons will use
the file :file:`/usr/share/defaults/telemetrics/telemetrics.conf`.
Disable telemetry
=================
To disable both of the telemetry daemons, run the following command:
.. code-block:: bash
sudo telemctl stop
.. _incl-opt-in-out-telemetry:
Opt-in to telemetry
===================
To opt-in to the telemetry services, simply enter the opt-in
command and start the service:
.. code-block:: bash
sudo telemctl opt-in
This removes the file :file:`/etc/telemetrics/opt-out` file, if it exists,
and starts the telemetry services.
.. note::
To opt-in but not immediately start telemetry services, you will need to
run the command :command:`sudo telemctl stop` after the :command:`opt-in`
command is entered. Once you are ready to start the service, enter the
command :command:`sudo telemctl start`.
Opt-out of telemetry
====================
To stop sending telemetrics data from your system, opt out of the
telemetry service:
.. code-block:: bash
sudo telemctl opt-out
This creates the file :file:`/etc/telemetrics/opt-out` and stops the
telemetry services.
.. _incl-opt-in-out-telemetry-end:
Remove the telemetry software bundle
====================================
To completely remove telemetrics from your system, use the :command:`swupd`
command to remove the telemetry software bundle:
.. code-block:: bash
sudo swupd bundle-remove telemetrics
Next steps
==========
* :ref:`telemetry-config`
.. _Intel privacy policies: https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html
source/clear-linux/guides/telemetrics/telemetry-journal.rst
+108
View File
@@ -0,0 +1,108 @@
.. _telemetry-journal:
telemctl journal
################
The telemctl ``journal`` command gives you access to features and options of
the telemetry journal to assist with system analytics and debug. The
:command:`sudo telemctl journal` has a number of options to help filter
records. Use :command:`-h` or :command:`--help` to view usage options.
.. code-block:: bash
sudo telemctl journal -h
::
-r, --record_id Print record with specific record_id
-e, --event_id Print records with specific event_id
-c, --classification Print records with specific classification
-b, --boot_id Print records with specific boot_id
-i, --include_record Include record content
-V, --verbose Verbose output
-h, --help Display this help message
Journal Output
**************
To see the listing of records in the journal, run the command:
.. code-block:: bash
sudo telemctl journal -V
This will produce output like the following:
.. list-table:: **Table 1. Using -V for Verbose output**
:widths: 10 30 20 20 20
:header-rows: 1
* - Classification
- Time stamp
- Record ID
- Event ID
- Boot ID
* - org.clearlinux/heartbeat/ping
- Fri 2018-09-21 00:00:57 UTC
- 269e8e4026e6aa440c4d2ed71e38efcd
- b3a51b5e62a008ed0b56d1740be67d48
- 853a75aa-da3b-4356-a085-079abab3ffe1
* - org.clearlinux/hello/world
- Fri 2018-09-21 17:53:21 UTC
- b06c8d31adf5ccc7d5d3f8959d8d3e72
- 57c64c79a9b911d68f4dab10a00267d7
- 853a75aa-da3b-4356-a085-079abab3ffe1
* - org.clearlinux/crash/clr
- Fri 2018-09-21 17:57:59 UTC
- b62cd4278672ae3331cf121bc7a8e1c6
- b6adb5751382c48eebb7ee007fe1790a
- 853a75aa-da3b-4356-a085-079abab3ffe1
Each line gives information about a distinct record. The :command:`-V` or
:command:`--verbose` option adds the header to identify the Classification,
Time Stamp, Record ID, Event ID and Boot ID for each record. The journal
feature can filter records according to the Classification, Record ID, Event
ID and Boot ID by using the :command:`-c`,:command:`-r`, :command:`-e` and
:command:`-b` options accordingly.
Payload Information
********************
From the previous output, you may want to get more information about the
record with the "org.clearlinux/crash/clr" classification to help debug a
crash. You can use the :command:`-c` and :command:`-i` options to see the payload of the record, like this:
.. code-block:: bash
sudo telemctl journal -c org.clearlinux/crash/clr -i
.. code-block:: console
org.clearlinux/crash/clr Tue 2018-09-25 18:43:50 UTC 07ae583edbd13829965d67e9ba97d70c 69c600470769c841649266178375d67e d32c13d1-fda0-49c6-8431-e6c5b29cbefa
Process: /usr/bin/bash
PID: 685
Signal: 11
Backtrace (TID 685):
#0 kill() - [libc.so.6]
#1 bash_tilde_expand() - [/usr/bin/bash]
#2 maybe_execute_file() - [/usr/bin/bash]
#3 main() - [/usr/bin/bash]
#4 __libc_start_main() - [libc.so.6]
#5 _start() - [/usr/bin/bash]
If you have records of multiple crashes, you can use the :command:'-r'
option to specify the record more precisely, rather than going by
classification. You can also specify a classification of record and use the
:command:'-i' option to see the payload of each record with that
classification.
Next steps
==========
Adding telemetry to your applications:
* :ref:`telemetry-api`