avocado-framework/avocado

.. _lts_52_0:

========
52.0 LTS
========

The Avocado team is proud to present another release: Avocado version
52.0, the second Avocado LTS version.

What's new?
===========

When compared to the last LTS (v36), the main changes introduced by this
versions are:

- Support for TAP[2] version 12 results, which are generated by default
  in test results directory (``results.tap`` file).
- The download of assets in tests now allow for an expiration time.
- Environment variables can be propagated into tests running on remote
  systems.
- The plugin interfaces have been moved into the
  ``avocado.core.plugin_interfaces`` module.
- Support for running tests in a Docker container.
- Introduction of the "Fail Fast" feature (``--failfast on`` option) to
  the ``run`` command, which interrupts the Job on a first test failure.
- Special keyword ``latest`` for replaying previous jobs.
- Support to replay a Job by path (in addition to the Job ID method and
  the ``latest`` keyword).
- Diff-like categorized report of jobs
  (``avocado diff <JOB_1> <JOB_2>``).
- The introduction of a ``rr`` based wrapper.
- The automatic VM IP detection that kicks in when one uses
  ``--vm-domain`` without a matching ``--vm-hostname``, now uses a more
  reliable method (libvirt/qemu-gust-agent query).
- Set ``LC_ALL=C`` by default on sysinfo collection to simplify avocado
  diff comparison between different machines.
- Result plugins system is now pluggable and the results plugins (JSON,
  XUnit, HTML) were turned into stevedore plugins. They are now listed
  in the ``avocado plugins`` command.
- Multiplexer was replaced with Varianter plugging system with defined
  API to register plugins that generate test variants.
- Old ``--multiplex`` argument, which used to turn yaml files into
  variants, is now handled by an optional plugin called ``yaml_to_mux``
  and the ``--multiplex`` option is being deprecated in favour of
  the ``--mux-yaml`` option, which behaves the same way.
- It’s now possible to disable plugins by using the configuration file.
- Better error handling of the virtual machine plugin (``--vm-domain``
  and related options).
- When discovering tests on a directory, the result now is a properly
  alphabetically ordered list of tests.
- Plugins can now be setup in Avocado configuration file to run at a
  specific order.
- Support for filtering tests by user supplied "tags".
- Users can now see the test tags when listing tests with the ``-V``
  (verbose) option.
- Users can now choose to keep the complete set of files, including
  temporary ones, created during an Avocado job run by using the
  ``--keep-tmp`` option (e.g. to keep those files for ``rr``).
- Tests running with the external runner (``--external-runner``) feature
  will now have access to the extended behavior for SIMPLE tests, such
  as being able to exit a test with the WARNING status.
- Encoding support was improved and now Avocado should safely treat
  localized test-names.
- Test writers now have access to a test temporary directory that will
  last not only for the duration of the test, but for the duration of
  the whole job execution to allow sharing state/exchanging data between
  tests. The path for that directory is available via Test API
  (``self.teststmpdir``) and via environment variable
  (``AVOCADO_TESTS_COMMON_TMPDIR``).
- Avocado is now available on Fedora standard repository. The package
  name is ``python2-avocado``. The optional plugins and examples
  packages are also available. Run ``dnf search avocado`` to list them
  all.
- Optional plugins and examples packages are also available on PyPI
  under ``avocado-framework`` name.
- Avocado test writers can now use a family of decorators, namely
  ``avocado.skip()``, ``avocado.skipIf()`` and ``avocado.skipUnless()``
  to skip the execution of tests.
- Sysinfo collection based on command execution now allows a timeout to
  be set in the Avocado configuration file.
- The non-local runner plugins, the html plugin and the yaml_to_mux
  plugin are now distributed in separate packages.
- The Avocado main process will now try to kill all test processes
  before terminating itself when it receives a SIGTERM.
- Support for new type of test status, ``CANCEL``, and of course the
  mechanisms to set a test with this status (e.g. via ``self.cancel()``).
- ``avocado.TestFail``, ``avocado.TestError`` and ``avocado.TestCancel``
  are now public Avocado Test APIs, available in the main `avocado`
  namespace.
- Introduction of the robot plugin, which allows Robot Framework tests
  to be listed and executed natively within Avocado.
- A brand new ResultsDB optional plugin.
- Listing of supported loaders (``--loaders \?``) was refined.
- Variant-IDs generated by yaml_to_mux plugin now include leaf node
  names to make them more meaningful, making easier to skim through
  the results.
- `yaml_to_mux` now supports internal filters defined inside the YAML
  file expanding the filtering capabilities even further.
- Avocado now supports resuming jobs that were interrupted.
- The HTML report now presents the test ID and variant ID in separate
  columns, allowing users to also sort and filter results based on those
  specific fields.
- The HTML report will now show the test parameters used in a test when
  the user hovers the cursor over the test name.
- Avocado now reports the total job execution time on the UI, instead of
  just the tests execution time.
- New ``avocado variants`` has been added which supersedes the
  ``avocado multiplex``.
- Loaders were tweaked to provide more info on ``avocado list -V``
  especially when they don't recognize the reference.
- Users can use ``--ignore-missing-references on`` to run a job with
  undiscovered test references
- Users can now choose in which order the job will execute tests (from
  its suite) and variants. The two available options are
  ``--execution-order=variants-per-test`` (default) or
  ``--execution-order=tests-per-variant``.
- Test methods can be recursively discovered from parent classes by upon
  the ``:avocado: recursive`` docstring directive.

Besides the list above, we had several improvements in our ``utils``
libraries that are important for test writers, some of them are listed
below:

- ``time_to_seconds``, ``geometric_mean`` and ``compare_matrices`` were
  added in ``avocado.utils.data_structures``.
- ``avocado.utils.distro`` was refined.
- Many ``avocado.utils`` new modules were introduced, like ``filelock``,
  ``lv_utils``, ``multipath``, ``partition`` and ``pci``.
- ``avocado.utils.memory`` contains several new methods.
- New ``avocado.utils.process.SubProcess.get_pid()`` method.
- ``sudo`` support in ``avocado.utils.process`` was improved
- The ``avocado.utils.process`` library makes it possible to ignore
  spawned background processes.
- New ``avocado.utils.linux_modules.check_kernel_config()``.
- Users of the ``avocado.utils.process`` module will now be able to access
  the process ID in the ``avocado.utils.process.CmdResult``.
- Improved ``avocado.utils.iso9660`` with a more complete standard API
  across all back-end implementations.
- Improved ``avocado.utils.build.make()``, which will now return the make
  process exit status code.
- The ``avocado.Test`` class now better exports (and protects) the core
  class attributes members (such as ``params`` and ``runner_queue``).
- ``avocado.utils.linux_modules`` functions now returns module name,
  size, submodules if present, filename, version, number of modules
  using it, list of modules it is dependent on and finally a list of
  ``params``.

It is also worth mentioning:

- Improved documentation, with new sections to Release Notes and
  Optional Plugins, very improved Contribution and Community Guide. New
  content and new examples everywhere.
- The avocado-framework-tests GitHub organization was founded to
  encourage companies to share Avocado tests.
- Bugs were always handled as high priority and every single version was
  delivered with all the reported bugs properly fixed.

When compared to the last LTS, we had:

- 1187 commits (and counting).
- 15 new versions.
- 4811 more lines of Python code (+27,42%).
- 1800 more lines of code comment (+24,67%).
- 31 more Python files (+16,48%).
- 69 closed GitHub issues.
- 34 contributors from at least 12 different companies, 26 of them
  contributing for the fist time to the project.


Switching from 36.4 to 52.0
---------------------------

You already know what new features you might expect, but let's emphasize the
main changes required to your workflows/tests when switching from 36.4 to
52.0

Installation
^^^^^^^^^^^^

All the previously supported ways to install Avocado are still valid and
few new ones were added, but beware that Avocado was split into several
optional plugins so you might want to adjust your scripts/workflows.

* Multiplexer (the YAML parser which used to generate variants) was
  turned into an optional plugin ``yaml_to_mux`` also known as
  ``avocado_framework_plugin_varianter_yaml_to_mux``. Without it Avocado
  does not require PyYAML, but you need it to support the parsing of
  YAML files to variants (unless you use a different plugin with similar
  functionality, which is now also possible).
* The HTML result plugin is now also an optional plugin so one has to
  install it separately.
* The remote execution features (``--remote-hostname``, ``--vm-domain``,
  ``--docker``) were also turned into optional plugins so if you need
  those you need to install them separately.
* Support for virtual environment (``venv``) was greatly improved and we
  do encourage people who want to use ``pip`` to do that via this method.

As for the available ways:

* Fedora/RHEL can use our custom repositories, either LTS-only or all
  releases. Note that latest versions (non-lts) are also available
  directly in Fedora and also in EPEL.
* OpenSUSE - Ships the 36 LTS versions, hopefully they'll start
  shipping the 52 ones as well (but we are not in charge of that
  process)
* Debian - The ``contrib/packages/debian`` script is still available,
  although un-maintained for a long time
* PyPI/pip - Avocado as well as all optional plugins are available in
  PyPI and can be installed via ``pip install avocado-framework\*``,
  or selectively one by one.
* From source - Makefile target ``install`` is still available but it
  does not install the optional plugins. You have to install them one
  by one by going to their directory (eg. ``cd optional_plugins/html``
  and running ``sudo python setup.py install``)

As before you can find the details in :ref:`Installing Avocado <installing>`.

Usage
^^^^^

.. note:: As mentioned in previous section some previously core
   features were turned into optional plugins. Do check your install
   script if some command described here are missing on your system.

Most workflows should work the same, although there are few little
changes and a few obsoleted constructs which are still valid, but you
should start using the new ones.

The hard changes which does not provide backward compatibility:

* Human result was tweaked a bit:
    * The ``TESTS`` entry (displaying number of tests) was removed as
      one can easily get this information from ``RESULTS``.
    * Instead of tests time (sum of test times) you get job time
      (duration of the job execution) in the human result
* Json results also contain some changes:
    * They are pretty-printed
    * As ``cancel`` status was introduced, json result contain an entry
      of number of canceled tests (``cancel``)
    * url was renamed to id (url entry is to be removed in ``53.0`` so
      this is actually a soft change with a backward compatibility support)
* The ``avocado multilex|variants`` does not expect multiplex YAML
  files as positional arguments, one has to use ``-m|--mux-yaml``
  followed by one or more paths.
* Test variants are not serialized numbers anymore in the default
  ``yaml_to_mux`` (``multiplexer``), but ordered list of leaf-node names
  of the variant followed by hash of the variant content
  (paths+environment). Therefore instead of ``my_test:1`` you can get
  something like ``my_test:arm64-virtio_scsi-RHEL7-4a3c``.
* ``results.tap`` is now generated by default in job results along the
  ``results.json`` and ``results.xml`` (unless disabled)
* The ``avocado run --replay`` and ``avocado diff`` are unable to parse
  results generated by ``36.4`` to this date. We should be able to
  introduce such feature with not insignificant effort, but no one was
  interested yet.

And the still working but to be removed in ``53.0`` constructs:

* The long version of the ``-m|--multiplex`` argument available in
  ``avocado run|multiplex|variants`` was renamed to ``-m|--mux-yaml``
  which corresponds better to the rest of ``--mux-*`` arguments.
* The ``avocado multiplex`` was renamed to ``avocado variants``
* The ``avocado multiplex|variants`` arguments were reworked to better
  suite the possible multiple varianter plugins:

    * Instead of picking between ``tree`` representation of list of
      variants one can use ``--summary``, resp ``--variants`` followed
      by verbosity, which supersedes ``-c|contents``, ``-t|--tree``,
      ``-i|--inherit``
    * Instead of ``--filter-only|--filter-out`` the
      ``--mux-filter-only|--mux-filter-out`` are available
    * The ``--mux-path`` is now also available in
      ``avocado multiplex|variants``

Test API
^^^^^^^^

Main features stayed the same, there are few new ones so do check our
documentation for details. Anyway while porting tests you should pay
attention to following changes:

* If you were overriding ``avocado.Test`` attributes (eg. ``name``,
  ``params``, ``runner_queue``, ...) you'll get an
  ``AttributeError: can't set attribute`` error as most of them were
  turned into properties to avoid accidental override of the important
  attributes.
* The ``tearDown`` method is now executed almost always (always when the
  ``setUp`` is entered), including when the test is interrupted while
  running ``setUp``. This might require some changes to your ``setUp``
  and ``tearDown`` methods but generally it should make them simpler.
  (See :ref:`Setup and cleanup methods` and following chapters for details)
* Test exceptions are publicly available directly in ``avocado``
  (``TestError``, ``TestFail``, ``TestCancel``) and when raised inside
  test they behave the same way as ``self.error``, ``self.fail`` or
  ``self.cancel``. (See :mod:`avocado`)
* New status is available called ``CANCEL``. It means the test (or even
  just ``setUp``) started but the test does not match prerequisites. It's
  similar to ``SKIP`` in other frameworks, but the ``SKIP`` result is
  reserved for tests that were not executed (nor the ``setUp`` was
  entered). The ``CANCEL`` status can be signaled by ``self.cancel`` or
  by raising ``avocado.TestCancel`` exception and the ``SKIP`` should be
  set only by ``avocado.skip``, ``avocado.skipIf`` or
  ``avocado.skipUnless`` decorators. The ``self.skip`` method is still
  supported but will be removed after in ``53.0`` so you should replace
  it by ``self.cancel`` which has similar meaning but it additionally
  executes the ``tearDown``. (See :ref:`Test statuses`
* The ``tag`` argument of ``avocado.Test`` was removed as it is part of
  ``name``, which can only be ``avocado.core.test.TestName`` instance.
  (See :meth:`avocado.core.test.Test`)
* The ``self.job.logdir`` which used to be abused to share state/data
  between tests inside one job can now be dropped towards the
  ``self.teststmpdir``, which is a shared temporary directory which
  sustains throughout job execution and even between job executions if
  set via ``AVOCADO_TESTS_COMMON_TMPDIR`` environmental value. (See
  :meth:`avocado.core.test.Test.teststmpdir`)
* Those who write inherited test classes will be pleasantly surprised as
  it is now possible to mark a class as avocado test including all
  ``test*`` methods coming from all parent classes (similarly to how
  dynamic discovery works inside Python unittest, see
  `docstring-directive-recursive` for details)
* The ``self.text_output`` is not published after the test execution. If
  you were using it simply open the ``self.logfile`` and read the content
  yourself.

Utils API
^^^^^^^^^

Focusing only on the changes you might need to adjust the usage of:

* ``avocado.utils.build.make`` calls as it now reports only
  ``exit_status``. To get the full result object you need to execute
  ``avocado.utils.build.run_make``.
* ``avocado.utils.distro`` reports ``Red Hat Enterprise Linux``/``rhel``
  instead of ``Red Hat``/``redhat``.
* ``avocado.process`` where the check for availability of ``sudo`` was
  improved, which might actually start executing some code which used
  to fail in 36.4.

Also check out the :mod:`avocado.utils` for complete list of available
utils as there were many additions between 36.4 and 52.0.

Complete list of changes
^^^^^^^^^^^^^^^^^^^^^^^^

For a complete list of changes between the last LTS release (36.4) and
this release, please check out `the Avocado commit changelog
<https://github.com/avocado-framework/avocado/compare/36.4...52.0>`_.