USAGE.md
While `tlsfuzzer` is currently more of a framework than a full-featured tool,
it still is possible to run tests against common server configurations with a
little bit of effort.
Simple workflow
===============
Preparation
-----------
To run the scripts you will need 3 libraries:
* [six](https://github.com/benjaminp/six) ([PyPI](https://pypi.python.org/pypi/six))
* [ecdsa](https://github.com/warner/python-ecdsa) ([PyPI](https://pypi.python.org/pypi/ecdsa))
* [tlslite-ng](https://github.com/tlsfuzzer/tlslite-ng) ([PyPI](https://pypi.python.org/pypi/tlslite-ng))
It's common that `six` is already installed, or is available from the operating
system repository.
Alternatively it can be installed by running `pip install six` as root.
The other two can be installed (again, using `pip install ecdsa` and
`pip install tlslite-ng`) or they all can be downloaded to a single location to
minimise dependence on root privileges and permanent changes to the system. For
the rest of this tutorial, I'll follow the latter approach.
**Note:** the above libraries and `tlsfuzzer` support both Python 2 and Python
3, but they require at least Python 2.6. If you are running a modern
distribution (RHEL 6 or later), using the provided `python` in the below
commands is sufficient, if you're running older distribution, you will need to
install new python, and use it for the below commands, usually switching
`python` to `python26` (or similar).
In other words, if you have `six` already installed, the environment can be
prepared by running the following commands:
```
git clone https://github.com/tlsfuzzer/tlsfuzzer.git
cd tlsfuzzer
git clone https://github.com/warner/python-ecdsa .python-ecdsa
ln -s .python-ecdsa/src/ecdsa/ ecdsa
git clone https://github.com/tlsfuzzer/tlslite-ng .tlslite-ng
ln -s .tlslite-ng/tlslite/ tlslite
```
Running tests
=============
When all dependencies are downloaded or installed, place yourself in the root
directory of the project (one with `tlsfuzzer`, `tests` and `scripts`
directories) and you can start running tests.
All tests support a minimum set of parameters:
* `-h` to specify the hostname of the server under test (tests usually
default to `localhost`)
* `-p` to specify the port of the server under tests (tests default to 4433)
* `--help` to display all options supported by a given script
* names of the scenarios that are to be run (if not provided, all tests in a
script are run)
so to test if a server running on `example.com` on port 433 is not vulnerable
to the
[Bleichenbacher attack](http://archiv.infsec.ethz.ch/education/fs08/secsem/bleichenbacher98.pdf),
you need to run the following command:
```
PYTHONPATH=. python scripts/test-bleichenbacher-workaround.py -h example.com -p 443
```
A run of it will look something like this:
```
(beginning omitted)
zero byte in last byte of random padding ...
OK
zero byte in first byte of random padding ...
OK
Test end
successful: 8
failed: 0
```
That prints the names of tests being run (e.g. "zero byte in first byte of
random padding") and the overall test result, that 8 tests were run and the
server behaved as expected, and in 0 situations the test failed.
**Note**: unless stated otherwise in the help message of a specific test case,
the scripts usually expect a HTTP server with no client certificate
authentication.
In such case (where there were no errors), that usually ends the testing and
identifies the server as following the relevant RFC documents (like
[RFC 5246](https://tools.ietf.org/html/rfc5246)) or not vulnerable to the
vulnerability.
Investigating errors
====================
Theory of operation
-------------------
To be able to read the error messages of `tlsfuzzer` scripts, it's necessary to
know a little about how it works internally.
The simplest test script is the
[test-conversation.py](https://github.com/tlsfuzzer/tlsfuzzer/blob/master/scripts/test-conversation.py),
in it you will find a lot of boilerplate, and a single test scenario:
```
conversation = Connect(host, port)
node = conversation
ciphers = [CipherSuite.TLS_RSA_WITH_AES_128_CBC_SHA,
CipherSuite.TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
node = node.add_child(ClientHelloGenerator(ciphers))
node = node.add_child(ExpectServerHello())
node = node.add_child(ExpectCertificate())
node = node.add_child(ExpectServerHelloDone())
node = node.add_child(ClientKeyExchangeGenerator())
node = node.add_child(ChangeCipherSpecGenerator())
node = node.add_child(FinishedGenerator())
node = node.add_child(ExpectChangeCipherSpec())
node = node.add_child(ExpectFinished())
node = node.add_child(ApplicationDataGenerator(
bytearray(b"GET / HTTP/1.0\n\n")))
node = node.add_child(ExpectApplicationData())
node = node.add_child(AlertGenerator(AlertLevel.warning,
AlertDescription.close_notify))
node = node.add_child(ExpectAlert())
node.next_sibling = ExpectClose()
conversations["sanity"] = conversation
```
What this code does, is it sets up a list of things to do and things to expect
from the server. In this case, it performs a simple
[RSA key exchange](https://tools.ietf.org/html/rfc5246#page-36), an HTTP GET
request and expects data back.
Nodes with `Generator` in the name are creating messages and sending them to
the server, nodes that have `Expect` in the name will pause execution (with a
timeout) and wait for a message from the server.
In some cases, multiple behaviours are acceptable, this is implemented using
the "siblings". In the above example, after sending the `close_notify` Alert to
server, the test case expects the server to either send an alert of its own
(`ExpectAlert`), *or* to just close the connection (`ExpectClose`).
If a node has no children, an implicit (TCP) connection close is placed there.
Expect nodes not only verify that the messages received match the expected
type, but also that it is matching other messages (e.g. that the Server Hello
doesn't advertise support for extensions that the Client Hello didn't include).
In other words, unless there are specific options set, tlsfuzzer will behave as
a strict, well-behaved TLS client.
General note
------------
Scripts expect specific *behaviour* from a server, not specific error or
message.
In other words, a *passing* test is a test to which the behaviour of the server
matched the expectation (in some cases that may mean that server did report a
failure through Alert message). A *failing* test is a test in which the server
did not match expected behaviour - for example did not detect an error it
should have detected and continued handshake.
Reading error messages
----------------------
Typical error message looks like this:
```
zero byte in first byte of random padding ...
Error encountered while processing node <tlsfuzzer.expect.ExpectAlert object at 0x7f96e7e56a90> (child: <tlsfuzzer.expect.ExpectClose object at 0x7f96e7e56ad0>) with last message being: <tlslite.messages.Message object at 0x7f96e79f4090>
Error while processing
Traceback (most recent call last):
File "scripts/test-bleichenbacher-workaround.py", line 250, in main
runner.run()
File "/root/tlsfuzzer/tlsfuzzer/runner.py", line 178, in run
node.process(self.state, msg)
File "/root/tlsfuzzer/tlsfuzzer/expect.py", line 571, in process
raise AssertionError(problem_desc)
AssertionError: Expected alert description "bad_record_mac" does not match received "handshake_failure"
```
The first line, informs us which scenario was running when the error occurred,
in this case "zero byte in first byte of random padding".
Second line tells us which node was being processed, in this case it was
`ExpectAlert` and its child is `ExpectClose`.
Finally, the last line tells us what went wrong, in this case the error was
that the test expected an alert with `bad_record_mac` but got
`handshake_failure`.
### Common errors
#### Alert description mismatch
Pattern:
```
Error encountered while processing node <tlsfuzzer.expect.ExpectAlert ...
...
AssertionError: Expected alert description "bad_record_mac" does not match received "handshake_failure"
```
Situations where the received alert does not match the expected one may be
sign of a vulnerability (like in the example with Bleichenbacher test above),
bug in implementation under test or RFC non-compliance. In *rare* cases it may
be caused by test case being too strict (e.g. in some situations sending
`insufficient_security` alert when the test expects `handshake_failure` is
valid behaviour).
#### Unexpected message - Alert
Pattern:
```
`Error encountered while processing node <tlsfuzzer.expect.ExpectServerHello ...
...
AssertionError: Unexpected message from peer: Alert(fatal, handshake_failure)``
```
`handshake_failure` alert received in place of Server Hello usually means that
the server did not accept any cipher or any extension settings we sent to it.
This may happen when the server has only an ECDSA certificate (support for them
is not advertised in many scripts in in tlsfuzzer)
or did not enable ciphers which are necessary for the test being run.
Most tests require TLS_RSA_WITH_AES_128_CBC_SHA cipher to be enabled. In cases
where the test checks handling of messages not applicable in RSA key exchange,
the ciphers used are other variants of the AES ciphersuites. Inspect specific
script to know more or read its help message.
#### Unexpected message - Certificate Request
Pattern:
```
Error encountered while processing node <tlsfuzzer.expect.ExpectServerHelloDone
...
AssertionError: Unexpected message from peer: Handshake(certificate_request)
```
Situation like this means that the server asked the client for certificate (did
send Certificate Request message) but the client (tlsfuzzer) did not expect
that. In cases like this either the server should be reconfigured to not ask
for client authentication, or a different test script should be used.
#### Unexpected message - Application Data
Pattern:
```
Error encountered while processing node <tlsfuzzer.expect.ExpectAlert ...
...
AssertionError: Unexpected message from peer: ApplicationData(len=8000)
```
**Note**: for most tests it will be `ExpectAlert`, but in general, we're
looking for a node right *after* `ExpectApplicationData`.
Test cases in general expect just one TLS record as a response to the HTTP GET
query. That limits the response to 16384 bytes (16kiB).
Situations where it is OK to send more than one Application Data message:
* when the response is larger than 16KiB
* when the server is optimised for latency
([Time to first byte](https://en.wikipedia.org/wiki/Time_to_first_byte)) and
**all** messages but the last have the same size (e.g. 4KiB, 4KiB, 4KiB and
277B would be OK)
* when the negotiated protocol is TLS 1.0 and the negotiated cipher suite is
using CBC mode, then the first message can be 1 or 0 bytes long, and others
just as above (this is mitigation of the
[BEAST](https://en.wikipedia.org/wiki/Transport_Layer_Security#BEAST_attack))
* the server under test is an echo server, not HTTP, then it may send two
Application Data packets as it will receive two lines as the input
Unfortunately that complexity means that the analysis needs to be performed
manually, using a tool like [Wireshark](https://www.wireshark.org/).
Some situations where multiple Application Data messages being sent is not ok:
* splits happening on line end - that leaks the line lengths to a passive
observer
* splits happening on HTTP headers end - that leaks the size of headers to a
passive observer
* 1/n-1 split in TLS 1.1 or later or in stream ciphers - it's unnecessary and
wastes bandwidth
#### Required extension
Pattern:
```
Error encountered while processing node <tlsfuzzer.expect.ExpectServerHello ...
...
AssertionError: Required extension renegotiation_info missing
```
**Note**: the specific extension depends on test case.
Error of this kind usually means that the server does not support functionality
necessary for the test or, in case of `renegotiation_info` does not support a
feature that many other servers consider mandatory.
#### Unrecognized AlgorithmIdentifier
Pattern:
```
Error encountered while processing node <tlsfuzzer.expect.ExpectCertificate ...
...
File "/home/hkario/dev/tlsfuzzer/tlslite/messages.py", line 1128, in _parse_tls12
x509.parseBinary(certBytes)
File "/home/hkario/dev/tlsfuzzer/tlslite/x509.py", line 92, in parseBinary
raise SyntaxError("Unrecognized AlgorithmIdentifier")
SyntaxError: Unrecognized AlgorithmIdentifier
```
This is an indication that the server has sent a certificate with an DSA or
EdDSA key. They are currently unsupported in `tlsfuzzer` or `tlslite-ng`.
Reconfigure the server to use RSA or ECDSA certificates.
If a server already is configured with ECDSA or RSA certificates, it indicates
that the system for selecting correct certificate (or certificate chain) when
the client does not adverise support for given key type is not working
correctly.
#### Connection refused or timeout in Connect
Pattern:
```
Error encountered while processing node <tlsfuzzer.messages.Connect ...
...
sock.connect((self.hostname, self.port))
File "/usr/lib64/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
```
and
```
Error encountered while processing node <tlsfuzzer.messages.Connect...
...
File "/usr/lib64/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
timeout: timed out
```
The hostname or the port are incorrect for the system or a firewall on the way
blocks communication.
In other words: communication failed before TLS got involved.
#### Unexpected closure from peer
Pattern:
```
Error encountered while processing node <tlsfuzzer.expect.ExpectServerHello ...
...
AssertionError: Unexpected closure from peer
```
**Note**: it may happen at any node, though most commonly on
`ExpectServerHello` and `ExpectAlert`.
Some TLS implementations (or some combinations of TLS implementation and an
application) do not send alerts. This makes testing such implementations much
harder, and verifying that the exchanged messages do not cause unintended
behaviour in the server requires running the tests with valgrind, ubsan, asan
and extended logging on server side. This makes running the tests and verifying
results much harder and specific for that one implementation. Because
`tlsfuzzer` aims to be a universal test suite and RFC conformance checker,
test cases are not written in a way that allows the server to not send alert
messages.
Case in point: if `test-bleichenbacher-workaround.py` would be written in a way
that server can respond either with an alert or by closing the connection, a
vulnerable behaviour in which the server sometimes sends the correct alert and
sometimes closes the connection would *not* be detected, reporting a false
negative to the user.
Test specific notes
===================
`test-bleichenbacher-workaround.py`
-----------------------------------
This test requires a HTTP server with a RSA ciphersuite (`TLS_RSA_WITH_*_*`)
enabled and not asking for client certificates (only for GnuTLS it's not a
default configuration).
In case the server is well-behaved (responds with `handshake_failure` in case
it cannot negotiate the client proposed ciphersuite) and does not support any
RSA ciphersuite `tlslite-ng` supports, *only* the "sanity" tests will fail.
This **does not** mean that the implementation is not vulnerable, only that the
given configuration isn't (assuming that the server does not implement any
uncommon ciphers, like Camellia, Aria or others unsupported by `tlslite-ng`).
A good test requires a server configuration that enables all RSA ciphers that
a given implementation supports (or, if implementation does not allow for
enabling some groups of ciphers together, multiple runs that together had all
ciphers enabled).
The test is tuned for testing over a WAN link. If the server is on a local
network, it is possible to speed up test execution significantly by passing the
option `-t 0.01` (in general, that number should be twice as big as the RTT to
the server, in seconds). Setting it too low can cause the test case to report
false
negatives!
While the test allows for setting the expected alert response to a Finished
message sent after malformed Client Key Exchange (using the `-a` option) the
alert sent must be the same for all tests (that is, if one half of scenarios
pass with default configuration and other pass with `-a 0` option set, it makes
the server **vulnerable** to the Bleichenbacher attack). In general, the
workaround requires the server *not* to treat the Finished message specially,
so the alert sent *should* be the same as the one generated while running
[test-fuzzed-MAC.py](https://github.com/tlsfuzzer/tlsfuzzer/blob/master/scripts/test-fuzzed-MAC.py).
Also note that if setting this option is necessary, it shows that the server is
not RFC compliant, which in turn, as you can see, makes testing it harder and
more complex.
While not testing for Bleichenbacher directly, the tests
[test-invalid-rsa-key-exchange-messages.py](https://github.com/tlsfuzzer/tlsfuzzer/blob/master/scripts/test-invalid-rsa-key-exchange-messages.py)
and [test-truncating-of-kRSA-client-key-exchange.py](https://github.com/tlsfuzzer/tlsfuzzer/blob/master/scripts/test-truncating-of-kRSA-client-key-exchange.py)
perform checks related to RSA key exchange. Failures there may be a sign of
other problems.
`test-record-size-limit.py`
---------------------------
This test is very stict on the size of returned reply and expects the reply
to be sent in a single record, if its size allows for that.
To quickly check what is the size that the server sends, it's possible to run
the test with this options:
```
test-record-size-limit.py --reply-AD-size 1 \
'check if server accepts maximum size in TLS 1.2'
```
or:
```
test-record-size-limit.py --reply-AD-size 1 \
'check if server accepts maximum size in TLS 1.3'
```
It should fail with
`AssertionError: ApplicationData of unexpected size: X, expected: 1`
where X is the value that needs to be passed as argument to `--reply-AD-size`.
To make the test case more deterministic, it is possible to specify the
exact (HTTP) request being sent to server using the `--request` option. For
example, to perform a GET request for a specific file, not the `/` object.
`--cookie` option must be used if the server sends `cookie` extension in
HelloRetryRequest message.
`--supported-groups` must to be used when the server sends `supported_groups`
in EncryptedExtension during a normal TLS 1.3 handshakes while
`--hrr-supported-groups` must be used when the server sends that extension
in handshakes that force the server to send HelloRetryRequest message.
In case the server limits the size it advertises in its extension,
then `--expect-size` can be used to set the expected limit. Note though,
the value expected from server in TLS 1.3 will be one byte larger than that for
TLS 1.2 and earlier protocols as this is the expected behaviour of servers that
support the biggest possible records. It also makes the size of actual
application data payload the same irrespective of negotiated protocol version.
`test-tls13-certificate-verify.py`
----------------------------------
This test verifies server's support for different Signature Algorithms for
client certificates and CertificateVerify messages. It tests all the
algorithms supported by tlsfuzzer, and verifies that only the advertised ones
are accepted by the server. It also verifies that algorithms incompatible with
the certificate type provided (e.g rsa_pss_rsae_sha256 with "rsa-pss"
certificates) are refused as well.
It also fuzzes signatures in various ways to make sure servers behave according
to TLS1.3 specifications.
NOTE: To test all algorithms it is necessary to run this test twice, once with
an "rsa" certificate and once with an "rsa-pss" certificate.
The list of server supported Signature Algorithm's must be provided via the -s
command line option, the default is set to match tlslite-ng sigalg selection.
The algorithms can be defined both via shorthand strings, type+hash strings,
type and hash can also be expressed via numeric ids.
Example: `-s "sha256+rsa rsa_pss_pss_sha384 8+11"`
For some correctness tests, only one among multiple algorithms of the same
type will be used. For example there is a test that checks that a "correct"
signature using one hash but being advertised as using different hash is
refused. The test machinery will select a signature algorithm matching the
first hash in an ordered list and use the second hash in the actual signature.
The ordered list of hashes to select from can be changed using the -o
parameter. The default is "sha256 sha384 sha512", in that order.
The test that uses the wrong mgf1 hash, assuming a server that accepts all
standard rsa algorithms, will choose an algorithm that uses the sha256 (the
first in the list) for the envelope, and actually uses sha384 (the second in
the list) for the signature operation.
To test different combinations one can simply change the order to something
like -o "sha512 sha256 sha384"; this will cause the test to send an envelope
for rsa_pss_pss_sha512 (for "rsa-pss" certificates), but will use sha256 as
the actual mgf1 hash when generating the signature.
Other tests also uses the ordered hash list to choose an algorithm among
multiple viable ones advertised by the server and can similarly be affected by
changing the ordered list of hashes.
At least two hashes must be provided in the list, although only one need
to be supported by the server. If a hash is not supported by the server it will
be skipped in test that select algorithms that need to be supported by the
server, skipped hashes may still be selected to generate signatures or invalid
envelope values as needed by the test.