In this document, we introduce different aspects of the implementation of our technique. Moreover, we explain how to re-run our experiments and access the raw data for our reported results. To facilitate all this, we provided the implementation of our technique on a virtual machine (artifact-os). The virtual machine can be run using Oracle VirtualBox and the credentials for it are as follows:
Username: artifact-admin
password: Aartifact
The virtual machine (artifact-os) contains:
The modified version of KLEE symbolic execution engine augmented with:
SUT-Specific test harnesses
The raw results for executed experiments and reported bugs
Scripts for compiling the tested SUTs and executing different experiments
In the following, we present details on each of these parts and show how to run experiments:
Our generic infrastructure is, in essence, a custom implementation of POSIX sockets’ API. The custom implementation of this API can be found in:
/home/artifact-admin/kleener/runtime/POSIX/socket_models.c
Our protocol-specific infrastructure contains the following components:
Different components related to the DTLS protocol can be found as mentioned in the following:
Parser & Serializer
/home/artifact-admin/kleener/runtime/Intrinsic/DTLS/records.c
Monitors
/home/artifact-admin/kleener/runtime/Intrinsic/DTLS/MONITORS/
For example, the monitor for checking the correctness of the content type as shown in the paper, can be found in:
/home/artifact-admin/kleener/runtime/Intrinsic/DTLS/MONITORS/content-type.c
Different components related to the QUIC protocol can be found as mentioned in the following:
Parser & Serializer
/home/artifact-admin/kleener/runtime/Intrinsic/QUIC/packets.c
Monitors
/home/artifact-admin/kleener/runtime/Intrinsic/QUIC/MONITORS/
For each SUT, a test harness which orchestrates a protocol interaction for testing is implemented. The harness configures and instantiates protocol entities, instructs them to perform the desired interaction (e.g., a handshake), and exits once the interaction has finished, one of the entities has aborted it, or when a monitor triggers an assertion failure.
For the DTLS protocol, all the test harnesses are located in:
/home/artifact-admin/Desktop/dtls-symbolic-execution-harness/
The corresponding DTLS libraries for these harnesses are located in:
/home/artifact-admin/Desktop/dtls-symbolic-execution-harness/repos/
Libraries and harnesses whose name ends with icst correspond to old library versions that were also tested by Asadian et al. [3].
The table below shows the correspondence between the harness and the DTLS library:
| The Library Name | The Test Harness Directory | The Corresponding Library |
|---|---|---|
| OpenSSL 3.0.2 | openssl-3.0.2-harness |
openssl-3.0.2-model |
| MbedTLS 3.1.0 | mbedtls-3.1.0-harness |
mbedtls-3.1.0-model |
| TinyDTLS (e4d547ce) | etinydtls-harness-develop |
etinydtls-standalone-develop |
| WolfSSL 5.3.0 | wolfssl-harness-5.3.0 |
wolfssl-standalone-5.3.0 |
| TinyDTLS (7068882) | etinydtls-harness-icst |
etinydtls-standalone-icst |
| TinyDTLS (94205ff) | etinydtls-harness-fragmentation-icst |
etinydtls-standalone-fragmentation-icst |
| OpenSSL 3.0.0-alpha12 | openssl-3-alpha-harness-icst |
openssl-3-alpha-icst |
For the QUIC protocol, all the test harnesses are located in:
/home/artifact-admin/Desktop/quic-symbolic-execution-harness/
The corresponding QUIC libraries for these harnesses are located in:
/home/artifact-admin/Desktop/quic-symbolic-execution-harness/repos/
This folder also contains fthe TLS libraries depended on by QUIC libraries (picotls, boringssl, openssl-1.1.1k).
Each QUIC library folder includes a file modifications, which contains all the modifications that were applied to the library.
The following table shows all the correspondence between the harness and the QUIC library:
| The Library Name | The Test Harness Directory | The Corresponding Library |
|---|---|---|
| LiteSpeed QUIC 3.2.0 | lsquic-harness |
lsquic-3.2.0 |
| quicly (482610bf) | quicly-harness |
quicly |
| picoquic (7f2fbdfb) | picoquic-harness |
picoquic |
For every tested library, the raw data for all the executed experiments can be found in the following directories:
The Library Name | The Path To Raw Data For Experiments
-------------------------- | -----------------------------------------
OpenSSL 3.0.2 | /home/artifact-admin/Desktop/dtls-symbolic-execution-harness/openssl-3.0.2-harness/experiments
MbedTLS 3.1.0 | /home/artifact-admin/Desktop/dtls-symbolic-execution-harness/mbedtls-3.1.0-harness/experiments
TinyDTLS (e4d547ce) | /home/artifact-admin/Desktop/dtls-symbolic-execution-harness/etinydtls-harness-develop/experiments
WolfSSL 5.3.0 | /home/artifact-admin/Desktop/dtls-symbolic-execution-harness/wolfssl-harness-5.3.0/experiments
OpenSSL 3.0.0-alpha12 | /home/artifact-admin/Desktop/dtls-symbolic-execution-harness/openssl-3-alpha-harness-icst/experiments
TinyDTLS^E (commit:7068882) | /home/artifact-admin/Desktop/dtls-symbolic-execution-harness/etinydtls-harness-icst/experiments
TinyDTLS^E (commit:94205ff) | /home/artifact-admin/Desktop/dtls-symbolic-execution-harness/etinydtls-harness-fragmentation-icst/experiments
LiteSpeed QUIC 3.2.0 | /home/artifact-admin/Desktop/quic-symbolic-execution-harness/lsquic-harness/experiments
quicly (482610bf) | /home/artifact-admin/Desktop/quic-symbolic-execution-harness/quicly-harness/experiments
picoquic (7f2fbdfb) | /home/artifact-admin/Desktop/quic-symbolic-execution-harness/picoquic-harness/experiments
For the sake of space we removed from experiment folders ancillary files KLEE generates e.g., to display human-readable output or produce statistics. By this, we saved over 20 GB of space.
For all the tested libraries, the test harness is already compiled against the library using wllvm, and is prepared to be symbolically executed with our tool. To facilitate running experiments, we provide a run.sh script that is located in the test harness directory. The script gets the name of the requirement to be checked as well as the side (client or server) on which the requirement should be checked. What follows is an example on how to execute the experiment that checks the validity of the content type (as mentioned in the paper) on TinyDTLS (e4d547ce).
1- By going the test harness directory, we have access to the run.sh script;
cd /home/artifact-admin/Desktop/dtls-symbolic-execution-harness/etinydtls-harness-develop
2- Once in the directory, we can execute the experiment of our choice using the following syntax:
./run.sh -e <experiment-name> -r <side-to-be-checked>
client or server)In this example, we use the following command to execute the experiment for checking the validity of the content type on the server side:
./run.sh -e content-type -r server
Executing the command starts the symbolic execution for checking the requirement using the respective monitor.
3- When the symbolic execution is finished, the engine reports the number of instructions executed during the experiment, the number of completed paths and the number of generated test cases. For our example, we get the following output for the above-mentioned experiment:
KLEE: done: total instructions = 1539245
KLEE: done: completed paths = 16
KLEE: done: generated tests = 8
One can find the generated test cases in a directory with the name of the requirement concatenated with the side that the requirements is checked on and an index (in this case, content-type-server-0).
/Desktop/dtls-symbolic-execution-harness/etinydtls-harness-develop/content-type-server-0
One can inspect the generated test cases using a utility called ktest-tool. Let us give an example for the content type experiment. To inspect the test case test000003.ktest, we use the following command in the content-type-server-0 directory :
ktest-tool test000003.ktest
The result contains the concrete value for the content type field (in this case the concrete value is 21):
...
object 2: name: 'content-type'
object 2: size: 1
object 2: data: b'\x15'
object 2: hex : 0x15
object 2: int : 21
object 2: uint: 21
object 2: text: .
Running experiments for both DTLS and QUIC harnesses follows a similar procedure as the one mentioned above. The table below the contains the name of the experiments for DTLS and QUIC.
| DTLS Experiment Name | Side To be Checked |
|---|---|
content-type |
client / server |
record-version |
client / server |
record-length |
client / server |
epoch |
client / server |
record-sequence |
client / server |
handshake-type |
client / server |
handshake-length |
client / server |
message-sequence |
client / server |
fragmentation-reassembly |
client / server |
unfragmented-message-offset |
client / server |
unfragmented-message-length |
client / server |
fragmentation-message-sequence |
client / server |
fragmentation-message-length |
client / server |
handshake-version |
client / server |
message-sequence-ch0hvr |
server |
message-sequence-ch2sh |
server |
signature-hash-alg |
client |
signature-hash-length |
client |
certificate-type-count |
client |
fragment-length |
client / server |
cookie-length |
client / server |
sessionid-length |
client / server |
extensions-length |
client / server |
| QUIC Experiment Name | Side To be Checked |
|---|---|
initial-token-length |
client |
version-negotiation |
server |
reserved-bits |
client / server |
fixed-bit |
client / server |
destid-length |
client / server |
srcid-length |
client / server |
inconsistent-versions |
server |
frame-type |
client / server |
ack-ranges |
client / server |
empty-token |
client |
client-generated-token |
server |
new-connection-id-length |
client / server |
retire-prior-to |
client / server |
handshake-done-receipt |
server |
new-token-length |
client |
packet-length |
client / server |
For DTLS harnesses, the script ./run.sh supports an additional option, enabled by -c, for specifying the ciphersuite to use in the handshake. Choices are psk, for a PSK-based ciphersuite (default), and ecc, for a ECC-based ciphersuite. A handshake using ECC-based ciphersuite involves more messages and exercises much more functionality, taking longer to execute. The client requirements certificate-type-count, certificate-type-count, signature-hash-alg and signature-hash-length only apply to ECC-based handshakes. Hence, to check certificate-type-count on TinyDTLS, we run:
./run.sh -e certificate-type -r server -c ecc
The resulting folder (e.g., certificate-type-count-client-0) will contain .err files corresponding to the bugs reported for TinyDTLS in the first row of Table 1. We used the ecc setting when testing TinyDTLS, and the (default) psk setting when testing all other DTLS SUTs.
In this final section, we mention how one can reproduce the bugs that are presented in Table 1 in the paper:
| SUT | Short Description of the Bug | How to Reproduce |
|---|---|---|
| TinyDTLS (e4d547ce) | Out-of-bound pointers when a client receives a CertificateRequest with an invalid value for the certificate_types_count field. | ./run.sh -e certificate-type-count -r CLIENT -c ecc |
| TinyDTLS (e4d547ce) | Out-of-bounds pointer when a client receives a CertificateRequest with an invalid value for the signature_hash_algorithms_length field. | ./run.sh -e signature-hash-length -r CLIENT -c ecc |
| TinyDTLS (e4d547ce) | Out-of-bounds pointers caused by a handshake message other than ClientHello whose msg_type field has been set to ClientHello. | ./run.sh -e handshake-type -r CLIENT and ./run.sh -e handshake-type -r SERVER |
| TinyDTLS (e4d547ce) | Assertion failure when a client expects a ServerHello message but instead receives a ServerKeyExchange. | ./run.sh -e handshake-type -r CLIENT |
| TinyDTLS (e4d547ce) | Out-of-bounds pointers caused by a ChangeCipherSpec message whose content_type field has been set to Alert | ./run.sh -e content-type -r CLIENT / ./run.sh -e content-type -r SERVER |
| WolfSSL 5.3.0 | Four (4) non-conformance bugs due to insufficient checking of length-related fields. | ./run.sh -e unfragmented-message-offset -r SERVER / ./run.sh -e handshake-length -r SERVER / ./run.sh -e fragment-length -r CLIENT / ./run.sh -e extensions-length -r SERVER |
| WolfSSL 5.3.0 | Non-conformance bug when the message sequence numbers (SNs) in the initial ClientHello and HelloVerifyRequest differ. | ./run.sh -e message-sequence-ch0hvr -r SERVER |
| WolfSSL 5.3.0 | Non-conformance bug when the message sequence numbers (SNs) in the second ClientHello and ServerHello differ | ./run.sh -e message-sequence-ch2sh -r SERVER |
| WolfSSL 5.3.0 | Non-conformance bug when a server proceeds with the interaction despite receiving a ClientHello message with a non-existing version. | ./run.sh -e handshake-version -r SERVER |
| WolfSSL 5.3.0 | Non-conformance bug when a client fails to update the highest validated record SN; thus, accepting messages with outdated SNs. | ./run.sh -e record-sequence -r CLIENT |
| LiteSpeed QUIC 3.2.0 | Non-conformance bug when a client (or server) fails to discard a packet with the fixed bit assigned to zero. | ./run.sh -e fixed-bit -r SERVER |
| LiteSpeed QUIC 3.2.0 | Three (3) non-conformance bugs occur where the incorrect error code is returned by a client (or server) in the presence of invalid input. | ./run.sh -e frame-type -r CLIENT / ./run.sh -e frame-type -r SERVER / ./run.sh -e client-generated-token -r SERVER / ./run.sh -e handshake-done-receipt -r SERVER |
| quicly (482610bf) | Non-conformance bug when a server fails to discard a NEW_TOKEN frame generated by a client and proceeds with the interaction. | ./run.sh -e client-generated-token -r SERVER |
| picoquic (7f2fbdfb) | Non-conformance bug when a server fails to discard a NEW_TOKEN frame generated by a client and proceeds with the interaction | ./run.sh -e client-generated-token -r SERVER |
| picoquic (7f2fbdfb) | Three (3) non-conformance bugs occur where the incorrect error code is returned by a client (or server) in the presence of invalid input. | ./run.sh -e frame-type -r CLIENT / ./run.sh -e frame-type -r SERVER / ./run.sh -e retire-prior-to -r CLIENT / ./run.sh -e new-connection-id-length -r CLIENT |