# Manual for Running our Experiments

In this README, we explain how to use our code to reproduce the results of our experiments.

## Prerequisites

To simplify setup of the experiments, we also provide a Dockerfile.

### Setup with Docker

Simply execute

```
docker build -f ./Dockerfile -t consequence-experiments:1.0.0 . && docker tag consequence-experiments:1.0.0 consequence-experiments:latest
```

to create a Dockerimage with all necessary setup for the experiments.

### Manual Setup without Docker

We strongly recommend using Docker to run the experiments. If you want to setup the environment yourself for any reason, you will have to install a lot of dependencies to be able to run all experiments with all methods we compare.

For running the LSTM, you will need to set up CUDA 11.0.3 together with the cudnn8 library, since this is the version for which tensorflow 2.4.1 was compiled.
To run CAntMinerPB you will also need a Java JRE. We used OpenJDK 11.

To run the experiments and our code in general, you need a Python 3.8 environment. We recommend to use Anaconda to set up a virtual environment. If you have "conda" and "make" installed, you can simply run

```
make create_conda_env
```

to set up a virtual environment.

Make sure that Cython is available:

```
pip install Cython && pip install -r requirements.txt
```

If your are on Windows, you need to install a precompiled gmpy2 wheel:

```
pip install https://download.lfd.uci.edu/pythonlibs/y2rycu7g/gmpy2-2.0.8-cp38-cp38-win_amd64.whl;
```

In case this link does not work, visit https://www.lfd.uci.edu/~gohlke/pythonlibs/ and get a suitable wheel file yourself.

Then, install the required Python dependencies of our project using

```
pip install -r requirements.txt
```

Some of our sources must be compiled with Cython:

```
make cython
```

If you want to run experiments with the multilabel classificator Boomer, run

```
cd prolothar_rule_mining/rule_miner/multilabel/boomer && make install
```

## Running the Experiments

If you run the experiments with Docker, the template for all commands is

```
docker run --rm -it --gpus='"device=0"' --entrypoint /bin/bash --volume ~/rule_mining_experiments:/opt/prolothar-rule-mining/experiments/rule_mining/comparison/results --volume ~/rule_mining_experiments/talos:/opt/prolothar-rule-mining/data2seq consequence-experiments:latest -c '<experiment command>'
```

The volumes are necessary such that you can access the generated files of the experiments, such as plots. Plug one of the following commands into <experiment command> to run an experiment. If you do not use Docker, you can directly execute the commands. In this case, set the environment variable "PYTHONPATH" to the project directory.

### Parameters for Artificial Models
All experiments with the parameter "--dataset artificial" also enable a set of parameters to control generation of artificial models:

The "--model" parameter sets the algorithm for generating artificial models for sequence generation. The value "eventflowgraph2" produces potentially narrow but deep graphs that can generate longer sequences that share many common parts. The value "eventflowgraph" produces broader graphs that can generate shorter sequences with independent behavior.

For both model types:

| Parameter                  | Description                                           | Default |
| -------------------------- | ----------------------------------------------------- | ------- |
| nr_of_categorical_features | nr of categorical features to generate                | 2       |
| nr_of_numerical_features   | nr_of_numerical_features                              | 3       |
| nr_of_categories           | nr of categories for categorical features             | 3       |
| max_condition_depth        | maximum depth of generated rule conditions            | 0       |
| size_of_artificial_dataset | number of generated instances                         | 10000   |
| nr_of_sequence_symbols     | size of the event alphabet                            | 20      |

For "--model eventflowgraph":

| Parameter                | Description                                                   | Default |
| -------------------------| ------------------------------------------------------------- | ------- |
| max_nr_of_nodes_in_model | maximal numbers of nodes in the generated event-flow graph    | 25      |
| edge_probability         | probability of connecting two nodes. determines graph density | 0.1     |

For "--model eventflowgraph2":

| Parameter       | Description                                                                                                             | Default |
| ----------------| ----------------------------------------------------------------------------------------------------------------------- | ------- |
| min_rule_length | minimal length of a generated rule list per level (for nested if-else rules), which is converted to an event-flow graph | 10 1 1  |
| max_rule_length | maximal length of a generated rule list per level (for nested if-else rules), which is converted to an event-flow graph | 20 3 1  |
| max_rule_depth  | maximal depth of the generated rule list, which is converted to an event-flow graph                                     | 1       |

### Noise Robustness (Figure 2 in the paper)

Experiment for destructive noise:
```
python ${PYTHONPATH}/experiments/rule_mining/comparison/run_varying_noise_level.py --dataset artificial --miners DATS ConSequence BoomerLP LSTM KNN PosCl --nr_of_instances 1000 --nr_of_repetitions 10 --remove_noise 0.0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 --model eventflowgraph2 --nr_of_cpus 1
```

Experiment for additive noise:
```
python ${PYTHONPATH}/experiments/rule_mining/comparison/run_varying_noise_level.py --dataset artificial --miners DATS ConSequence BoomerLP LSTM KNN PosCl --nr_of_instances 1000 --nr_of_repetitions 10 --add_noise 0.0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 --model eventflowgraph2 --nr_of_cpus 1
```

You will find the generated results under "experiments/rule_mining/comparison/results".

### Different Classifiers (Figure 3 in the paper)

```
python ${PYTHONPATH}/experiments/rule_mining/comparison/run_varying_nr_of_attributes.py --dataset artificial --miners ConSequence_Gerd ConSequence_Classy ConSequence_CAntMinerPB --nr_of_instances 1000 --nr_of_repetitions 10 --nr_of_attributes 10 20 30 40 50 60 70 80 90 100 --model eventflowgraph2 --nr_of_cpus 1
```

You will find the generated results under "experiments/rule_mining/comparison/results".

### Sample Complexity (Figure 4 in the paper)

```
python ${PYTHONPATH}/experiments/rule_mining/comparison/run_comparison.py --data RollingMill --nr_of_repetitions 10 --nr_of_instances 10 100 500 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 --miners DATS ConSequence BoomerLP LSTM KNN PosCl --nr_of_cpus=1
```

You will find the generated results under "experiments/rule_mining/comparison/results".

### Results on Real-World Datasets (Table 2 in the paper)

You will find the generated results of the following experiments under "experiments/rule_mining/comparison/results".

#### Production Dataset:

```
python ${PYTHONPATH}/experiments/rule_mining/comparison/run_comparison.py --data Production --nr_of_repetitions 10 --nr_of_instances 180 --miners DATS ConSequence BoomerLP LSTM KNN PosCl --nr_of_cpus=1
```

#### Sepsis Dataset:

```
python ${PYTHONPATH}/experiments/rule_mining/comparison/run_comparison.py --data Production --nr_of_repetitions 10 --nr_of_instances 625 --miners DATS_CatNB ConSequence BoomerLP LSTM KNN PosCl --nr_of_cpus=1
```

#### Rolling Mill Dataset:

```
python ${PYTHONPATH}/experiments/rule_mining/comparison/run_comparison.py --data RollingMill --nr_of_repetitions 10 --nr_of_instances 39386 --miners DATS ConSequence BoomerLP LSTM KNN PosCl --nr_of_cpus=1
```

Since the PosCl method is really slow on the Rolling Mill dataset, we recommend a separate run with increased number of CPUs. We used 32 in our experiments.

#### Software Dataset:

```
python ${PYTHONPATH}/experiments/rule_mining/comparison/run_comparison.py --data dconvert_class --nr_of_repetitions 10 --nr_of_instances 400 --miners DATS ConSequence BoomerLP LSTM KNN PosCl --nr_of_cpus=1
```

### Case Studies

You conduct a single run on a dataset for each individual method. This is how we generated results for the case study. For the Rolling Mill dataset, we were only allowed to publish an anonymized version, i.e. event names, attribute names and categorical values were integer labeled and numerical values rescaled. You will find all datasets in the "dependencies/data-public-1.11.1.tar.gz" archive.

#### Running ConSequence on a single dataset:

```
python ${PYTHONPATH}/experiments/rule_mining/rudi/single_run_on_dataset.py --data Sepsis'
```

You will find the results under "experiments/rule_mining/rudi/results".

#### Running Boomer on a single dataset:

```
python ${PYTHONPATH}/experiments/rule_mining/boomer/single_run_on_dataset.py --data Sepsis'
```

You will find the results under "experiments/rule_mining/boomer/results".

#### Running PosCl on a single dataset:

```
python ${PYTHONPATH}/experiments/rule_mining/cls/single_run_on_dataset.py --data Sepsis'
```

You will find the results under "experiments/rule_mining/cls/results".

#### Running LSTM on a single dataset:

```
python ${PYTHONPATH}/experiments/rule_mining/data_to_sequence_lstm/single_run_on_dataset.py --data Sepsis'
```

You will find the results under "experiments/rule_mining/data_to_sequence_lstm/results".

## Seed for Pseudo-Random Number Generators

In all experiments, you can add the parameter "--random_seed 42".

## Run ConSequence on your own dataset

To run Consequence on your own dataset, execute

```
python ${PYTHONPATH}/experiments/rule_mining/rudi/single_run_on_dataset.py --data {path_to_file.arff}'
```

Make sure, that the name of your file ends with ".arff". Your .arff file should be UTF-8 encoded and should have the following format:

```
@RELATION "Name of your dataset"

@ATTRIBUTE "color" {"green","red","blue"}
@ATTRIBUTE "size" NUMERIC
@ATTRIBUTE "sequence" {"[a,b,c]","[a,a,b,c"]}

@DATA
"green",0.54,"[a,b,c]"
"red",0.12,"[a,a,b,c]"
"blue",0.42,"[a,b,c]"
```

To avoid any unexpected problems, avoid the use of spaces or special characters in attribute names, values and events.
You will find the results under "experiments/rule_mining/rudi/results".

In case you are running the experiments with Docker, either include your dataset into the container by modifying the Dockerfile or use a volume.