Additional metrics to compare QSS3 and Geant4¶

Efficiency¶

One of the experiments aimed at comparing Geant4 against QSS3 consisted in varying simultaneously several accuracy parameters (i.e., `epsilon`, `stepMax` and `deltaChord`). In particular, the ranges explored for each parameter were the following:

• `stepMax`: 0.2 mm, 1 mm, 2 mm, 10 mm, 20 mm
• `deltaChord`: 0.01 mm, 0.05 mm, 0.25 mm, 1 mm, 2 mm
• `epsilon`: 1e-10, 1e-9,..., 1e-3

One possible way of visualizing the information obtained is fixing a given value of, for example, `deltaChord`, and plotting error and simulation time curves for every value of `stepMax`, using two y-axes:

Here we can see that, around `epsilon`/`dQRel = 1e-7`, QSS3 offers an error which is about three orders of magnitude below that offered by Geant4. Also, this error keeps decreasing one order of magnitude for each additional order of magnitude of `dQRel`. Simulation times tend to increase for both methods but, since Geant4's errors do not change, this extra computation time can be understood as an efficiency loss.

Considering this, and in an attempt to consolidate the outcomes of every simulation we performed, we defined a efficiency metric that combines both the error and the simulation time:

`efficiency = 1 / (max_x_err * t_sim)`

Since ideally we want these two values to be as small as possible, the efficiency should be high when this goal is achieved.

The image attached below shows an efficiency plot taking into account each of the simulations:

It can be observed that Geant4 stays inside an efficiency region that does not change as `epsilon` decreases. Another key observation is the fact that, the lower `dQRel`, the higher the efficiency achieved by QSS3. Despite needing more computation time to satisfy harder precision requirements, the error, as shown above, decreases one order of magnitude at a time, which in turn makes the efficiency higher. When reaching the rightmost end of the plot (`epsilon`/`dQRel = 1e-9`), QSS3's efficiency is about three orders of magnitude higher than Geant4's.

Speedup metrics¶

As we already know, in order to study how QSS3 performs against Geant4 when facing discontinuities, the model was extended with planes that cross the x-axis at regular intervals. For different numbers of planes between 0 and 200, the errors and simulation times obtained are summarized in the next plot:

It can be seen that the error for `epsilon`/`dQRel = 1e-5` is significantly lower for QSS3. Also, as the `G4_t_sim` curves show, the higher the number of planes, the longer it takes Geant4 to complete the simulations. On the opposite, QSS3 seems to be less sensitive to this variation. This observation can be better understood using the following speedup metric:

`speedup_i = G4_t_sim_i / QSS3_t_sim_i`

where `i` is the number of planes (of course, both times should be measured using the same relative precision). This value increases when QSS3 times outperform those of Geant4.

The corresponding plot of this metric is shown next:

• Plot using mean values of 70 runs:

For every value of `epsilon`, we can see that the speedup increases as the number of planes increases. Also, the speedup decreases as `epsilon` decreases. This is expected since QSS3 needs more time to satisfy harder precision requirements.

However, it is still not clear if this speedup comes from better discontinuity handling, as we expect, or if other factors are involved. This motivated the definition of additional speedup metrics. First, we introduced the `normalizedSpeedup`:

`normalizedSpeedup_i = speedup_i / speedup_0`

This metric is useful to eliminate any hidden multiplicative factor in the simulators. The results are shown in the following plot:

This time, the three curves are quite similar but still the speedup is higher as the number of planes increases.

• Plot using mean values of 70 runs:

Finally, we defined a last speedup metric to capture the computational effort of handling discontinuities. For each method, we first defined the notion of `penalty` for handling `i` planes as the difference between the simulation time when using `i` planes and the simulation time using no planes at all. This value can be interpreted as the additional time the method requires to handle `i` crossing planes. Thus, we have the `penaltyQuotient` metric defined as follows:

`penaltyQuotient_i = G4_penalty_i / QSS3_penalty_i = (G4_t_sim_i - G4_t_sim_0) / (QSS3_t_sim_i - QSS3_t_sim_0)`

The next image shows the results obtained for this metric:

It can be seen that this quotient seems to stabilize for a high number of planes. Also, an important conclusion is that QSS3 offers a discontinuity handling speedup that is between two and three orders of magnitude against Geant4.

• Plot using mean values of 70 runs:

Statistics for the experiments of 70 runs¶

Geant4¶

Planes Epsilon Mean simulation time [s] Standard deviation [s]
0 0.001 1.12071 0.017754
0 0.0001 1.125 0.0381632
0 1e-05 1.11771 0.0201545
1 0.001 1.84429 0.0452792
1 0.0001 1.84771 0.0478143
1 1e-05 1.84929 0.0630266
5 0.001 3.64771 0.0489948
5 0.0001 3.64557 0.0398438
5 1e-05 3.64557 0.0425822
10 0.001 5.96843 0.0675835
10 0.0001 5.98714 0.0664217
10 1e-05 5.98586 0.0585416
20 0.001 11.4146 0.119579
20 0.0001 11.4103 0.13756
20 1e-05 11.4241 0.127711
40 0.001 21.2599 0.239473
40 0.0001 21.542 0.234415
40 1e-05 21.582 0.215543
80 0.001 43.0771 0.452208
80 0.0001 43.215 0.418954
80 1e-05 43.548 0.419362
100 0.001 51.4607 0.440566
100 0.0001 51.6797 0.584705
100 1e-05 51.862 0.512809
120 0.001 62.8 0.59377
120 0.0001 62.9546 0.675355
120 1e-05 63.053 0.712092
150 0.001 76.1953 0.584809
150 0.0001 76.4023 0.756468
150 1e-05 76.5356 0.804877
200 0.001 98.8056 1.10258
200 0.0001 98.8197 1.02737
200 1e-05 98.8664 0.958688

QSS3¶

Planes dQRel Mean simulation time [s] Standard deviation [s]
0 0.001 0.400413 0.00357368
0 0.0001 0.855355 0.00677208
0 1e-05 1.83983 0.0101865
1 0.001 0.403255 0.00337518
1 0.0001 0.866389 0.0228614
1 1e-05 1.85673 0.0137374
5 0.001 0.405564 0.00343875
5 0.0001 0.871047 0.00841416
5 1e-05 1.87363 0.0220114
10 0.001 0.408367 0.00507485
10 0.0001 0.87721 0.0182026
10 1e-05 1.88529 0.0199692
20 0.001 0.412702 0.00515439
20 0.0001 0.889147 0.0101678
20 1e-05 1.89298 0.022848
40 0.001 0.421196 0.00720345
40 0.0001 0.906655 0.00835951
40 1e-05 1.91798 0.0215266
80 0.001 0.436378 0.00696315
80 0.0001 0.946367 0.0110264
80 1e-05 1.9598 0.0220239
100 0.001 0.442792 0.00582263
100 0.0001 0.963277 0.010012
100 1e-05 1.9852 0.0258647
120 0.001 0.450626 0.00574638
120 0.0001 0.981664 0.0108898
120 1e-05 1.99794 0.0174163
150 0.001 0.461638 0.00701499
150 0.0001 1.00894 0.0102798
150 1e-05 2.02968 0.0195574
200 0.001 0.479826 0.00475082
200 0.0001 1.05507 0.0121799
200 1e-05 2.08569 0.0173966