Additional metrics to compare QSS3 and Geant4¶
Efficiency¶
One of the experiments aimed at comparing Geant4 against QSS3 consisted in varying simultaneously several accuracy parameters (i.e., epsilon, stepMax and deltaChord). In particular, the ranges explored for each parameter were the following:
stepMax: 0.2 mm, 1 mm, 2 mm, 10 mm, 20 mmdeltaChord: 0.01 mm, 0.05 mm, 0.25 mm, 1 mm, 2 mmepsilon: 1e-10, 1e-9,..., 1e-3
One possible way of visualizing the information obtained is fixing a given value of, for example, deltaChord, and plotting error and simulation time curves for every value of stepMax, using two y-axes:
Here we can see that, around epsilon/dQRel = 1e-7, QSS3 offers an error which is about three orders of magnitude below that offered by Geant4. Also, this error keeps decreasing one order of magnitude for each additional order of magnitude of dQRel. Simulation times tend to increase for both methods but, since Geant4's errors do not change, this extra computation time can be understood as an efficiency loss.
Considering this, and in an attempt to consolidate the outcomes of every simulation we performed, we defined a efficiency metric that combines both the error and the simulation time:
efficiency = 1 / (max_x_err * t_sim)
Since ideally we want these two values to be as small as possible, the efficiency should be high when this goal is achieved.
The image attached below shows an efficiency plot taking into account each of the simulations:
It can be observed that Geant4 stays inside an efficiency region that does not change as epsilon decreases. Another key observation is the fact that, the lower dQRel, the higher the efficiency achieved by QSS3. Despite needing more computation time to satisfy harder precision requirements, the error, as shown above, decreases one order of magnitude at a time, which in turn makes the efficiency higher. When reaching the rightmost end of the plot (epsilon/dQRel = 1e-9), QSS3's efficiency is about three orders of magnitude higher than Geant4's.
Speedup metrics¶
As we already know, in order to study how QSS3 performs against Geant4 when facing discontinuities, the model was extended with planes that cross the x-axis at regular intervals. For different numbers of planes between 0 and 200, the errors and simulation times obtained are summarized in the next plot:
It can be seen that the error for epsilon/dQRel = 1e-5 is significantly lower for QSS3. Also, as the G4_t_sim curves show, the higher the number of planes, the longer it takes Geant4 to complete the simulations. On the opposite, QSS3 seems to be less sensitive to this variation. This observation can be better understood using the following speedup metric:
speedup_i = G4_t_sim_i / QSS3_t_sim_i
where i is the number of planes (of course, both times should be measured using the same relative precision). This value increases when QSS3 times outperform those of Geant4.
The corresponding plot of this metric is shown next:
- Plot using mean values of 70 runs:
For every value of epsilon, we can see that the speedup increases as the number of planes increases. Also, the speedup decreases as epsilon decreases. This is expected since QSS3 needs more time to satisfy harder precision requirements.
However, it is still not clear if this speedup comes from better discontinuity handling, as we expect, or if other factors are involved. This motivated the definition of additional speedup metrics. First, we introduced the normalizedSpeedup:
normalizedSpeedup_i = speedup_i / speedup_0
This metric is useful to eliminate any hidden multiplicative factor in the simulators. The results are shown in the following plot:
This time, the three curves are quite similar but still the speedup is higher as the number of planes increases.
- Plot using mean values of 70 runs:
Finally, we defined a last speedup metric to capture the computational effort of handling discontinuities. For each method, we first defined the notion of penalty for handling i planes as the difference between the simulation time when using i planes and the simulation time using no planes at all. This value can be interpreted as the additional time the method requires to handle i crossing planes. Thus, we have the penaltyQuotient metric defined as follows:
penaltyQuotient_i = G4_penalty_i / QSS3_penalty_i = (G4_t_sim_i - G4_t_sim_0) / (QSS3_t_sim_i - QSS3_t_sim_0)
The next image shows the results obtained for this metric:
It can be seen that this quotient seems to stabilize for a high number of planes. Also, an important conclusion is that QSS3 offers a discontinuity handling speedup that is between two and three orders of magnitude against Geant4.
- Plot using mean values of 70 runs:
Statistics for the experiments of 70 runs¶
Geant4¶
| Planes | Epsilon | Mean simulation time [s] | Standard deviation [s] |
|---|---|---|---|
| 0 | 0.001 | 1.12071 | 0.017754 |
| 0 | 0.0001 | 1.125 | 0.0381632 |
| 0 | 1e-05 | 1.11771 | 0.0201545 |
| 1 | 0.001 | 1.84429 | 0.0452792 |
| 1 | 0.0001 | 1.84771 | 0.0478143 |
| 1 | 1e-05 | 1.84929 | 0.0630266 |
| 5 | 0.001 | 3.64771 | 0.0489948 |
| 5 | 0.0001 | 3.64557 | 0.0398438 |
| 5 | 1e-05 | 3.64557 | 0.0425822 |
| 10 | 0.001 | 5.96843 | 0.0675835 |
| 10 | 0.0001 | 5.98714 | 0.0664217 |
| 10 | 1e-05 | 5.98586 | 0.0585416 |
| 20 | 0.001 | 11.4146 | 0.119579 |
| 20 | 0.0001 | 11.4103 | 0.13756 |
| 20 | 1e-05 | 11.4241 | 0.127711 |
| 40 | 0.001 | 21.2599 | 0.239473 |
| 40 | 0.0001 | 21.542 | 0.234415 |
| 40 | 1e-05 | 21.582 | 0.215543 |
| 80 | 0.001 | 43.0771 | 0.452208 |
| 80 | 0.0001 | 43.215 | 0.418954 |
| 80 | 1e-05 | 43.548 | 0.419362 |
| 100 | 0.001 | 51.4607 | 0.440566 |
| 100 | 0.0001 | 51.6797 | 0.584705 |
| 100 | 1e-05 | 51.862 | 0.512809 |
| 120 | 0.001 | 62.8 | 0.59377 |
| 120 | 0.0001 | 62.9546 | 0.675355 |
| 120 | 1e-05 | 63.053 | 0.712092 |
| 150 | 0.001 | 76.1953 | 0.584809 |
| 150 | 0.0001 | 76.4023 | 0.756468 |
| 150 | 1e-05 | 76.5356 | 0.804877 |
| 200 | 0.001 | 98.8056 | 1.10258 |
| 200 | 0.0001 | 98.8197 | 1.02737 |
| 200 | 1e-05 | 98.8664 | 0.958688 |
QSS3¶
| Planes | dQRel | Mean simulation time [s] | Standard deviation [s] |
|---|---|---|---|
| 0 | 0.001 | 0.400413 | 0.00357368 |
| 0 | 0.0001 | 0.855355 | 0.00677208 |
| 0 | 1e-05 | 1.83983 | 0.0101865 |
| 1 | 0.001 | 0.403255 | 0.00337518 |
| 1 | 0.0001 | 0.866389 | 0.0228614 |
| 1 | 1e-05 | 1.85673 | 0.0137374 |
| 5 | 0.001 | 0.405564 | 0.00343875 |
| 5 | 0.0001 | 0.871047 | 0.00841416 |
| 5 | 1e-05 | 1.87363 | 0.0220114 |
| 10 | 0.001 | 0.408367 | 0.00507485 |
| 10 | 0.0001 | 0.87721 | 0.0182026 |
| 10 | 1e-05 | 1.88529 | 0.0199692 |
| 20 | 0.001 | 0.412702 | 0.00515439 |
| 20 | 0.0001 | 0.889147 | 0.0101678 |
| 20 | 1e-05 | 1.89298 | 0.022848 |
| 40 | 0.001 | 0.421196 | 0.00720345 |
| 40 | 0.0001 | 0.906655 | 0.00835951 |
| 40 | 1e-05 | 1.91798 | 0.0215266 |
| 80 | 0.001 | 0.436378 | 0.00696315 |
| 80 | 0.0001 | 0.946367 | 0.0110264 |
| 80 | 1e-05 | 1.9598 | 0.0220239 |
| 100 | 0.001 | 0.442792 | 0.00582263 |
| 100 | 0.0001 | 0.963277 | 0.010012 |
| 100 | 1e-05 | 1.9852 | 0.0258647 |
| 120 | 0.001 | 0.450626 | 0.00574638 |
| 120 | 0.0001 | 0.981664 | 0.0108898 |
| 120 | 1e-05 | 1.99794 | 0.0174163 |
| 150 | 0.001 | 0.461638 | 0.00701499 |
| 150 | 0.0001 | 1.00894 | 0.0102798 |
| 150 | 1e-05 | 2.02968 | 0.0195574 |
| 200 | 0.001 | 0.479826 | 0.00475082 |
| 200 | 0.0001 | 1.05507 | 0.0121799 |
| 200 | 1e-05 | 2.08569 | 0.0173966 |