Phone meeting with John Apostolakis 3222012

Marc's notes

We started with a list of questions sent to John, as a primer for what we’re interested in discussing.

“We're thinking about one invocation of the "transportation process", and wondering about the things that have to be calculated. Is the following list complete?
1. Movement (in space) of the particle. Does the particle also move in time?
2. Change in momentum of the particle.
3. Does the particle lose energy? If so, does this happen at more than one location along the trajectory during a single invocation of the transportation process?
4. How is the information regarding crossing of boundaries between volumes used?
5. Is there anything that is dependent on the variety of solver used to integrate the equations of motion?
6. Are the chords used in the calculation an artifact of the discrete steps taken during the integration of the equations of motion, or is there some other relevance to them?
7. What is the role of the "step length" as determined by choosing from step lengths samples from the candidate physics processes?
8. For a given step length (proposed from the selected physics process), how is the size of the "delta step" or the number of delta steps determined?
9. When the proposed step length is larger than the distance to the next volume boundary, is the step limited by geometry, thus ending at the boundary?
10 What is the role of "safety", and how is it calculated?

We are also interested in obtaining a clearer understanding of how the transportation process is either integrated or interleaved with the physics processes that describe interactions with matter.”

Particle does move in time; in electric fields, the time is an integration variable.

Change is momentum is only in direction, in a magnetic field.

Propagation in field does NOT account for energy loss. No energy loss in a single step from the SteppingManager. The SteppingManager is the thing that polls the physics processes.

The propagation algorithm breaks up the cuve path into straight pieces, such that the saggital distance is small compared to the errors one cares about. If a volume boundary is reached, the step is ended. The physics process remembers that it has consumed some amount of the distance being stepped through, and relies on the fact that the next time it is queried it is propagating the same particle. We need to look into how the Physics Processes are keeping track of particle information. We also need to study the SteppingManager. The distance to a boundary is unknown to a physics process. The physics processes keep track of interaction lengths traversed, even when moving across volumes of different materials. Every time the transportation moves a particle into a new volume, new physics process proposals are calculated -- but the calculated path length is remembered.

“Tracking” is physics processes together with transportation and geometry.

Along the step, interactions can do things like multiple scattering, and the transportation can bend the trajectory.

Because the geometry system can only deal with straight lines in determining distances to surfaces, the curved trajectory is broken into chords with sufficiently small saggitas so that the boundary crossing calculation does not make large errors. This saggital distance is a parameter tunable by the user.

Candidate integration steps can be rejected because they are “too coarse”; that is, their saggital distance is too large, or the fractional error in the calculation is too large. This fractional error is also a tunable parameter.

If a volume boundary is not reached, the transportation will continue until the proposed physics process step length has been satisfied.

John notes about 50% of calculation time is taken by field and geometry and integration of equations of motion.

Q: Do we need to group particles by the approximate number of substeps they will need, in order to help keep from wasting GPU time during the step? Everyone seems to say yes (including John).

The “safety” computed in the geometry is the distance to the nearest volume from a particular point. Each process (or transportation) is required to have this value at start, at each substep, and at the final point. This value may allow one to use larger steps (because one doesn’t need to worry about crossing boundary volumes). EM particles emitted at any point that are smaller than the safety don’t have to be emitted. Some versions of the ionization process can produce secondaries along the path. The “isotropic safety” can also be used for optimization purposes.

Soon concentrated several times on the “AlongStepDoIt” function... we need to understand what this does. I don’t have much of an idea.

SteppingManager is responsible for doing tracking. At each step, “ionization” will say how much energy loss will happen and a step length, other processes will also do their things. Only the resulting proposed step length is sent to the transportation. The SteppingManager asks each process for the length of each step, and chooses the smaller. Each process is then told about the step size that was chosen, and what process occurred at the final point. Each process has the chance to modify the particle’s energy or momentum along the step, even if it is not the chosen process. All processes with an “along step” action can do this. “post step” actions are the ones that limit the step. Some processes will always be called, for “techinical reasons” e.g. scintillation or (maybe) multiple scattering. A process always called at the end of the step is called “forced”. This can be especially important for neutral particles, which might get reflected at boundaries.

We have to give very hard thought to what grouping of particles is done, to “batch up” processing on the GPU. There are many opportunities.

For John’s student’s work, he intends to start with dealing with neutral particles, and to try to implement some of the processes for running on the GPU. He’ll skip over what the SteppingManager does. John agrees with Rene that grouping particles by physical volumes makes sense. He also thinks that handling EM interactions on the GPU makes sense, but hadronic interactions are much more complicated. “It is completely inappropriate to vectorize it”. Electrons and gammas will be the first target for John.

John has some “ray tracing” geometry that uses 5 types of solids, in CUDA. This is not sufficiently realistic for real work.

John is dubious about having a final version of Geant4 that uses a GPU only for particle transport; he thinks the full calculations (physics processes and geometry) on the GPU.

Jim's notes

Particle move in time? yes, but at end of the movement. for electric field it is an integration variable.
Not renormalize the momentum as it moves.
Propagation does not account for energy loss now, but they did walk about it. No loss during one step. Accounted for after movement is complete.
Boundaries in the geometry are checked during one step by that the propagation can be stopped.
A step that is truncated because of the volume crossing (in propagation) will continue down that path to the original given length after depositing energy (and making hit). Physics processes not evaluated in this case.
Physics and geometry are independent. All physics processes need to be inspected if during one step a volume boundary is crossed because the material is likely to change.
Cached information is reset at certain points - when volume crossed.
tracking = geometry + transportation.
Optimization - volume crossing is treated as a new step from the physics processes? confused a bit.
Typically the same physics processes are polled (active for the particles in this step) for a volume crossing during a step to recalculate things like interaction length, but probably not much more than that.
Lots of this is just a choice - how far to move a particle. Sounds somewhat based on optimizing to be able to get the job done in a reasonable time.

tracking - asks what will happen to particles by physics processes, they give lengths (propagation is limited by geometry and by transportation). Energy deposits are made at end of step, regardless if ended by transportation or geometry.

only linear paths can talk to the geometry. chords aid in the calculation of the errors.
integration step candidates and chord choice sounds like it is based on a heuristic.

50% of time spent in geometry and integration according to John.

John calls the integration steps the “substeps”. The physics steps are the “master” steps.
Will need to keep the variable path length design (used in RK4), or group like lengths together in the particle group. It will be tricky to choose the path length carefully.
There is rescaling done before the transportation process is invoked.

I believe we are still going in a reasonable direction.

“safety” in geometry - distance of a point to a nearest volume. transportation must know for starting point, also for intermediate points, and for final point. this helps determine how far a particle can safely go.
isotropic safety.
energy loss process - far steps might be tricky? because of how the energy might be deposited along the path.

There is a poll - stepping manager asks all processes - what is your step? it picks one and tells processes the chosen value. The important thing here is all the processes that have an “along step” action. There is also an “end step” action that is important. (don’t understand where this part of the conversation went and why this is relevant - I could not type fast enough).

There was a talk about the tracking process in Geant4 by Mark. John A will look for it.

Will start with neutral particles and simple physics list (photoelectric, pair production). Go around the stepping manager. Charged particle is complicated. Similar particles in similar places (volumes) grouped together.
Looking at electrons and gammas (80% of the current CPU time). Wants a reasonably complete simulation - different than our idea. Doing this with five types of solids and ray tracing.

We have the paper from John’s student on the GPU work they are doing. They are not far enough along to determine how good this work is.

Particles may typically take a few substeps.