• 1-Dimensional

• (1) Reflectance

• (2) Transmittance

• (6) Sub Surface Scattering

• (8) Wavelength

• 2-Dimensional

• (-) Reflectance / Transmittance / SSS

• (4) Mirror vs. Diffuse

• (5) Index of Refraction

• Sensor modeling

• (7) noise

• (3) integration

• (1) 2.5D range finding (Depth as a function of angle)

• Finding the depth to first interface behind each pixel.

• (2) 3D range finding

• Finding the depth to hidden/back surfaces in the scene.

• (3) SSS estimation

• Estimating the mean-free-path length of a material from a single pixel.

• (-) velocity finding (only with wavelength)

• Calculate doppler shift taking into account multiple bounces to tell more about the motion of a surface element than the direct light path.

• (-) lossless matting (with time gating)

• Gate the response delay in time to collect light from only close objects.

• (-) decompose into number of bounce layers

• (-) the general class of thing where you use low quality RIF data enhance steady state, high-res data

• (-) enhanced human vision

• Create a display for humans that encodes almost all of the information about the steady state reflectance function but encodes additional information from the RIF.

---

Misc. Notes

1d case:

- have a list of active photos/wave fronts

- update them, expand tree

- collect items that hit sensor

- save their expressions to form function

- plot function and think

sensor modeling:

L(t) = Sensor(x,LightSource(RIF(BlipAtPoint(x,t))))

- sensor integrates over a window of some shape (box)

- light pulse integrates over a window of some shape (box)

- ambient light (subtract min)

- shot noise (ignore)

- electrical noise (gaussian)

- quantization noise (ignore)

crazy ideas for 2d?

- make stuff out of tons of points (need invent some way to shadow to make sure occlusion works)

- track piece-of-circle wavefronts

- what kind of "sensor" and "light source" will we have?

    - point

    - ray

computation:

- bound the world

- triangulate

- use generalized panels for eye and light also (points, line segments, and triangles)

- BRDF + BTDF = BSDF (Bidirectional Scattering Distribution Function) (assumed to be constant wrt time): S(X,n,n') (units: none, encodes a ratio for all input and output normal pairs at point X)

- incoming light over a surface: irradiant flux function I(X,n,t) (units: "power at point X", Watts, Joules/s, (common)photons/sec,  imulses in this function  have area measured in energy and width measured in seconds)

- outgoing light over a surface: radiant flux function R(X,n,t) (units: "power at point X", same as above)

- propagation combinator combines radiances from adjacent panels to form irradiance at panel in question: Propagate(Ra,Rb,Rc,Rd) ::= I(X,n,t) = Sum_L(MaxDot(n,n(X-Y))*Int_Y(Ra(Y,n(X-Y),t-c*d(X-Y))/d(X-Y)^2) -- L in legs a,b,c,d, Y in side L

- scattering combinator combines irradance at a panel and the scattering function to form radiances: Scatter(I,S) ::= R(X,n,t) = Sum_n'(S(X,n,n')*I(X,n',t)

- initially: define all S for each interface and R != 0 for emissive surfaces and R = 0 for the rest
- global transport combinator: Global(R, S) ::= Scatter(Propagate(Rs),S) for all panels on in adj simplex
- total irradiance is sum of each of irradiance function after infinite applications of G
- eye should be modeled as S=0 surface
- empty interfaces have S(X,n,n') = 1.0*I{n=n'}
- pulse light should be modeled as R_initial = R(X,n,t) = E_0*d(t-0) -- where d is delta function and pulse carries total energy E_0 per unit angle per unit time (radiance or radiant energy)
- constantly emmissive panels have R_initial = R(x,n,t) = P_0 -- a constant power that is not a function of time (radiant flux or radiant power)
- better names?: radiometric/photometric impulse response function

- need to define Transient Rendering Equation: R_{n+1} = R_{0} + Global of R_{n}, want R*
- R* = R_{0} + Global of R* --> R* - Global of R* = R_{0} -> (Identity - Global) of R* = R_{0} --> R* = Inverse(Identity-Global) of R_{0}
- Global is defined in terms of both Scatter and Propagate which are both, in general, not invertible (hella sums and integrals!), but we can invert (Identity - Global) with a Neumann series!!
- R* = (Identity + Global + Global^2 + Global^3 + ...) of R_{0}
- yay

"Transient Photometric Response Function"
- transient: short lived behaviour as distinct from steady-state
- photometric: light collected by an observer (radiometric used if there is no observer)
- response: we are kicking the scene with a single Dirac delta impulse of light with unit energy
- function: behaviour is only characterized as a function of several variables, not just response=17.2435

Simulation algorithm:


(*)

---

Misc

• Huygens wavefront tracing: A robust alternative to conventional ray tracing

• http://citeseer.ist.psu.edu/541860.html

• The rendering equation (seminal paper on path tracing, from 1986)

• http://portal.acm.org/citation.cfm?id=15902

• Eikonal Rendering

• http://www.mpi-inf.mpg.de/resources/EikonalRendering/index.html

• Pharr, Matt and Humphreys, Greg (2004). Physically Based Rendering : From Theory to Implementation. Morgan Kaufmann.

• A practical grid-based method for tracking multiple refraction and ... (seismology)

• http://www.blackwell-synergy.com/doi/pdf/10.1111/j.1365-246X.2006.03078.x?cookieSet=1

• Sonar image simulation by means of ray tracing and image processing

• http://portal.acm.org/citation.cfm?id=1139660

• Sonar Image Interpretation and Modelling. - Autonomous Underwater ...

• http://ieeexplore.ieee.org/iel3/3780/11039/00532430.pdf?arnumber=532430

• A 3D simulator for the design and evaluation of sonar system instrumentation
• http://www.iop.org/EJ/article/0957-0233/10/12/302/e91202.pdf
  

---

---

Introduction

Existing video capture systems, such as handheld camcorders, measure the amount of light received as a function of time simultaneously at many points in a sensor. However the sampling rate of such devices is on the order of a billion times too slow for capture of the TPRF for table-top scenes. However, current LIDAR technology already measures changes in light at this time-scale, although it does not capture the type of data in which we are interested.  Nonetheless, it is indeed possible to capture a TPRF with existing devices.

In computational photography there is an analogy to our work.  In this field, there are many models that describe more information about light in a scene than is necessary to reproduce a single, traditional photograph. For example, lightfields [the lightfield, lightfield rendering] capture the angular dependence of a light through a surface, whereas our work explores the short-term time dependence of light at a point. Lightfields have allowed for interesting and unexpected results such as the refocusing of images, changing the eyepoint of a camera, and synthetic aperture photography. Similarly, we expect that exploration of the TPRF to open up large, new application areas.

The challenge is to model the TPRF in a way that will allow us to answer questions about what kinds of applications might be possible given this new information. We have developed a physically relevant generalization of the (vacuum) rendering equation that we call the transient rendering equation. Our contribution is an initial understanding of the TPRF as well as an analytic method of approximating the function for quite general scenes. Additionally, we propose two toy applications that make use of the TRPF, including scene disambiguation and hidden shape recovery.

Reviewers: Josh McCoy, Sonia Arteaga

Draft Distributed: Nov 6

Feedback Received: Josh - Nov 6, Sonia - Nov 5

Revised: Nov 9

---

Related Work

Outside of the field of computer science, the analysis of reflected waves is an area of extensive interest and research. Close to our aim but in a different media, sonar-related research usually analyzes the transient effects of bouncing sound waves explicitly as a function of time. There is extensive modeling of multiple reflections, the effects of various media, and noise models in sensors ["Sonar Image Interpretation and Modeling"]. There is even work that attempts to build models of sound propagation using techniques from graphics ["Sonar image simulation by means of ray tracing and image processing"]. Though the particular models used in these simulations are mostly specific to the propagation of sound, and we hope to replicate their modeling efforts for the propagation of light.

In graphics rendering, we have models of steady-state light propagation [the rendering equation], as well as models of surfaces [BRDF] and volumetric materials [volume rendering equation] interacting with light.  The images that result from these complex models are easily assembled with photorealistic results using techniques such as raytracing ["Physically Based Rendering"], wavefront tracing ["Huygens wavefront tracing: A robust alternative to conventional ray tracing"], and eikonal rendering methods ["Eikonal Rendering"]. All of these methods attempt to explain the sum of light over time intersecting with an image plane, where we desire to investigate the instantaneous irradiance as a function of time.  Common to all of these methods is an explanation based in physics that assumes that the speed of light is infinite at the level of approximation they assume.

In the study of LIDAR imagery, there is analysis of the reflected light of a pulse as a function of time, explicitly taking into account the speed of light. However, strong assumptions are made about the shape of the scene and the distance of the sensor to the surfaces of interest ["Laser Radar"]. In addition, their analysis techniques focus on 2.5-D range imaging, and largely ignore inter-reflections within a scene, which are of particular interest in our work.

The fields of medical imaging as well as the spectral analysis of solids and gasses has also advanced research relating to the TPRF ["Temporal analysis of reflected optical signals for short pulse laser interaction with nonhomogeneous tissue phantoms", "Theory and simulation of ultrafast carrier dynamics and nonlinear pulse propagation in quantum dot semiconductor optical amplifiers"]. However, this research is more concerned with the diffraction of various wavelengths of electromagnetic waves as a result of various reflections at a very small spatial scale. Nonetheless, from this work, we know that the data we intend to simulate can be directly sensed in the physical world with existing hardware ["Spectro-Temporal imaging of optical pulses with a single time lens"].  Our work ignores diffraction and related effects based on the assumption of interest in visible light (photometry) and noticeable propagation delay.

Reviewers: Josh McCoy, Sonia Arteaga

Draft Distributed: Oct 24

Feedback Received: Josh - Oct 29, Sonia - Oct 26

Revised: Nov 5

---

Supporting Figures

1d world case:
- 1d world
- wavefront tree
- plot of the function

2d world case:
- 2d world
- plot of the function

(some) application example:
- hidden model
- plot
- feature point
- recovered model

Reviewers: Josh McCoy, Sonia Arteaga

Draft Distributed: Nov 12

Feedback Received: Josh - Nov 12, Sonia - Nov 13

Revised: Nov 14

---

Model

- need motivation for model
In order to formulate the TPRF for a scene in a given world, we turn to a formal definition of how light moves around a scene.  The rendering equation tells us how to formulate the steady-state light transport for quite general scenes, but it lacks the time depedence that is critical in our setting.
- Adapting slightly for our terminology, recall the rendering equation states that the radiance at a point in a given direction is the sum of the emissive radiance at that point in that direction and the result of applying a global light transport operator to the radiance at all other surfaces.  That is to say: R* = R_{0} + Global of R*.
- In our setting, we care about the light as a function of time, the radiant flux.
- The general statement of the rendering equation applies to our setting, with R now a function of time.  The global transport operator will also induce time-dependent effects in the functions it operates on, taking into account propagation time.
- In the next subsections, we lay out the assumptions of our model and formulate the Transient Rendering Equation, which we will use to derive expressions for the TPRF.
- R(X,n) --> R(X,n,t)
- Global --> Global {with speed of light not equal to infinity}

Assumptions

worlds:
- objects represented by many interfaces with flat, thin surface segments
- assume the existence of virtual interfaces so that the region of interest in the world is separated into simplices whereby light passes undisturbed within the body of the simplex (any valid generalized triangulation)

interfaces:
- have initial radiant flux R_{0}=0 (no wait, its just the contribution of the first lump of light in free space) (some DF, R(X,n), units are power, expected photons emitted per unit time per unit area per unit angle)
- have initial irradiant flux I_{0}=0 (always zero initially!!) (some DF, I(X,n), units are power, expected photons collected per unit time per unit area per unit angle)
- have scattering kernel (a BSDF, S(X,n,n'), unitless)

real interfaces:
- S = some kernel
 - 1d: just two scalars (reflectance and transmittance)
 - 2d: a 2d function
 - 3d: a 4d function
- R_{0} = 0 (assume all surfaces are non-emissive)

virtual interfaces:
- S = Identity (light in equals light out)

boundary interfaces: (virtual interfaces that do not connect to another simplex)
- S = 0 (ignore all light)

details for 1, 2, and 3 dimensional worlds:
- 1d:
 * interfaces are points with distinct locations in the world line
 * simplices are line segments (nonangles?) adjoining adjacent points in the world line
- 2d:
 * interfaces are line segments with distinct endpoint locations in the world plane
 * simplices are triangles adjoining adjacent points in the world plane
- 3d:
 * interfaces are triangles with distinct vertex locations in the world 3-space
 * simplices are tetrahedra adjoining adjacent points in the world plane

Transient Rendering Equation
- The Transient Rendering Equation we will define here has the same operator form as the standard rendering equation.  As such, there are a few operators we must define.
- Recall assumed model defines R_0,I_0, and S for every interface.
- Define Scatter(I_i) = R_{i+1} = R(X,n,t)_{i+1} = Integral_n(S(n,n')*I(X,n',t))
- Define Propagate(adjacent R_i in simplex) = I_i = I(X,n,t)_i = Sum_{L in adj}( Integral_{Y on L}(let d = dist(X-Y), n' = normal(X-Y) in R(X,n',t-c*d)*PosDot(n,n') ))
- Define Global(all R_i) = R_{i+1} Scatter of Propagate of R_i
- Recall operator form of standard rendering equation: R* = R_{0} + Global of R*
- Solve for R*: R* = Inverse(Identity-Global) of R_{0}
- Rewrite with Neumann Series: R* = (Identity + Global + Global^2 + Global^3 + ...) of R_{0}
- Apply composition: R* = R_{0} + Global of R_{0} + Global^2 of R_{0} + Global^3 of R_{0} + ...
- Recognize: R* = R*(X,n,t) now defined in terms of all R_0, I_0, S and scene geometry.
- Note: we can fine R*(Q,n,t) for any point Q inside of a simplex by summing the radiant flux contribution from the boundaries of that simplex taking care to sample with the correct normal direction and propagation delay

Deriving TPRFs
- The transient rendering equation was able to give us the form of R*(Q,n,t), the overall radiant flux at a any point in the world.  In free space, we assumed the scattering function was the identity, so we immediately know the irradiance for points in free space given the previous result.  If we take this function, I*(Q,n,t), and multiply it by some Observation operator (represented by function O that gives a sensitivity for every normal direction) we can read off the TPRF.
- Describe 1d example world:
 - imagine world with two half-reflectance, opaque interfaces distance alpha apart, with the light at the center of the world
 - observer has O(left)=1 and O(right)=0 and is at position o
 - light has initial radiance given by R_{0} = delta(t-0) as assumed by the model
 - recall: R* = R_{0} + Global of R_{0} + Global^2 of R_{0} + Global^3 of R_{0} + ...
 - want: O of Identity of R*
 - R* = R(Q,n,t) = delta(t-c|d|) +0.5*delta(t-c(w+|Q|+2*w)) +0.25*delta(t-c(0+|Q|+4*w))+0.125*delta(t-c(w+|Q|+6*w)) ... +(0.5^i)*delta(t-c(|d|+2*i*w))+(0.5^i)*delta(t-c({i is odd}*w+|Q|+2*i*w)) + ... )
 - note that reflected impulses lose power at a geometric rate in this scene and the evaluation of R* could be likely truncated quite soon in practice

- Describe 2d example world:
 - imagine  wold with two half-reflectance, lambertian, opaque interfaces...

General rendering equation: \mathsf{R_*} = \mathsf{R_0} + \mathsf{G} \mathsf{R_*}

---

Applications

(intro)
Recall that we needed to have a model of the TPRF that could support intelligent reasoning about properties of the scene.  In this section, we consider a pair of toy applications that illustrate exactly this ability.  First, we must describe a simple sensor model that can allow us to turn our analytic TPRF into plausible measurements.  Next, we use these measurements and application specific knowledge to come up with simulated measurements for other possible worlds and see which is closest to the observed data.

Sensor Modeling

In order to model a realistic sensor, it is necessary to ground out the TPRF in physical units.  Recall that the TPRF, evaluated at a given time, described an irradiant flux.  If we know the wavelength of the light in the scene, we can interpret the TPRF as describing the instantaneous expected number of photons per time unit that pass through the eye point.  As photons come in discrete quantities, we consider integrating the TPRF over short windows of time, to get an expected total of photons seen in the window.

 

Composition

Without taking into account any noise, there are still adaptations we would like to make do our model to get more realistic simulated measurements.  Importantly, our assumptions about the delta function envelope for the light pulse and observer sensitivity, as well as that the scene is devoid of light apart from our flash are wildly unrealistic.

First, our model assumes that the scene is illuminated by only an infinitely short pulse of light.  This, for several reasons, is physically impossible.  Instead, it is safer to assume that the light source emits some total amount of energy with some wide envelope in time instead of a delta function.  Fortunately, the correct TPRF can be synthesized for arbitrary envelopes in this way.  This ability derives directly from basing out analysis on the delta function.

Similarly, an observer would have to be able to sense the instantaneous irradiant flux in order to recover the TPRF directly.  This corresponds to having some video camera light device which integrates over an infinitely short shutter time.  Again, this is impossible physically.  We assume, instead, that the observer, when collecting a given sample in time, has some wide envelope for sensitivity.  Because convolution is associative, we can combine this envelope with that of the light source and come up with an estimated photon count for the experiment that is simply the area under convolution of the TPRF with some combined envelope, which we approximate with a Gaussian envelope.

Finally, realistic scenes are lit by more than just the test flash, meaning that an attempt to measure the TPRF will produce measurements that include extra light.  We assume that this additional light varies so slowly with respect to time so that it can be well approximated by a constant irradiant flux at each point.  That is, it is equivalent to adding some constant flux to the TPRF (which could be subtracted off from measured data after direct measurement of the constant value).

Taken together, our sensor model is concisely stated with the following expression.  Note that the measurement is proportional to an expected total energy.
    Measurement(X) = Integral_{all t}( TPRF(X,t) * LightEnvelope(t) * SensitivityEnvelope(t) )
    (note "*" above denotes convolution)

Noise

At this point, we have still only considered an idealized experiment.  To make our sensor model more realistic there are various sources of noise to consider.  Shot noise (intensity variation due to the random amount of photos actually collected during the integration period) will have a much greater effect on sensors measuring the transient properties of light than traditional application.  However, the effects of shot noise can be mitigated by using more intense flashes of light.  Next, digital quantization noise (inaccuracies due to coarse representation of values measured at particular timesteps) is also ignored because it is patterns of light over time that is of interest in our applications, with only minimal information coming from the particular values at each time step.  Finally, additional noise sources that we do not explicitly model cannot be so easily ignored.  To account for the combined effects from noise in different parts of the sensor, we apply simple additive gaussian white noise to each simulated measurement.

Thus, unsurprisingly, our combined sensor model is stated with the following expression:
    Measurement_i(X) = Integral_{all t}( TPRF(X,t) * LightEnvelope(t) * SensitivityEnvelope_i(t) ) + GaussianNoise_i

Scene Disambiguation

Eye and light, v-shaped object pointed down above eye and light, one side is dark, one side is light, where IS THE LIGHT? The example case we will consider is the estimation of a light position in a scene.
- all surfaces are lambertian with reflectance alpha

Shape Recovery

Scene with two boxes, want to know how deep one of the boxes is, by looking at the reflection off of the other
- all surfaces are lambertian with reflectance alpha

---

Future Work

Improving the model: (make case that each interestingly effects transient behavior)
- subsurface scattering: either properly create microgeometry and apply our analyis OR (more interestingly) define a transient volume rendering equation and re-derive the TRPF in this case
- optical density and wavelength: annotate interfaces with differing index of refraction and sample from BSDF differently or allow for general curved rays with continuous variation of optical density (would invalidate assumptions made on triangulation)
- phosphorescence: track energy density at points along interfaces and model decay

Building a real sensor:
- modify LIDAR imager
- high frequency oscilloscope and photosensor

Applications:
- formalized traditinal (2.5D) range finding in terms of TPRF and clarify unspoken assumptions
- attempt full 3.0D range finding
- esimtate subsurface scattering
- decompose TPRF into single bounce layers
- investigate ways to incorporate information extracted from live TPRF into traditinal video display (for augmented human vision)

---

Conclusion

In this paper we have introduced the idea of the transient photometric reflectance function and claimed that the function is not unreasonable to measure from the physical world.  We derived approximations to this function from analytic solutions to the transient rendering equation.  Next, we showed that, even with a sensor model that relaxes many of the original assumptions of the model, relevant information about a scene can still be recovered.  Finally, we have identified several areas for future development.  We are hopeful that this research will spur new inquiry into the transport of light in a scene that takes into account the transient effects of propagation.