|
Overview Current research interests include applications of statistical learning in the arts, face and eye tracking. |
|
| Projects | |
|
VisualIDs: User interfaces need scenery, and suitable scenery can be invented with computer graphics techniques. |
|
Pose Space Deformation is a creature skinning/deformation algorithm that combines aspects of skeleton-driven skinning and blendshapes and improves on each. Paper appeared in Siggraph 2000; additional notes, code examples, video. |
|
A simple and quite effective approach to subsurface scattering for skin, developed for Matrix Reloaded by George Borshukov and myself
(Siggraph 2003 sketch)
.
(Also see these other Matrix virtual actor sketches:
face tracking,
hair, cloth,
image-based lighting, others).
Expect to see this in games: Sander, Gosselin, and Mitchell described how to do this in real time on an ATI GPU in a Siggraph 2004 sketch and GDC presentation and Nvidia improve on this texture space diffusion approach for the subsurface component of their real-time skin rendering. |
|
Mapping the mental space of game genres. Starting from a large online survey, we use a classical psychological scaling technique, updated with a modern manifold learning approach, to algorithmically produce maps of the mental space of video game genres. |
|
Is software engineering engineering?
Engineering has been defined as the solution of practical problems using mathematical and scientific principles. Is it possible to apply mathematical and scientific principles to the solution of important problems in software engineering, such as the prediction of development schedules and the assessment of productivity and quality? No: algorithmic complexity results can be directly intepreted as indicating that software development schedules, productivity, and reliability cannot be objectively and feasibly estimated and so will remain a matter of intuition.
Large Limits to Software Estimation
Supplementary Material, link to Slashdot discussion |
|
Perceptual segmentation for NPR sketching. Existing NPR sketching schemes have segmented strokes using relatively simple measures such as curvature extrema. These schemes disregard the successive approximation nature of sketching -- that large details are sketched first, and small details (even if they contain curvature maxima) are only added later. This Graphite05 paper shows that spectral clustering can better approximate the perceptually guided successive approximation used in sketching. |
|
Face inpainting. Which of these two people has thicker lips? A web survey shows that humans find this question well posed and are usually able to guess correctly. This suggests that population face statistics can guide face inpainting. On the other hand, we find that face proportion statistics are clearly non-Gaussian, so simple PCA based approaches will leave room for improvement. We extend an active appearance model to employ local, positive-only reconstruction, resulting in surprisingly good extrapolation. |
|
Feature tracking. "Fast Template Matching", Vision Interface 1995. Describes a frequency-domain algorithm for normalized cross-correlation (the convolution theorem gives un-normalized cross correlation). Developed for Forest Gump, it reduced overnight tracking runs to an hour or so (i.e. 10-15x speedup) and has been used extensively in subsequent ILM projects. This algorithm is now used in several commercial software packages
including Apple Shake and Motion.
conference paper, expanded/corrected version: .pdf, ps.gz, html. |
|
Generalized fractals. "Generalized stochastic subdivision", ACM Transactions on Graphics July 1987 applies estimation theory to remove artifacts in the popular fractal subdivision construction and generalize it to arbitrary non-fractal power spectra (e.g. ocean waves), producing a multiscale texture synthesis. This is an early application of Gaussian Process regression in graphics.
The work is discussed in the book Peitgen and Saupe, The Science of Fractal Images (Springer-Verlag 1988), section 2.6.
The 1987 article with additional figures is here
(pdf,
html preview)
Also see ``Is the Fractal Model Appropriate for Terrain?'', (pdf) |
|
Accelerated Blendshape Animation. Blendshape modelers spend considerable time attempting to make blendshapes that do not interfere. This technique reduces the interference effect in blendshape animation. |
|
Automatic Lip-Sync: Automated lip-synch and speech synthesis for character animation (CHI 87) describes an algorithm for identifying the mouth positions corresponding to speech in an audio soundtrack. Co-articulation was approximated with a now-standard smoothing approach (smoothing splines). The algorithm has been covered in Siggraph character animation courses and is the subject of U.S. Patent No. 4,913,539. |
|
Painting with Textures. "Texture synthesis for digital painting", Siggraph 84 argued that the expressiveness of traditional painting media can be considered as a texture synthesis problem, and presented texture synthesis algorithms suitable for interactive painting. Note that this was a number of years prior to the introduction of programs like Photoshop and Painter. A small gallery of paintings made with this program, 1982-83. Paper, missing some figures. |
|
Sparse Convolution is a simple texture synthesis method that can resynthesize some (random phase) textures given a sample. It was introduced in the 84 Siggraph paper (above), and explained in more detail in this siggraph89 paper. This algorithm removes the directional artifacts found in the standard implementation of Perlin noise and has other capabilities. The algorithm is used in RenderDotC and Houdini. |
|
Coherent Phase Texture Synthesis
(unpublished research) Many texture synthesis approaches produce more or less 'random phase' textures, i.e., coherent sharp 'structural' details are not present. This synthesis algorithm has a coherence knob that can produce such structural details in a synthetic texture. Sample image |
|
Neural Network pattern synthesis.
"Creation by Refinement" (ICNN88) describes a neural net algorithm for
example-based pattern synthesis. This simple algorithm is a new application
for neural nets--other neural algorithms involve recognition, mapping, and
optimization rather than pattern synthesis. It would seem to be an appropriate
approach for partially structured problem domains such as music composition
and texture synthesis. Some initial (and not very impressive) explorations of
these areas appeared in the MIT Press book Music and Connectionism; a proposal
for image-based texture synthesis and a trivial demonstration of such appears
in the proceeding Graphics Interface 91.
(pdf, missing some figures). |
|
Agent interface metaphor. Soft Machine: a personable interface, Graphics Interface 84 described an early conversational interface agent (possibly the first system in which the agent is personified with a graphical likeness) employed as a user interface. In this system, a user accessed a video database via stateless conversation with a graphical robot. The system used speech synthesis, lip-synch, and restricted domain speech recognition orchestrated by a standard conversational Lisp program. The use of primitive behaviours in the robot (subtle head movement, two `emotional' states) and (especially) humor in the robot's responses was intended to compensate for limited speech recognition accuracy. This work was completed perhaps a decade before the advent of the `agent' buzzword. A video showing the system interacting with an architectural database on videodisk. |
|
Eye contact by image-based rendering. "Teleconferencing Eye Contact" (joint work with M.Ott and I.Cox at NEC Research). This is possibly the first work to address the videoconferencing eye contact problem using stereo, an approach that has now become popular. Several cameras are mounted around the monitor, and stereo reconstruction is used to form a crude 3D depth model which is filled in and smoothed using scattered data interpolation; this model is then texture mapped and rotated to achieve eye contact. A non-real-time implementation of the technique shown at CHI93 showed that within a small range of orientation, the impression of eye contact is apparently determined by the overall proportions of the facial projection rather than the eye projection specifically, and thus even a low-resolution depth reconstruction provides an impression of eye-contact. This technique is the subject of US patent #5,359,362. |
 |
 |
JP Lewis
Bio & CV
Previous affiliations include USC, Stanford, Interval Research, Industrial Light & Magic, Disney, MIT
