• Should support functional composition and transformation. This basically means that it will support operations like a + b where a and b are geometry values and the result is a geometry value or f(a) where f is a pure function from geometry value to geometry value.
• A geometry value can be rendered by one call without any other companion values.
• Very limited magic and hard coded properties. Only properties relevant for rendering, vertex position, pixel color, and topology, are semi-magical in that a render call will look for them. All other properties and all calculations, such as those that specify layout or apply lighting, are non-magical and expressed explicitly by manipulating/transforming geometry values.

The above goals ensure that the abstraction is easy to use and that we can easily modularize geometry-building code using standard in-language constructs such as procedures, objects, or functions (including high-order functions). Now, let’s introduce the abstraction via an example using a neutral syntax:

// recipe for face
// create topology with 4 vertices and 2 primitives
face = new_topology<D3>(4, 2);
// set indices of topology primitives to (0,1,2) and (0,2,3)
// per_primitive creates a function applied to all primitives by their integer id.
face = set_indices(face, per_primitive(id => (0, 1+id, 2+id)));

The above code defines a face as a primitive triangle topology (D3 = 3 points) with four vertices and two triangles. The code then defines an index buffer that differs according to primitive ids (per_primitive…). As a result, there are two primitives formed from vertices (0,1,2) and (0,2,3). Now we define the spatial properties of a face:

// set vertex positions to 0:(-1,-1,0), 1:(-1,+1,0), 2:(+1,+1,0), 3:(+1,-1,0)
// per_vertex creates a function applied to all vertices by their vertex id.
face = make_position(face, per_vertex(id => (id/2*2-1,(id+1)%4/2*2-1,0)));
// face texture coordinates are similar to its positions, except in 2D and from 0,0 to 1,1.
face = make_texcoord(face, per_pixel((face.position.xy + 1) / 2));
// make face normal, just point it along the -z axis.
face = make_normal(face, (0,0,-1));
// give face a transform, initially this transform is identity.
face = make_transform(face, identity);

Positions are defined per vertex using a formula over vertex ids, which could have also been expressed as a table indexed by vertex ids if we wanted (formulas are more compact). Texture coordinates are defined according to the 2D position and shifted so that they are from (0,0) to (1,1) rather than (-1,-1) to (+1,+1). Additionally, texture coordinates are defined on a per-pixel basis, so that expressions that depend on them are never interpolated a the pixel shader. One normal is defined for the face and used as the normal for all of the face’s vertices and pixels. Finally, the face is given a matrix transform property with an initial value of the identity matrix. This call also ensures that the position and normal of the face are transformed appropriately according to the transformation matrix (normal transformation will ignore the matrix’s translation components). Now to form a cube from the face:

// duplicate the face 6 times to create a cube.
cube = duplicate(6, face);
// each face is translated out to (0,0,-1) and rotated according to its id
// note: we use open gl row major transform matrices, so prepend the matrices.
// transforms update position and normals,
// for normals the translate component is ignored when the transform is applied, of course.
// face 0 is rotated 0.0pi around (0,1,0)
// face 1 is rotated 0.5pi around (1,0,0)
// face 2 is rotated 0.5pi around (0,1,0)
// face 3 is rotated 1.0pi around (1,0,0)
// face 4 is rotated 1.5pi around (0,1,0)
// face 5 is rotated 1.5pi around (1,0,0)
cube = prepend_transform(cube, per_duplicate1(id =>
  rotate((0+id%2,1-id%2,0), (i/3+(id%3).Min(1))*.5pi) * translate(0,0,-1));

The first statement duplicates the face six times to form the beginnings of a cube. The next statement prepends a transform that translates/rotates the position and normal of each face according to its ID, which is simply its duplicate ID. Now…we have a complete cube! Incidentally, we could just encapsulate all of this code into a simple make_cube function, or better yet, just use the cube value as a prototype anytime we need a cube value. Continuing on, we color and light the cube:

// makes the cube ready for rendering by giving it a color
// property, color is initially transparent
cube = make_color(cube, transparent);
// prepare cube for lighting, causes color to be computed via lighting equations.
cube = prepare_lighting(cube);
// prepare materials.
cube = add_material(cube, specular(power=3));
cube = add_material(cube, diffuse(knob=100%,ambient=100%));
// set color of cube according to face id, so each face has a different color.
cube = set_material_color(cube, per_duplicate1(id => color_table[id]));
// light cube with an ambient light at 50% of white.
lights = ambient_light(color = .5 * white);
// directional light 50% white at a weird angle
lights = add_light(lights, directional_light(color = .5*white, direction = (-.5,-.5,+1)));
// spot light 50% white directed straight on.
lights = add_light(lights, spot_light(color = .5*white, direction=(0,0,+1),
                                      position=(0,0,-10),outer=4dg, inner=2dg));
// set the lights for this cube, same lights can be applied to other geometries.
cube = apply_lights(cube, lights);

This code adds a color property to the cube (before it had none!) allowing it to be rendered, or for our purposes, prepared for lighting. The prepare_lighting call adds a bunch of properties related to lighting as well as ensuring that the cube’s color is now computed via a lighting algorithm (configured by lighting properties). The next line’s adds diffuse and specular materials to the cube, set a material color for each face, and creates lights to light the cube. The lights value is used to light the cube geometry value but could also be applied light other geometry values in the same scene.

Stepping back, lighting is a fairly complicated algorithm and there are many different was to do it. For this reason, lighting is not built into the geometry abstraction, all of the lighting related functions (prepare_lighting, apply_lights, add_material) are all explicitly defined outside of the geometry abstraction and alternative lighting schemes can be defined in the same way. To finish up the rendering:

// rotation provided by world matrix, can be updated via user input.
cube = prepend_transform(cube, world);
// finally, a camera...
camera = perspective_camera(position = (0,0,-4), fov = .25pi);
cube = apply_camera(cube, camera);
// we get a cube, multi color faces, lit with three lights.
render(cube);

The world matrix provides for rotation and scaling, which will trickle down to the matrix that is used to transform face normal’s and vertex positions. A camera is created and then applied to the cube to provide for standard 3D view and projection matrices, and finally the cube is rendered. As a bonus, let’s apply normal mapping to the cube faces by updating the normal of each face to include the results of a texture lookup:

map = convert_to_normal_map(folded_paper_texture);
// extract normals from textures
// per-pixel texture coordinates to sample per-pixel normals.
cube = set_normal(cube, map[cube.texcoord]).rotate(cube.normal.angle_between((0,0,-1)));

Finally, you might get something like this:

I haven’t implemented this geometry abstraction yet, so the above rendering was developed from the way Bling is today.

In the above code, we are building up the type of the geometry value along with its structure. The type of the value indicates what transformations the geometry value supports (e.g., setting an existing property’s value) and prevents invalid transformations from occurring (e.g., applying normal-based lighting without specifying a normal!). However, I’m not aware of any static type system that allows us to mix and match independent types, so I’ll probably implement this through dynamic typing in C#.

Another issue that I need to address in more detail is that of what value one is referring to during a transformation. There is the value embedded in the value we are building from and the final value after the value is completed and when it undergoes processing (obtained from an implicit self parameter). In the above code, we mostly use the former semantics, but some operations obviously depend on the latter semantics; e.g., the normal I refer to in a lighting calculation is the final transformed version of the normal and not the current normal of a lighting calculation. Ah, kind of a headache.

I promised to talk about geometry shading in my last post, but it will have to wait. Basically, the constituent values used to build up a geometry value are not necessarily constants and can be dynamic values from the external environment; e.g., a slider thumb’s state, a texture sample, or a noise value. When the cube value is processed, if a dynamic value is encountered that affects the topology of the geometry, then a geometry shader is needed. We’ll still have to add some operations to transform the topology (possibly with a different primitive type) and to “step” the geometry when obvious state is involved. I’ll definitely talk about this next time.

• Pixels as children of geometric primitives. The position of a pixel is fixed and a pixel shader only specifies each pixel’s color and optional depth. It often makes sense to have other properties beyond color and depth at the pixel level for better shading computations; e.g., a normal that is computed on a per-pixel basis via a parametric equation or via normal mapping. Other properties can be accessed from previous vertex and geometry shading phases, where interpolation is used to determine the value of the property from the value of the property in the enclosing primitive’s vertices. Quality improves when properties are computed on a per-pixel basis rather than interpolated; e.g., consider Phong vs. Gouraud shading.
• Vertices as used to define primitives. A vertex shader specifies the position of each vertex, and will compute per-vertex properties that are later used in later geometry and pixel shading.
• Primitives that are used to form complete topologies. A primitive is either of a point, line, or triangle topology, with triangle being the primitive that most commonly reaches the rendering stage. A primitive is defined by multiple vertices, and encloses multiple pixels. Geometry shading works as a primitive translator, where one primitive can be translated into zero or more primitives of possibly a different topology. Primitives created during geometry shading are defined by specifying their vertices.
• A topology as a collection of primitives. Its initial definition is as a set of vertices, a topology for the vertices, and optionally an index buffer to allow for arbitrary vertex sharing between primitives, or an implicit adjacency relationship with explicit breaks.
• A geometry as a topology + vertex layout properties. Geometry shading updates both vertex topology and properties, hence it is modifying the entire geometry. Technically, geometry does not include pixel color, which is more of a skin around the geometry, but we can throw pixel color into a geometry abstraction for completeness, although a term like object might be more appropriate to describe geometry + skin.

At the end of the day, only three properties need to be explicitly specified during shading to accommodate rendering: a geometry as a topology of vertices to form primitives, a position for each vertex, and a color for each pixel in each primitive.

What we are looking for in a geometry abstraction is to allow a single value g that can by itself undergo rendering (e.g., render(g)) where the pixel, vertex, and geometry shader can all be inferred by the composition of g. As a value, g explicitly encodes everything needed for shader generation and is formed through pure functional programming techniques that include composition and transformation. To meet the rendering requirement, g not only carries around a formula that defines vertex topology, but also the formula for each vertex’s position and the formula for each pixel’s color. These formulas in turn can depend on secondary properties that are subject to transformations that are applied to the geometry or state defined outside of the geometry; e.g., a slider’s thumb position.

As described in the last post, we can define an abstraction for geometry that supports composition, duplication, and transformation. Ideally, rendering could then involve forming a geometry in a clean functional way through multiple compositions and transformations, and passing the resulting geometry into a render command. Transformations not only include modifying the layout of the geometry by rotating, scaling, and translating it, but also applying color and lighting to the geometry and the pixels contained within it, or whatever else is required for a complete rendering specification. Lighting could even be applied to the constituent parts of a geometry before they are composed. The various properties needed to make this possible include thinks like diffuse, specular, glass, and refractive materials as well as additional non-geometry constituents such as directional, ambient, and spot lights. Essentially, the geometry would then become a mini-scene graph.

Scene graphs are common in retained graphics APIs such as WPF 3D and Java 3D. Basically, a scene graph is a graph of the elements that affect scene rendering, and then becomes the basis for what is shown on the screen. Since we are only interested in what can be efficiently rendered in a shader pipeline, we have to keep the graph nodes mostly homogeneous: duplicating and transforming a geometry in the graph is fine, but composing completely different geometries with different lighting schemes is probably not going to work in the context of one rendering call (instead, render the geometries in separate rendering calls).

The primary difference then with my previous geometry abstraction is that this new geometry embeds properties and transformations at every level of composition. A level in a geometric composition can omit or duplicate properties from a higher level, and transformations (e.g., lighting or layout) should input a property from the lowest level that it exists in the geometry that the transformation is attached to; e.g., a normal property at the pixel level is preferred from a normal property at the vertex level as it is more accurate for lighting computations. The problem with this approach is that we can lose static typing: a property might not exist at any level, then what? Right now I’m willing to live with dynamic checking since it will occur relatively early in Bling when code is generated at application startup.

That’s it for now. Geometry shading will fall out of transformations that cannot be resolved statically. I’ll talk about this in my next post.

Bling does a good job of abstracting vertices and pixels: in Bling shader code one value represents both the vertex and pixel, where the vertex position and pixel color are specified with respect to this value. Bling infers whether the code is referring to a vertex or pixel based on a simple dependency analysis: we are referring to a pixel whenever we refer to anything that is only available in the pixel shader, otherwise we are referring to a vertex and the computation result (if used in the pixel shader) will be interpolated based on the analogous results in the primitive’s other vertices. However, Bling’s vertex/pixel abstraction breaks down when dealing with geometry: the input of a a rendering is simply the vertex count and primitive data and Bling has no high-level geometry abstraction. Bling’s lack of a geometry abstraction prevents geometries from being manipulated, composed, and constructed at a high-level, and prevents geometry shaders from being expressed at all unless shader partitioning is expressed explicitly. And this is a pity because geometry shaders can do so many cool things such as dynamically forming fur, hair, feathers, sparkles, blobs, and so on. We definitely need to make manipulating geometries easier.

So here is my new realization: give geometries their own abstraction in Bling, defined as one of the following:

• The atomic case is a a collection of adjacent primitives. Actually, the only things that need to be specified for this primitive geometry is the number of vertices, vertex adjacency (if more than one primitive is defined), and primitive topology. Vertex properties such as positions and normals are not included as they will be inferred via dependency analysis within the shader. Examples: the face of a cube (four vertices to define two triangles) or the result of sampling a parametric surface such as a sphere.
• A composition of two different geometries (gn = g0 + g1). The geometries composed must be of the same primitive topology. Example: we could compose a face with a four simple triangles to form a pyramid.
• A duplication of the same geometry N times (gn = N * g0). We define this separately from geometry composition as it could correspond (but might not depending on efficiency) to geometry instancing in the GPU API. For duplication to be meaningful, each duplicated geometry will always undergo a transformation based on the duplication index. Example: a face is duplicated four times to form a cube, or duplicating a cube N times to do some crude voxel rendering.
• A transformation of one geometry’s properties (gn = f(g0)). This is a bit difficult to define since there is no explicit common set of vertex properties, and indeed a vertex might not need a normal or even a position if it is not going to be rendered. However, the most common transformations will be on the position and normal, as the geometry is translated, scaled, and rotated. Example: rotating a face in 6 different ways to represent each face of a cube.

The benefit of this approach is immediately apparent: depending on our need we can define geometries mathematically or load them from a mesh. As properties aren’t included explicitly in the geometry, we can mix and match separate geometries as long as they share the same topology. We can then synthesize the index buffer (if needed) automatically, removing this burden from the user, and infer where geometry instancing occurs.

Unfortunately, vertex properties are more difficult to specify. Before, each vertex had one vertex ID and an optional instance ID, which we would use to compute the vertex’s position and pixel’s color. These could then be used as indexes into a table (in the case of a mesh) to lookup initial values for properties like position and normal, or they could be used in more rich computations; e.g., computing a per-pixel normal from the derivative of a parametric surface. At any rate, we inferred what properties to include in the vertex buffer in a way that allowed an expressed computation to span the CPU and GPU, which is a very good feature from an optimization and reuse perspective.  Now if we allow geometries to be composed, duplicated, and transformed, vertex addressing obviously becomes much more difficult when inferring properties. There are few directions that we could go in to solve this problem:

1. We could assign an ID to a vertex based on the order that it was defined in a composition, but often vertex properties are defined relative to their geometries, and such a scheme would basically ruin reuse of code of that computes properties for a certain composed geometry.
2. We could come up with some hierarchical Id scheme. However, vertices are generally thrown all together (in the vertex shader at least), and as such it would be very difficult to reason about vertices whose IDs are not homogenous.
3. The other obvious option is to embed properties in geometries, which would solve the geometry reuse problem but then would tie geometries down to a common set of properties that they had to share in order to be combined. Yet this might be reasonable if the set of per-vertex properties are relatively fixed and it was easy to deal with exceptions easily. Whatever properties are declared do not necessarily correspond to what properties go in the vertex buffer; e.g., the position and normal properties of a sampled parametric surface can refer to dynamic CPU-only values (such as a slider’s position), where inference would synthesize the vertex buffer accordingly to include only static values. I’m still thinking about this problem but I guess I’m working in the direction of somehow re-introducing per-vertex properties, although I might have to throw away dynamic typing to make it work in C#’s type system (hopefully, generics will be enough).

Any solution that I come with has to play friendly with geometry shaders, which have the ability transform geometries based on dynamic information. I’ll get into this with my next post.

Double3Bl m = -q.XYZ().Square;
m = m.Sw<Y,X,X>() + m.Sw<Z,Z,Y>();

Double3Bl n = q.W * q.XYZ();
Double3Bl r = q.Sw<X,X,Y>() * q.Sw<Y,Z,Z>();
Double3Bl t = r.Sw<X,Z,Y>() - n.Sw<Z,X,Y>();
Double3Bl u = n.Sw<Y,Z,X>() + r.Sw<Y,X,Z>();
Double3Bl p = argB;
Double3Bl v = m * p.XYZ() + t * p.YZX() + u * p.ZXY();
return 2d * v + p.XYZ();

Sometimes we use extension methods (e.g., XYZ()), and sometimes we use the Sw method, which is slightly more verbose but still reasonably concise. For type safety, we define the X, Y, Z, and W classes so that X <: Y <: Z <: W, then if the target is a 4 vector, the type parameter bounds for all the swizzle parameters is W , while if it’s a 2 vector, the type parameter bounds for all the swizzle parameters is Y, disallowing Z and W. The X, Y, Z, and W classes can then be instantiated as type parameters to determine the index they represent.

Long description/example for those that are interested:

In Bling, 3D programmers are expressed by the programmer in pure C# code rather than HLSL code—at run-time the C#-encoded shaders are dynamically translated into HLSL code that is then compiled into DirectX 10 shaders. Shader bodies can directly refer to dependency properties, which are translated into uniform slots (aka global parameters) that are maintained via efficient dynamically generated CLR code (via the DLR). A simple complete example that uses a camera but no lighting:

CameraZ = new LabelSliderBl(this) { LabelName = "Z", Minimum = -40d, Maximum = +10d, Value = -4d, };
RotateX = new LabelSliderBl(this) { LabelName = "RotateX", Minimum = 0d, Maximum = 2d, Value = .9d, };
RotateY = new LabelSliderBl(this) { LabelName = "RotateY", Minimum = 0d, Maximum = 2d, Value = .9d, };
CubeTopology Cube = new CubeTopology();
Point3DBl WorldCenter = new Point3DBl(0, 0, 0);
Point3DBl CameraPosition = WorldCenter + new Point3DBl(0, 0, CameraZ.Value);
PerspectiveCameraCl Camera = new PerspectiveCameraCl() { Position = CameraPosition, };
ColorBl[] FaceColors = new ColorBl[] {
 Colors.Red, Colors.LightGreen,
 Colors.Blue, Colors.Yellow,
 Colors.Purple, Colors.Salmon,
};
Device.Render(Cube.Count, 16, (IntBl n, IntBl m, IVertex vertex) => {
 QuaternionBl qx = new QuaternionBl(new Point3DBl(1, 0, 0), RotateX.Value.PI());
 QuaternionBl qy = new QuaternionBl(new Point3DBl(0, 1, 0), RotateY.Value.PI());
 PointBl mxy = new PointBl(m % 4, m / 4);
 mxy = (mxy / 4d) - (.5 + .25) / 2d;
 vertex.Position =
(Camera * mxy.InsertZ(0).Translate() * .1.Bl().XXX().Scale() * (qx * qy).Matrix).Transform(Cube[n]);
 vertex.Color = FaceColors.Table(n / 4);
}, Cube.IndexBuffer);

The example defines 3 sliders (Z, RotateX, RotateY) with Slider-like Value dependency properties, these properties are then referred to in the Device.Render call to define the position of each vertex in a cube (defined through a utility cube topology class), where Z drives the perspective camera’s Z position while RotateX and RotateY are used to define rotations around the X and Y axis respectively. The cube is instanced 16 times, where n is the vertex ID and m is the instance ID. As with other shader-based programming models, all the logic for rendering is specified directly in the shader; e.g., the effects of the camera must be multiplied into the position computation.

This approach has full support for WPF dependency property capabilities, including WPF-based animation. Partitioning shader computations between pixel and vertex shaders is also inferred, with the principle that computation occurs in the vertex shader  as possible unless the programmer specifies otherwise. For example, consider the following code with a spotlight where all lighting computations are done in the vertex shader:

var Slider = new LabelSliderBl(this) {
 Minimum = -.1, Maximum = +.1, Value = 0, LabelName = "Light",
};
LightBl light = new SpotLightBl() {
 Direction = (new Point3DBl(Slider.Value, Slider.Value, +1)).Normalize,
 Color = Colors.White,
 Position = new Point3DBl(0, 0, -2.5),
 InnerConeAngle = 2.ToDegrees(),
 OuterConeAngle = 4.ToDegrees(),
};
MaterialCl diffuse = new DiffuseMaterialCl() {
Factor = .5,
Ambient = { Factor = .5 }
};
Device.Render(Cube.Count, DIM * DIM, (IntBl n, IntBl m, IVertex vertex) => {
Point3DBl WorldPos;
… /* ellided position computations */
 Vertex.Color = diffuse.ApplyLights(light + new AmbientLightBl() { Color = Colors.White }).
 ApplyAt(WorldPos, normal, CameraPosition).ForColor(clr);
}, Cube.IndexBuffer);

Result:

For better quality, we can force the lighting computation down into the pixel shader by specifying that world positions are computed on a per-pixel basis:

Device.Render(Cube.Count, DIM * DIM, (IntBl n, IntBl m, IVertex vertex) => {
Point3DBl WorldPos;
… /* ellided position computations */
 Vertex.Color = diffuse.ApplyLights(light + new AmbientLightBl() { Color = Colors.White }).
 ApplyAt(WorldPos.PerPixel(), normal, CameraPosition).ForColor(clr);
}, Cube.IndexBuffer);

The result is an improvement in quality if you look closely:

Other nice features:

· An explicit signature mode where uniforms, vertex, pixel, and geometry shaders are specified directly as annotated C# interfaces. This mode gives the user more control without requirement too much boiler plate, and is also the only way we can currently express DX10 geometry shading.

· Render target textures, ability to specify multiple render out passes, e.g., for simulating heat diffusion:

· Camera, light, and material model based on WPF 3D, with enhancements (e.g., a glass material).

· Meta-programming of shader trees to do things like automatically compute the derivative of an expression or automatically compute the upper bound of vertices that could be produced by a geometry shader.

· Lots of graphic utility classes to ease the expression of shaders and hide the math; e.g., one line of code to normal map a texture onto a parametric surface.

· DX10/WPF airspace issues addressed by overlay window.

Caveats:

· Right now Bling is only meant for prototypes and experiments, and not meant for production use; e.g., no DX resource management has been implemented. Bling is an effective way of rapidly exploring design ideas, which are then produced using a more established/involved technology (e.g., Direct3D).

· No XP support (DX10), textures won’t work on pre-SP3 Vista.

· Even though Bling’s retained model is based on WPF dependency properties, there is no support for anything like XAML.

· No way to host WPF content in DX10.

A download of Bling 3.1 is available on Codeplex.

The process of abstraction (verb) is that of hiding details so that one can focus on fewer concepts at a time. An abstraction (noun) then is a hider of details. A programming language provides various primitives for constructing custom abstractions, such as procedures, classes, functions, properties, or rules—let’s refer to these as abstraction primitives to distinguish them from computation primitives, which are used to build up computations that use or are hidden by abstractions. Using abstraction primitives, a library then define and implement an API as a set of abstractions that permit access to some kind of service. Of course, real life is often more complex: one or more languages can be involved in the definition of a library (e.g., WPF and C#, Visual Basic, XAML, and HLSL), while libraries will often be built from or to interoperate with abstractions from other libraries. 

Libraries are increasingly defined as frameworks that provide heavy large-grained abstractions for assembling systems in specific domains; for example consider a web framework (Ruby on Rails) or an IDE framework (Eclipse). Programmers customize a framework’s abstractions from the top-down according to their needs and assemble them together according to a rigid architecture. The trade off of using a frameworks includes the learning curve of the framework’s abstractions and architecture, as well as a sacrifice in flexibility where the framework becomes difficult or impossible to program outside of its intended purpose.

I am going to refer to framework abstractions as fixed abstractions because they are analogous to abstractions used to program a GPU through a fixed-function graphics pipeline as opposed to the abstractions used to program a GPU through a programmable pipeline. In the past, GPUs encoded in hardware only various fixed functions to manipulate vertices, light pixels, and so on. To program the GPU, the programmer manipulated various abstractions that represent fixed GPU functions through an API such as DirectX or OpenGL. As with a framework, these fixed-function APIs allowed for some customization of abstractions from the top-down, come with rigid architectures, and sacrifice flexibility in what they can express.

For the reason of flexibility alone, fixed-function pipelines are being rapidly replaced programmable pipelines that have the ability to express arbitrary custom effects through small programs known as shaders, which run directly on the GPU. The programmable pipeline completely supersedes the fixed-function pipeline.  For example, whereas a specific kind of light (e.g., spotlight) and material (e.g., specular) are fixed abstractions that would be customized and installed in a fixed-function pipeline, now it is merely sufficient to take the position of the vertex or pixel being shaded and return a color! At first glance, such an API seems a lot harder to use in simple cases where well non-custom understood lighting is being used, but then such cases can rely on reusable procedures that package up the lighting math. Beyond dramatically increased flexibility, the programmable pipeline approach also has a better learning curve as their are only a few lightweight abstractions to learn with virtually no architecture. For an excellent write up what the move from fixed-function to programmable pipelines means, see Tim Sweeney’s Ars interview on the subject.

What do programmable pipelines and shaders have to do with software libraries and frameworks? They hint at an alternative to frameworks: rather design a library with many large-grained abstractions (fixed functions) that programmers customize down to what they want, instead we could design a library with a few small-grained versatile abstractions (shaders) that programmers then build up to what they want. We refer to the latter as programmable abstractions, which are smaller and more generally used than fixed abstractions. Large-grained reuse then comes from assembling the programmable abstractions together and packaging them up through language-level abstraction primitives (e.g., procedures). Programmable abstractions are not a new concept: they underpin many libraries, especially those built in functional programming languages. Additionally, programmable abstractions can be promoted to language-level abstractions with domain-specific languages (e.g., pipes in shell script), where the key to defining a good DSL is to base it on versatile programmable abstractions rather than fixed abstractions. I feel that a term for such abstractions is needed because of the confusion surrounding languages, frameworks, libraries, DSLs, and something in between (highly versatile libraries that are not quite DSLs).

I’m not exactly sure what the threshold between a programmable abstraction and a fixed abstraction is, it might not be very clear cut. Also, the benefits of programmable abstractions are negated if excessive amounts of boilerplate are needed to use them, languages can go a long way in eliminating this boilerplate and making programmable abstractions more feasible (e.g., see express tree lifting in LINQ). To be continued, in the next part I’ll provide some concrete examples of differences between fixed and programmable abstractions.

New features in Bling 3.0:

• Improved support for multiple forms in lighting in pixel shaders reusing WPF3D abstractions when possible. Now a lighting effect can be added to a widget by specifying a WPF3D light, a material (similar to but not a WPF3D material), a normal map and/or a height map loaded from a texture (some kinds of lights require an eye position). Lighting effects can be combined via addition. WPF shader examples:

1. Normal mapped diffuse material + red specular material point light effect.
2. Glass material + point light effect (also normal mapped).
3. Diffraction material + point light effect (again normal mapped).
4. Diffuse material + point light effect with parametric normals (computed vs. loaded from an image).

• Pixel-level transition abstractions that support composition. Example:

Intermediate progress of a ripple + sin transition composed via Min (take the min color from each transition’s result, where start and end are the same).

• Hacked prototype-quality WPF-friendly web browser that supports transformations and pixel shader effects. Browser will refresh every 100 milliseconds, so flash animations will work if a bit choppy. Link clicking is supported, but not keyboard input (need more interop code). Example:

Paper normal mapped browser navigated to Bling’s codeplex page.

• Simplified physics engine (more examples in the future).
• Initial experimental support for DirectX 10, focusing right now on parametric surfaces. DirectX compositions look very much like Bling WPF pixel shaders and can refer directly to WPF dependency properties (just like a WPF pixel shader can). Example:

1. A sphere, no distortion
2. A sphere + a egg crate distortion
3. (2) with more magnitude
4. (3) with more frequency
5. (4) with less magnitude + the WPF sliders that are controlling it all.

Caveats:

• Bling 3 is not backwards compatible with the previous version of Bling. Existing code requires porting.

ToDo:

• Work on DirectX abstractions (right now we just have a start).
• Would like to support Silverlight 3.

We (myself, my intern lighting-export Li SiYu, and a former intern Wang Chao who worked on the browser part) started with code from Chris Cavanagh’s Blog to render a background IE instance into a WPF writeable bitmap, which we optimized using some sample code from David Teitlebaum. The advantage of this approach is that a writeable bitmap is a first class entity in WPF and so can be transformed and distorted, whereas a standard WPF web browser doesn’t support these features due to airspace issues. The disadvantage of our approach is that its a big hack suitable only for playing around (e.g., in the above pic) and we can only update the browser about once every 100 milliseconds, so videos and Flash are extremely choppy (but still very lively!).

The web browser content is projected onto a diffuse material that is lit by a point light. We bind the point light’s position to the mouse so the lighting changes dynamically with user interaction. The height of the light is bound to the slider beneath the address bar. This code sets up the lighting:

ThumbBl LightXY = new ThumbBl();
Action UpdateLightXY = PointBl.Assign(LightXY, canvas.MousePosition);
this.MouseMove += (xx, yy) => UpdateLightXY();
Lighting lighting = new DiffuseMaterialCl().Apply(new PointLightBl() {
  Position = new Point3DBl(LightXY, HeightOfLight),
  Color = Colors.White,
});

This code mostly involves databinding, except for tracking the mouse position, which we do manually. The “Apply” method of material exercises basic lighting equations according to the specified light (diffuse material + point light in our case) to create a lighting result. The lighting code for a diffuse material is encoded in Bling is something like this:

Point3DBl Dir = LightSource.DirectionToPosition(Position);
DoubleBl Dis = Dir.Length;
Dir = Dir.Normalize;
LightingResult = ((LightSource.Color.ScRGB * Dir.Dot(Normal).Max(0) *
  LightSource.UseAttenuation(Dir, Dis))) * Input.ScRGB;

Lights are distinguished by their direction (constant for a directional light, difference between position being shaded and light position for point light) and their attenuation (a bit more math).

We apply the lighting result to the canvas containing the browser via normal mapping, which uses pre-baked texture to add depth and naturalness to the surface being lit. The textures used here make the browser look like it is printed on slightly folded paper! The texture scrolls as the scrollbar is used to scroll browser content, though not at the same magnitude. Code:

Texture HeightField = new ImageBrushBl(HeightImage);
Texture NormalMap = new ImageBrushBl(NormalImage);
HeightField = HeightField.Distort(uv => new PointBl(uv.X, (uv.Y + scrollBar.Value).Frac));
NormalMap =     NormalMap.Distort(uv => new PointBl(uv.X, (uv.Y + scrollBar.Value).Frac));
canvas.Effect.Custom = lighting.Apply((HeightField)HeightField, (NormalMap)NormalMap,
                              canvas.Size.AddZ(0.015d), canvas.CenterSize.AddZ(100d));

Textures in WPF are brushes, so we load our normal mapping files (HeightImage and NormalImage) as image brushes. We distort the resulting textures by shifting them vertically by the scrollbar’s value, which are then converted into proper texture coordinates via Frac (an HLSL intrinsic defined as Math.Abs(Math.Round(n))). We then explicitly coerce the two textures into a height field and normal map respectively, for use in normal mapping. These explicit coercions convert the textures according to a popular normal map and height field texture representations (Crazy Bump and various Photoshop normal mapping plugins). The other two arguments used for normal mapping are the 3D size of the canvas and the eye position we want to use. The code for normal mapping is as follows:

public PixelEffect Apply(HeightField Heights, NormalMap Normals, Point3DBl Size, Point3DBl Eye) {
  return new PixelEffect(input => new Texture(UV => {
    Point3DBl Position = new Point3DBl(UV * Size.XY(), Heights[UV] * Size.Z);
    var eyeDir = (Eye - Position).Normalize;
    var UV0 = UV + eyeDir.XY() * Heights[UV] * Size.Z;
    return this[Position, Normals[UV0], Eye][input, UV0];
  }));
}

A pixel effect is simply a texture-to-texture function that is suitable for use as a pixel shader. A texture is defined as a point-to-color function that can be loaded from a file (as we did with HeightField and NormalMap) but can also be defined programmatically, which is what we are doing here. The math for the resulting texture is beyond the scope of this blog post but is not very difficult: basically we extract heights and normals from the height field and normal map according to the current texture coordinate UV, adjust them for the 3D size of our target, and bias UV by using an eye direction and the current height.  This is then plugged into our lighting result, which in turn is used to compute the resulting pixel from the input texture and the biased texture coordinate. Video:

This effect will work for any WPF application, not just our browser, notice the text field at the top, the slider below that, and the scrollbar tot he side, which are all WPF widgets!

The code for this sample is available in the Bling 3 distribution, which will hopefully be released soon!

The drawback of such programmable abstractions is convenience and repetition—no one wants to reinvent the wheel while inventing an effectively round wheel requires complex processes. Consider implementing a custom layout algorithm in WPF, boilerplate make expressing programmable layouts difficult and repetitive: one must define a new panel, define minimum, maximum, and preferred sizes, and deal manually with layout changing events. Later on, this panel can only be reused in its entirety: parts of the algorithm must be re-implemented in a new panel. Likewise, boilerplate in Direct3D makes it difficult to setup even simple scenes, while boilerplate to interface with shaders prevents the modularization of lighting math that can run either on the CPU or GPU. The solution to the boilerplate problem is to eliminate boilerplate. For example in expressing layouts, convenient and concise databinding to arbitrary expressions makes it practical to encode arbitrary layout constraints that automatically react to dynamic changes. Likewise, the direct expression of shader code regardless of whether on the CPU or GPU allows for the modularization of lighting code into simple functions. This brings us to the second principle of Bling’s design: avoid inconvenience and repetition through boilerplate-hiding abstractions such as those concerning databinding or function composition. Code can then simply be organized into methods or functions.

Other Bling principles:

• To reduce confusion and keep the API as small and simple as possible, avoid redundant abstractions. For example, Bling does not support WPF storyboards because they overlap with databinding and explicit clocks to synchronize animations. Likewise, styles and triggers are easily replaced with constraints expressed via databinding.
• Focus on math. Almost everything in UI is math heavy, be it layout, animation, gradient blending, lighting, or physics.  Therefore it makes a lot of sense to optimize our abstractions for the expression and modularization of math. Bling includes an extensive math library, support for composite tuple, matrix, and function operations, simple algebraic solving, and symbolic derivation.
• Programmer friendliness through reasonable defaults. The days when a rectangle does not show up on the screen because the programmer forgot to set a stroke or color should be over! For programmer friendliness, Bling strives to ensure that all properties have default values, which then cause something noticeable to happen on the screen.
• Avoid abstraction performance penalty through dynamic code generation. Layers of abstraction can be expensive in performance critical parts of a UI, such as in layout, lighting, animation, or physics. This expense can be eliminated by using abstraction-heavy code to generate unabstracted code at run-time. Furthermore, code can be generated to run on GPUs as well as CPUs, leading to even greater performance improvements while isolating programmers from the otherwise inconveniences of GPU programming.
• To support code generation and symbolic analysis, manipulate expression trees, not values. Bling is like LINQ in that it is based on manipulating expression trees. Various tricks such as the use of operator overloading allows expression tree compositions to resemble comparable operations on values; e.g., a + b can form an addition tree node with a and b sub-trees.

One solution is to encode graphic techniques in a high-level functional programming language, where functions allow us to effectively represent, combine, and manipulate graphic techniques. Take for example lighting computations that can be represented as a function Position3D –> Normal3D –> InputColor –> ResultColor, which can then be combined via addition: take two lighting computations l1 and l2, where a succinct l1 + l2 expands to p –> n –> c –> l1(p)(n)(c) + l2(p)(n)(c). As graphics is math-intensive, it benefits dramatically from function representations.

Unfortunately, their are a couple of drawbacks to this approach. First, the execution of a functional program is drastically slower than imperative code. However, as most graphics computations are moving to GPUs via memory-restricted shader code, this problem can be solved by generating shader code from functional programs. Such is the approach taken by Conal Elliott’s Haskell-hosted Vertigo library done at MSR. In Vertigo, geometries can be expressed as parametric surfaces, surface normals for lighting are computed via symbolic differentiation. The second drawback is more one of familiarity: the techniques seem to require abandoning our existing mainstream languages and tools for dramatically different and less familiar languages such as Haskell. However, while functional programming definitely benefits from functional languages, functional programming techniques can definitely be applied in more familiar languages. As a mainstream language, C# even provides a lambda construct to support programmers who want to dabble in functional programming; e.g., by using LINQ to query databases.

I’m implementing a functional graphics library in Bling for C#. Bling supports the creation and composition of expression trees as seemingly normal C# expressions, allowing us to easily compose code in a high level language and then ship the result to the GPU as a vertex or pixel shader. This allows us, among other things, to support a style of programming very similar to how Vertigo is used in Haskell. For example, consider the definition of a sphere parametric surface in Bling:

public static readonly PSurface Sphere = new PSurface((PointBl p) => {
  PointBl sin = p.SinU;
  PointBl cos = p.CosU;
  return new Point3DBl(sin[0] * cos[1], sin[0] * sin[1], cos[0]);
});

PSurface is a wrapper around a function from PointBl to Point3DBl, which are wrapper types around point (R2) and 3D point (R3) expression trees. SinU and CosU define scaled sine and cosine values over points between [0,0] and [1,1] by using the coordinates as percentage for 2PI angles. Rendering a sphere in Bling is simply a matter of creating some light and defining how many vertices we want to sample in the resulting vertex buffer:

PSurface Surface = Geometries.Sphere;
DirectionalLightBl dirLight = new DirectionalLightBl() {
  Direction = new Point3DBl(0, 0, 1),
  Color = Colors.White,
};
ParametricKeys SurfaceKeys = new ParametricKeys(200, 200);
Device.Render(SurfaceKeys, (IntBl n, PointBl uv, IVertex vertex) => {
  vertex.Position = (Surface * (world * view * project))[uv]; ;
  Point3DBl norm = (Point3DBl)(Surface.Normals[uv] * world).Normalize;
  Point3DBl usePosition = (Point3DBl)((Surface[uv] * (world)));
  vertex.Color = dirLight.Diffuse()[usePosition, norm](n.SelectColor());
});

where world, view, and project are standard 4×4 matrices used in translating position in 3D scenes. This code creates a directional light coming from the bottom (reused from WPF 3D no less!) and defines a set of surface keys over 200 by 200 vertices, meaning 40,000 vertices are sampled in the rendered sphere. The light is applied to the sphere in a render function using a world transformed position of each vertex along with a world transformed normal that is automatically computed via the derivative of the parametric surface. The color for each vertex is selected based on the vertex index, leading to a nice spiral pattern in the sphere that also gives us an idea of vertex topology. Here is the result:

As described in the Vertigo paper, interesting geometries can also be formed from surfaces via displacement by height fields. Copying the examples in the vertigo paper, consider an eggcrate height field that whose definition in Bling has a simple structure similar to sphere:

public static readonly HeightField EggCrate = new HeightField((PointBl p) => {
  p = p.SinCosU;
  return p.X * p.Y;
});

The displacement method on surfaces is then defined as follows:

public PSurface Displace(Func<PointBl, DoubleBl> HeightField) {
  return new PSurface((PointBl p) =>
    this[p] + Normals[p] * HeightField(p));
}

As with lighting computations, surface normals are used to determine the proper direction in which to apply the height field. We can then apply the egg crate height field to a sphere as follows:

PSurface Surface = Geometries.Sphere.
  Displace((Geometries.EggCrate.Frequency(20d).Magnitude(sliderZ.Value * .5)));

The frequency and magnitude of a height field are adjusted via methods that have one line implementations (again, this is described in the Vertigo paper). One twist is that we control the magnitude by a slider (Z), which is a standard WPF slider with a value dependency property. Now, by moving the slider up, the sphere becomes deformed via the egg crate height field:

Moving the slider even farther up, the sphere becomes spiky:

The change in deformation occurs in real time, so we see a smooth animation as the sphere becomes deformed.

Bling will automatically determine what code executes outside of the shader in the form of either uniforms (per-shader external variables) or vertex buffers (per-vertex data). Bling will also determine what code runs in pixel shaders vs. vertex shaders, and will automatically generate vertex input and output structures to accommodate communication between the two kinds of shaders. Bling will also generate code to handle update uniforms and vertex buffers during rendering, requiring absolutely no boilerplate code on behalf of the programmer. Instead, the programmer can just focus on what graphics techniques they want to use, without needing to worry about adapting them to fit on the underlying graphics architecture. For the above example, here is the HLSL code that Bling generates, which is then rendered using Direct3D 10:

Since the code is auto-generated, it is not very pretty, but it doesn’t have to be. I still need to work on the conversion process to minimize the number of vertex parameters passed in as input, as these are limited to 16 and can easily be exhausted when more external values (e.g., sliders) are added to parameterize geometry and lighting.

The code for this example is currently available in our codeplex SVN repository at https://bling.svn.codeplex.com/svn/Bling3, and I’m currently working on cleaning things up for a new release of Bling (titled Bling 3) that will support rich UI construction in WPF and preliminary support for DirectX 10 (sorry XP users!) as described in this post.