Matt's Homepage

One of the key mathematical foundations of machine learning is using gradient descent to find maxima and minima in a multi-dimensional data set. Gradient descent is good, but getting the most out of it can sometimes leave you wringing your hands or doing a lot of painful mathematical investigation and analysis. Investigations that can quickly tax even a mathematics major.

Sergui Puscas shows us a different, more intuitive way to find maxima and minima by using swarming and flocking techniques. It’s a pretty fun read.

Rendering to offscreen surfaces is a key component to any graphics pipeline. Post-processing effects, deferred rendering, and newer global illumination strategies all use it. Unfortunately, implementing offscreen rendering on OpenGL ES is not well documented. OpenGL ES is often used on embedded/mobile devices, and until recently, these devices haven’t typically had the graphics bandwidth to keep up with new rendering techniques. Compound this with the fact that many mobile games have simple gameplay/small screens that do not need such complex lighting models, many people now use off the shelf engines for their games, and that there is still a good amount of mobile hardware out there that doesn’t even support render to offscreen surfaces, and it is no surprise that few people use the technique and it’s not well discussed.

In implementing offscreen rendering for OpenGL ES, I turned to the very good OpenGL ES Programming book as it has a whole chapter on framebuffer objects. When I tried the samples in the book, however, I was having a lot of difficulty getting it working on my linux-based mobile device. A lot of the implementation examples use a technique of creating framebuffer objects using textures, but you can also use framebuffer objects via something called render buffers. One reason this is good to know is because many hardware vendors support very few render-to-texture formats. You can often find yourself struggling with your implementation not working because the output formats aren’t supported.

Thankfully, I found this article and thought I’d copy the information here since it’s the only place I’ve seen working code that demonstrated the technique. It also includes the very important step of reading the output format and uses glReadPixels() so you can validate that you were writing correctly to the offscreen renderbuffer surface.

In my case, on an Intel graphics part, I found that the format (which is also the most recommended one) that worked  was GL_RGB/GL_UNSIGNED_SHORT_5_6_5. Steps 1-8 is standard OpenGL ES setup code that is included so you can verify your setup. Step 9 is where the glFrameBuffer and glRenderBuffer objects are created.

    #define CONTEXT_ES20

    #ifdef CONTEXT_ES20
        EGLint ai32ContextAttribs[] = { EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE };
    #endif

    // Step 1 - Get the default display.
    EGLDisplay eglDisplay = eglGetDisplay((EGLNativeDisplayType)0);

    // Step 2 - Initialize EGL.
    eglInitialize(eglDisplay, 0, 0);

    #ifdef CONTEXT_ES20
    // Step 3 - Make OpenGL ES the current API.
    eglBindAPI(EGL_OPENGL_ES_API);

    // Step 4 - Specify the required configuration attributes.
    EGLint pi32ConfigAttribs[5];
    pi32ConfigAttribs[0] = EGL_SURFACE_TYPE;
    pi32ConfigAttribs[1] = EGL_WINDOW_BIT;
    pi32ConfigAttribs[2] = EGL_RENDERABLE_TYPE;
    pi32ConfigAttribs[3] = EGL_OPENGL_ES2_BIT;
    pi32ConfigAttribs[4] = EGL_NONE;
    #else
    EGLint pi32ConfigAttribs[3];
    pi32ConfigAttribs[0] = EGL_SURFACE_TYPE;
    pi32ConfigAttribs[1] = EGL_WINDOW_BIT;
    pi32ConfigAttribs[2] = EGL_NONE;
    #endif

    // Step 5 - Find a config that matches all requirements.
    int iConfigs;
    EGLConfig eglConfig;
    eglChooseConfig(eglDisplay, pi32ConfigAttribs, &eglConfig, 1,
                                                    &iConfigs);

    if (iConfigs != 1) {
        printf("Error: eglChooseConfig(): config not found.n");
        exit(-1);
    }

    // Step 6 - Create a surface to draw to.
    EGLSurface eglSurface;
    eglSurface = eglCreateWindowSurface(eglDisplay, eglConfig,
                                  (EGLNativeWindowType)NULL, NULL);

    // Step 7 - Create a context.
    EGLContext eglContext;
    #ifdef CONTEXT_ES20
        eglContext = eglCreateContext(eglDisplay, eglConfig, NULL,
                                               ai32ContextAttribs);
    #else
        eglContext = eglCreateContext(eglDisplay, eglConfig, NULL, NULL);
    #endif

    // Step 8 - Bind the context to the current thread
    eglMakeCurrent(eglDisplay, eglSurface, eglSurface, eglContext);
    // end of standard gl context setup

    // Step 9 - create framebuffer object
    GLuint fboId = 0;
    GLuint renderBufferWidth = 1280;
    GLuint renderBufferHeight = 720;

    // create a framebuffer object
    glGenFramebuffers(1, &fboId);
    glBindFramebuffer(GL_FRAMEBUFFER, fboId);

    // create a texture object
    // note that this is commented out/not used in this case but is
    // included for completeness/as example
    /*  GLuint textureId;
     glGenTextures(1, &textureId);
     glBindTexture(GL_TEXTURE_2D, textureId);
     glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
     glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);                             
     //GL_LINEAR_MIPMAP_LINEAR
     glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
     glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
     glTexParameteri(GL_TEXTURE_2D, GL_GENERATE_MIPMAP_HINT, GL_TRUE); // automatic mipmap
     glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, renderBufferWidth, renderBufferHeight, 0,
                  GL_RGB, GL_UNSIGNED_BYTE, 0);
     glBindTexture(GL_TEXTURE_2D, 0);
     // attach the texture to FBO color attachment point
     glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
                         GL_TEXTURE_2D, textureId, 0);
     */
     qDebug() << glGetError();
     GLuint renderBuffer;
     glGenRenderbuffers(1, &renderBuffer);
     glBindRenderbuffer(GL_RENDERBUFFER, renderBuffer);
     qDebug() << glGetError();
     glRenderbufferStorage(GL_RENDERBUFFER,
                           GL_RGB565,
                           renderBufferWidth,
                           renderBufferHeight);
     qDebug() << glGetError();
     glFramebufferRenderbuffer(GL_FRAMEBUFFER,
                               GL_COLOR_ATTACHMENT0,
                               GL_RENDERBUFFER,
                               renderBuffer);

      qDebug() << glGetError();
      GLuint depthRenderbuffer;
      glGenRenderbuffers(1, &depthRenderbuffer);
      glBindRenderbuffer(GL_RENDERBUFFER, depthRenderbuffer);
      glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_COMPONENT16,     renderBufferWidth, renderBufferHeight);
      glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, depthRenderbuffer);

      // check FBO status
      GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
      if(status != GL_FRAMEBUFFER_COMPLETE) {
          printf("Problem with OpenGL framebuffer after specifying color render buffer: n%xn", status);
      } else {
          printf("FBO creation succeddedn");
  }

  // check the output format
  // This is critical to knowing what surface format just got created
  // ES only supports 5-6-5 and other limited formats and the driver
  // might have picked another format
  GLint format = 0, type = 0;
  glGetIntegerv(GL_IMPLEMENTATION_COLOR_READ_FORMAT, &format);
  glGetIntegerv(GL_IMPLEMENTATION_COLOR_READ_TYPE, &type);

  // clear the offscreen buffer
  glClearColor(1.0,0.0,1.0,1.0);
  glClear(GL_COLOR_BUFFER_BIT);

  // commit the clear to the offscreen surface
  eglSwapBuffers(eglDisplay, eglSurface);

  // You should put your own calculation code here based on format/type
  // you discovered above
  int size = 2 * renderBufferHeight * renderBufferWidth;
  unsigned char *data = new unsigned char[size];
  printf("size %d", size);

  // in my case, I got back a buffer that was RGB565
  glReadPixels(0,0,renderBufferWidth,renderBufferHeight,GL_RGB, GL_RGB565, data);

  // Check output buffer to make sure you cleared it properly.
  // In 5-6-5 format, clearing to clearcolor=(1, 0, 1, 1)
  // you get 1111100000011111b = 0xF81F in hex
  if( (data[0] != 0x1F) || (data[1] != 0xF8))
      printf("Error rendering to offscreen buffern");

  QImage image(data2, renderBufferWidth,  renderBufferHeight,renderBufferWidth*2, QImage::Format_RGB16);
  image.save("result.png");

PIX is a performance tuning and debugging tool for game developers – that hadn’t been updated in years for the desktop. It survived on in three generations of Xbox consoles, but there was no desktop love. No longer! Microsoft just announced PIX beta is now available for analyzing DirectX 12 games on Windows.

PIX on Windows provides five main modes of operation:

• GPU captures for debugging and analyzing the performance of Direct3D 12 graphics rendering.
• Timing captures for understanding the performance and threading of all CPU and GPU work carried out by your game.
• Function Summary captures accumulate information about how long each function runs for and how often each is called.
• Callgraph captures trace the execution of a single function.
• Memory Allocation captures provide insight into the memory allocations made by your game.

Go to the Microsoft blog to download it for free.

My first homework assignment for my self-driving automotive class was to find lanes on still, then on captured video. Here was my first attempt, which seems to have come out pretty well. I still want to improve it a bit with some more frame-to-frame smoothing.

Useful links/techniques:

• Hough Lines
  • OpenCV description of Hough lines
  • Example of using OpenCV Hough lines
  • Another OpenCV Hough line example
• Least Squares method to find a line of best fit through points
• Python
  • Setting up PyCharm to use OpenCV that you installed elsewhere (in our case Anaconda). Basically, you set the search path
  • Alpha blending OpenCV images
  • Sorting tuples

In doing some robustness testing, I needed to crash my Linux VM intentionally to see if the rest of the system survived. I was thinking of writing a kernel module that just did something silly like dereference null or divide by zero, but turns out you can do it much easier than that. Just call panic.

panic.c:
#define __NO_VERSION__
#include <linux/version.h>
#include 
#include 

int init_module(void)
{
    panic(" insert lame excuse here");
    return 0;
}

Build with:

 gcc -I/usr/src/linux/include -D__KERNEL__ -DMODULE -o panic.o -c panic.c

Here’s a good link about building kernel modules without full kernel source.

Now, run the following insmod, and your system will kernel panic.

 insmod panic.o 

Credit to Paul’s Journal

Writing the OpenGL context creation plumbing on Linux isn’t 100% intuitive. It’s always good to have a handy getting started guide. With developers increasingly using engines such as Unity, the number of people writing directly to OpenGL is shrinking – and so are some of the online resources to get over this initial step. I decided to collect a few links and copy critical sample code here for when I need to whip something up quickly.

Some links to good ‘hello world’ Linux GL samples:
https://github.com/jckarter/hello-gl
https://www.opengl.org/wiki/Tutorial:_OpenGL_3.0_Context_Creation_(GLX)
http://www-f9.ijs.si/~matevz/docs/007-2392-003/sgi_html/ch04.html

Code:
Taken from: https://www.opengl.org/sdk/docs/man2/xhtml/glXIntro.xml

Below is a minimal example of creating an RGBA-format X window that’s compatible with OpenGL using GLX 1.3 commands. The window is cleared to yellow when the program runs. The program does minimal error checking; all return values should be checked.

#include < stdio.h >
#include < stdlib.h >
#include < GL/gl.h >
#include < GL/glx.h >

int singleBufferAttributess[] = {
    GLX_DRAWABLE_TYPE, GLX_WINDOW_BIT,
    GLX_RENDER_TYPE,   GLX_RGBA_BIT,
    GLX_RED_SIZE,      1,   /* Request a single buffered color buffer */
    GLX_GREEN_SIZE,    1,   /* with the maximum number of color bits  */
    GLX_BLUE_SIZE,     1,   /* for each component                     */
    None
};

int doubleBufferAttributes[] = {
    GLX_DRAWABLE_TYPE, GLX_WINDOW_BIT,
    GLX_RENDER_TYPE,   GLX_RGBA_BIT,
    GLX_DOUBLEBUFFER,  True,  /* Request a double-buffered color buffer with */
    GLX_RED_SIZE,      1,     /* the maximum number of bits per component    */
    GLX_GREEN_SIZE,    1, 
    GLX_BLUE_SIZE,     1,
    None
};


static Bool WaitForNotify( Display *dpy, XEvent *event, XPointer arg ) {
    return (event->type == MapNotify) && (event->xmap.window == (Window) arg);
}
int main( int argc, char *argv[] )
{
    Display              *dpy;
    Window                xWin;
    XEvent                event;
    XVisualInfo          *vInfo;
    XSetWindowAttributes  swa;
    GLXFBConfig          *fbConfigs;
    GLXContext            context;
    GLXWindow             glxWin;
    int                   swaMask;
    int                   numReturned;
    int                   swapFlag = True;

    /* Open a connection to the X server */
    dpy = XOpenDisplay( NULL );
    if ( dpy == NULL ) {
        printf( "Unable to open a connection to the X servern" );
        exit( EXIT_FAILURE );
    }

    /* Request a suitable framebuffer configuration - try for a double 
    ** buffered configuration first */
    fbConfigs = glXChooseFBConfig( dpy, DefaultScreen(dpy),
                                   doubleBufferAttributes, &numReturned );

    if ( fbConfigs == NULL ) {  /* no double buffered configs available */
      fbConfigs = glXChooseFBConfig( dpy, DefaultScreen(dpy),
                                     singleBufferAttributess, &numReturned );
      swapFlag = False;
    }

    /* Create an X colormap and window with a visual matching the first
    ** returned framebuffer config */
    vInfo = glXGetVisualFromFBConfig( dpy, fbConfigs[0] );

    swa.border_pixel = 0;
    swa.event_mask = StructureNotifyMask;
    swa.colormap = XCreateColormap( dpy, RootWindow(dpy, vInfo->screen),
                                    vInfo->visual, AllocNone );

    swaMask = CWBorderPixel | CWColormap | CWEventMask;

    xWin = XCreateWindow( dpy, RootWindow(dpy, vInfo->screen), 0, 0, 256, 256,
                          0, vInfo->depth, InputOutput, vInfo->visual,
                          swaMask, &swa );

    /* Create a GLX context for OpenGL rendering */
    context = glXCreateNewContext( dpy, fbConfigs[0], GLX_RGBA_TYPE,
				 NULL, True );

    /* Create a GLX window to associate the frame buffer configuration
    ** with the created X window */
    glxWin = glXCreateWindow( dpy, fbConfigs[0], xWin, NULL );
    
    /* Map the window to the screen, and wait for it to appear */
    XMapWindow( dpy, xWin );
    XIfEvent( dpy, &event, WaitForNotify, (XPointer) xWin );

    /* Bind the GLX context to the Window */
    glXMakeContextCurrent( dpy, glxWin, glxWin, context );

    /* OpenGL rendering ... */
    glClearColor( 1.0, 1.0, 0.0, 1.0 );
    glClear( GL_COLOR_BUFFER_BIT );

    glFlush();
    
    if ( swapFlag )
        glXSwapBuffers( dpy, glxWin );

    sleep( 10 );
    exit( EXIT_SUCCESS );
}