Matt's Homepage

I just spent an hour or two tracking down the last 16-byte memory leak in a project that I was working on. What it turned out to be was actually something EXPLICITLY cited in the Microsoft reference manual as having been fixed.  Yet it turned out not to be.

I had finished instrumenting my program for CRT memory reporting. CRT memory leak reporting is normally very handy as it tells you exactly what lines of code are the source of the leak, how many bytes, and dumps the data that’s leaking. All it requires is that you instrument your headers as described here <http://msdn.microsoft.com/en-us/library/x98tx3cf.aspx>

// this goes before ANY other includes in ALL your headers
#define _CRTDBG_MAP_ALLOC
#include <blah>
#include <blah>
...
 main()
{
    _CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF ); // tells leak detector to dump report at any program exit
    _CrtSetBreakAlloc(18); // set breakpoint on a specific allocation number
}

With all this done you can run the program and then see the results of the report in the Visual studio output window when the program exits:

Detected memory leaks!
Dumping objects ->
C:PROGRAM FILESVISUAL STUDIOMyProjectsleaktestleaktest.cpp(20) : {18}
normal block at 0x00780E80, 64 bytes long.
Data: < > CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD
Object dump complete.

If you use the _CrtSetBreakAlloc() with the allocation number ({18}in this case), when the program starts up in debug mode the debugger will automatically stop on that allocation. You can then look at the call stack and get a great idea of whats leaking and why. Overall, it’s very handy and helped me track down all my leaks.

Well, the problem was that I fixed all the leaks but 2 in my particular program.  I reduced the code down until I was down to just main() that dumped the memory report and exited.   The problem was that it STILL reported a leak! I added a fake leak, and the very first allocation (that was leaking) in the program had a higher allocation number than the allocations that were supposedly leaking.  This implied quite strongly that the leak is happening BEFORE I actually even get into the program. How is that possible?

Well, there are a few things that get allocated for you before your program starts. Fixed-sized arrays, (int array[10];) string literals, and static variables are all objects are allocated on the stack (as opposed to the heap) are usually allocated at load time. When I looked at all the static allocations in my program, I saw this:

class Bar
{
    ...
private:
    static std::string m_name;
    static std::string m_value;
}

Do you see any problem?  I didn’t – at first. The problem is that std::string is a STL object that has a constructor.  Statics are usually allocated at load time. Which means their constructors are likely being called during load time BEFORE the main() is hit. We assume that the corresponding destructors are called after main exits. Apparently, the CRT memory tracking starts right as the program starts but is supposed to ignore any CRT allocations (ones made by the standard C libraries).  Apparently these static allocations by STL objects are captured.  Since we report leaks BEFORE we hit the program exit, we haven’t actually hit those ‘hidden’ destructors for those static strings yet.  Those allocations are still reported as active – and hence ‘leaked’.  I only caught this because I’d initialized those strings with string literals.  The leak dumped the memory and I recognized the strings. So a good way to test if this is happening to you is set your object data to something unique and make sure those unique values appear in the leak report memory dump.

Yet another clue is that the value of an uninitialized string consists of a number that appears to be a memory address (while the addresses change each run, other allocations in that run have roughly the same addresses), and a bunch of zeros. This likely corresponds to a pointer to the character buffer, and an int that indicates size. This is further confirmed by the fact the leaks are 8 bytes each in x86 build, and 16 bytes on a x64 build. This corresponds to the sizeof(int)+sizeof(void*) sizes on the respective builds.

Solution:
The solution is to ignore what is (likely) a false report, or change the static std::string objects to non-static or static std::string* pointers. Then make an appropriate initializer/destructor that creates/destroys those string objects. This means the memory is allocated/deallocated on the heap in the scope of the program run. The other thing you do is file a bug with Microsoft and put a few comments on the documentation website that indicates the statics are still leaking and they haven’t really fixed it yet. 🙂

Links:
MS article about CRT memory debug operations that says this bug had been fixed (but it has not as of VS2012)

http://msdn.microsoft.com/en-us/library/x98tx3cf.aspx

So, I am working on something that requires I programmatically know what’s in a DX11 shader file.  One of the cool things you can use to figure out how many constant buffers, get string names from position streams, etc – is the DirectX Shader Reflection system.  It gives you the ability to query a loaded shader blob for names/types/etc.  However, it’s not quite as straightforward as one might expect to use.  The docs were pretty incomplete up until the DX11 version.  But here’s the basics of the ID3D11ShaderReflection system:

1. Load your vertex/pixel shaders.  Get the actual shader blob – or extract it from the FX system if you used that.
2. Bind it to a shader reflection object using D3DReflect:
   

   pd3dDevice->CreatePixelShader( pPixelShaderBuffer->GetBufferPointer(), pPixelShaderBuffer->GetBufferSize(), g_pPSClassLinkage, &g_pPixelShader );
   ID3D11ShaderReflection* pReflector = NULL;
   D3DReflect( pPixelShaderBuffer->GetBufferPointer(), pPixelShaderBuffer->GetBufferSize(), IID_ID3D11ShaderReflection, (void**) &pReflector);

3. Query the reflection object for whatever constant buffer, input/output stream, or other info you want using the many methods available.  It’s a GREAT way to test your shaders at load time so you don’t get cryptic runtime errors later when you try to actually draw objects.

Now, the gotcha’s:

1. You must also include this:

   #include <D3DCompiler.h>

   Or you’ll get a compile error with D3DReflect() – despite the fact the official Microsoft docs seem to say you should (only) include:

   #include <D3DShader.h>

2. Finally, you MUST have the DirectX SDK include directory listed before the windows include directory in the Visual Studio compiler include directory list or you’ll get compile errors in D3DShader.h. This is apparently because the windows headers that come with Visual Studio actually have a few DX definitions in them that conflict with the DX SDK header definitions.  
   In Visual Studio, check: Tools/Options -> Project and Solutions -> VC++ Directories and make sure Windows SDK include & library paths appear AFTER DirectX include & library paths.  See here for the thread.

From the wayback machine…

I found a half-completed post on when I was first learning to program on the iPhone.  I think these things are all still valid…
So, I’ve embarked on an ambitious programming project.  One that is now 1/3 of the way complete (first third was the ‘heaviest lifting’ part of the project – took about a month of outsourced grunt work).  The next 1/3 involves programming an app on the iPhone/iTouch.  I thought that would be the easy part, but as it turns out – there is a substantial learning curve that one should be aware of before starting this endeavor.  Here are some of the learnings I’ve found so far after having just got the bare bones of my own app up and running.

1. Apple’s documentation is very robust – but frustratingly ‘useless’ at times.  They provide many very detailed and excellent vertical stacks of information.  Want to learn about pooled allocators?  They have a document that gives you all the information you could ever want.  But not in the context of how you’d actually use them – or why you hit various problems you hit.   You’ll find yourself sifting around from document to document trying to find out why your NSString object is throwing an exception when returning from a function – only to find the real reason is buried in the memory management section of the Cocoa programming guide – not anywhere near the NSString documentation or their samples. 
2. You need to learn Objective C and Cocoa. – and learn them in THAT order.  It’s annoying, but try to find a good Objective C book first.  There aren’t many of them.  There are lots of bad ones though.  Learn how to write a class, add functions with multiple calling parameters, how to make calls to classes, and how allocation works.  That last one is key.  Really spend time learning why:
   [[NSString alloc] init]
   is the right way to allocate and initialize an object and then learn why:
   NSString *s = [[NSString alloc] initFromString:@”hello”];
   [s appendString:@”this string now leaks”];
   leaks memory.  And how the autoAllocator pools work/don’t work.  This is all huge and the first biggest gotcha’s your get hit with right off the bat.
3. You’ll struggle to get even a 5 line program working at first.  Your first Objective C/Cocoa programs will be very painful.  The simplest things will feel like fighting a brick wall.  I tried to do a simple enum, and kept getting burned by an invalid class definition.  Wha?  The objective C compiler on the iPhone requires you typedef your enums and structs:
   enum newEnum { a, b, c, d }; (NOPE – will cause a compiler error in whatever line follows this definition)
   typedef enum { a, b, c, d } newEnum; (yes – works)
   Just millions of little things like that.  Again, logical if you put your old C/gcc hat on, but it’s been a bit since I last had that hat on and it had some dust on it…
4. The GUI IDE is great – if you do things they way THEY want them and only for straight-forward designs.  The interfaces for GUI controls holds very much like Win32/X programming philosophies, albeit the syntax is very different.  X Code comes with it’s own WYSIWYG GUI editor, but I found it very limited if you’re going to come up with any kind of innovative interface.  If you just want some standard buttons/etc – you’re probably fine.  But if you want to have some interesting scrolling effects/etc – you best be ready to spend another few days learning how the UI control systems work underneath and experimenting.  The built-in GUI editor reminds me of the old VS2005 GUI editor that you could drag-n-drop controls, compile, and you’d see the results.  If you want any dynamic elements – I haven’t found a way to do that using their gui editor.  And if you want to embed controls on scrolling panels and some other more modern effects – you’ll be tossing the GUI editor altogether.  Trouble is, you spend a good bit of time learning how the built-in GUI system works, find out it won’t do the thing you need without tons of digging through forums and ‘tricking’ the thing to do what you want, then give up and have to learn a bunch of NEW stuff when you give up and decide to just dynamically/hand-generate the controls yourself.  Frustrating.