Book Review: The Hunger Games

Book Review: The Hunger Games

Time for audio book review #2. I finished the audiobook version of ‘The Hunger Games’.  The first of a trilogy by Suzanne Collins.

undefined

It’s a futuristic tale told in the dystopian future country Panem; built in the remains of North America.  Years ago, the 12 districts of North America fought with the Capital, and lost.  As part of their surrender terms, each district must send one boy and one girl to appear in the annual Hunger Games – a televised fight to the death of 24 contestants until only 1 remains as victor.  Katniss is from the poor, coal producing district 12 (which is likely old Appalachia by its description). Through a turn of events, Katniss is elected as this year’s representative for the games.  She’s a scrappy, tom-boyish girl who knows how to bow hunt (it’s illegal, but it’s the only way to keep her family fed after their father died in the mines).  The rest of the story is how the bloody games, and relationship with fellow tribute Peeta (a baker’s son that showed her kindness many years ago), progress over the days in the wilderness where the competition is set.

First off, this book was written for young adults.  So don’t expect a lot of adult themes and a little more on the ‘simple’ side of word/dialogs/story development.  I know if I’d read this as a kid, I probably would have loved it; and I wouldn’t say I was at all disappointed as an adult.  It held my interest for the 9 hours it took to get through the audiobook version.

The story itself follows much in the lines with such stories as The Running Man or Deathrace 2000.  We’re going to follow Katniss through her brutal and poor life in her beaten district.  Her selection, the crazy lead-up to the games in which she’s dressed up, paraded around, interviewed on television, and treated something like royalty.  This is all to gain ‘sponsors’ who can buy expensive gifts to help particular contestants during the actual battle.  Then in they go, and the latter half of the book is all about what they do in the piece of wilderness controlled by the game makers to maximize excitement over the days as the battles unfold.  None of the battles (save 1 which I’m wondering how they’ll show on the movie) are overly bloody or brutal – but there is definitely violence – toned to a young-adult level of course.  You won’t have people attacking each other with chainsaws; but death does abound.

Overall, it was an imaginative book worth the 9 hours of listening – or about 2 weeks of commuting in audiobook time. 🙂  I have to say I was pleasantly uplifted by this book after having read The Golden Compass series – which left a very poor taste in my mouth for ‘modern’ young-adult sci-fi literature.  There is still plenty of good social commentary (poverty and police-states of the districts, the brutality and dehumanizing treatment of the less fortunate by the rich and bored, children being forced into adult roles of supporting and raising families as well as turned into killers, etc), but none of it is so preachy it becomes a theme in itself.  It’s the story is what moves the plot forward, and it’s a good story.  In fact, in Collin’s descriptions of the things Katniss  sees in the capital, I even got a few ideas for software projects I’d like to try out/experiment with.  Not bad sci-fi.  There were some times when I wanted to put a boot in the rear of Katniss’ character as she waffles between being a hard-as-nails competitor, and second-triple-and-quadruple guessing herself as to other people’s motivations.  Some of the ways the fights turn out are a touch unbelievable (she gets a strange reprieve from death at one point that left me pondering for a long time before I could even feel ok with it) and the author does give her some cheats/lucky breaks; but nothing so egregious as to disrupt the story. It ends on a bit of a hanging note; which is fine considering it’s part 1 of a trilogy. But there were a few themes like Katniss’ small shows of defiance to the Capitol’s games/rule that are started in this book, but don’t go too far.  I’m hoping those themes are picked up in the next books, as I’ve already reserved the audio versions of books 2 and 3 from the library.

A final, interesting observation.  While I was ‘reading’ through the book, there were some definite themes that I don’t think male sci-fi writers would have put in, or spent as much time on.  In fact, the whole bits during the run-up to the games themselves were the most revealing to me.  Suzanne Collins spends longer with themes like the dresses Katniss wears, the waxing/hair-plucking/’beautification’, the clothing designers and interview show, and the confused feelings/trying to figure out the motivations of others that Katniss has were noticeably different than things I’ve read by other male sci-fi writers.  It’s doubtful someone like Ray Bradbury would have spent as much time with those details/parts.  While they didn’t particularly change the story IMHO, they were just a noticeable…difference of tone that made me appreciate how many sci-fi stories are by male writers.

Fortuitous timing, as the movie is just about to be released.  So far, trailers seem to show it’s a near-exact transfer of the book.  Which is a refreshing change of Hollywood’s usual pace of b*stardizing most good books/stories/childhood toys. While some of the production values look a little low, I know I’ll watch it at some point; and get the next two books in this trilogy.

Overall, I give the book a solid B for being a good story.

Name of the Wind – Patrick Rothfuss

Name of the Wind – Patrick Rothfuss

Been on an audio-book kick most of this year.  I’ve got a 30 min commute each way to/from work each day – so why not get some awesome audio books in?  I’ll start up with reviews of the ones I’ve listened to thus far.

Finished listening to The Name of the Wind by Patrick Rothfuss about 2-3 months ago.  It’s a pretty hefty ‘read’ – coming in at 22 hours (672 pages) of listening.  But, considering I get about an hour of listening per day, it only took about a month of commuting.

Why try to re-invent the reviews on Amazon?  I just pulled the quotes I saw that sound like what I read:

Set in an unnamed imaginary world, Kvothe (“pronounced nearly the same as ‘Quothe’ “), the hero and villain of a thousand tales, is presumed dead but actually now lives as the simple proprietor of the Waystone Inn. Prompted by a biographer who realizes his true identity, Kvothe starts to tell his life story. This is the riveting first-person narrative of Kvothe, a young man who grows to be one of the most notorious magicians his world has ever seen. From his childhood in a troupe of traveling players, to years spent as a near-feral orphan in a crime-riddled city, to his daringly brazen yet successful bid to enter a legendary school of magic, The Name of the Wind is a masterpiece that transports readers into the body and mind of a wizard. Kvothe is driven by twin imperatives—his desire to learn the higher magic of naming and his need to discover as much as possible about the Chandrian, the demons of legend who murdered his family.

That is just about the best description I could come up with.  As for the storytelling, there are a few patterns of Rothfuss’s writing who’s repetition begins to make you feel like he got a bit lazy (like the fact this kid has the absolute WORST luck in the same way several times – which always end with him out of money).  He also gives Kvothe some lucky breaks and coincidences that feel a bit like cheats, but those are very minor points.  Overall, there’s very little to gripe about.   It’s certainly not as epic and grand a tale as Tolken; with grand battles and deep theological / philosophical underpinnings and imagery.  You won’t walk away from the story feeling like you want to ponder again and again the deeper meanings, but its originality and story make up for any loss of the gradious.  No, this is the story of a kid growing up under crazy circumstances and struggles his way through in a really unique world.

The coolest point has to to be one of the most amazing and well-thought out descriptions of how a wizard/magician of the D&D/fantasy sort works internally.  The descriptions, realism, and imaginative mechanics of how magic is done as Kvothe re-tells his stories is truly captivating and original.  None of this ‘he waves his wand and poof’.  Rothfuss has thought out a definite system of how the ‘magic’ happens – and that alone is enough to make this worth reading this book.  In fact, I was told this was ney-on required reading for any dungeon master or DM wizard character.

Overall, I give this a solid A-.  It’s not a sweeping epic – but it’s originality and story will keep you to the end.  I took a breather with a different genre of book after this one, but have already started his second book in the series if that gives you any clue to it’s goodness.

Halloween!

Halloween!

Had some great fun giving candy out to the kiddies in the neighborhood.  Also put up my usual window display – which got rave reviews!

Why _CrtDumpMemoryLeaks can make you think you’re leaking memory – but you’re not

Why _CrtDumpMemoryLeaks can make you think you’re leaking memory – but you’re not

I just spent an hour or two tracking down the last 16-byte memory leak in a project that I was working on. What it turned out to be was actually something EXPLICITLY cited in the Microsoft reference manual as having been fixed.  Yet it turned out not to be.

I had finished instrumenting my program for CRT memory reporting. CRT memory leak reporting is normally very handy as it tells you exactly what lines of code are the source of the leak, how many bytes, and dumps the data that’s leaking. All it requires is that you instrument your headers as described here <http://msdn.microsoft.com/en-us/library/x98tx3cf.aspx>

// this goes before ANY other includes in ALL your headers
#define _CRTDBG_MAP_ALLOC
#include <blah>
#include <blah>
...
 main()
{
    _CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF ); // tells leak detector to dump report at any program exit
    _CrtSetBreakAlloc(18); // set breakpoint on a specific allocation number
}

With all this done you can run the program and then see the results of the report in the Visual studio output window when the program exits:

Detected memory leaks!
Dumping objects ->
C:PROGRAM FILESVISUAL STUDIOMyProjectsleaktestleaktest.cpp(20) : {18}
normal block at 0x00780E80, 64 bytes long.
Data: < > CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD
Object dump complete.

If you use the _CrtSetBreakAlloc() with the allocation number ({18}in this case), when the program starts up in debug mode the debugger will automatically stop on that allocation. You can then look at the call stack and get a great idea of whats leaking and why. Overall, it’s very handy and helped me track down all my leaks.

Well, the problem was that I fixed all the leaks but 2 in my particular program.  I reduced the code down until I was down to just main() that dumped the memory report and exited.   The problem was that it STILL reported a leak! I added a fake leak, and the very first allocation (that was leaking) in the program had a higher allocation number than the allocations that were supposedly leaking.  This implied quite strongly that the leak is happening BEFORE I actually even get into the program. How is that possible?

Well, there are a few things that get allocated for you before your program starts. Fixed-sized arrays, (int array[10];) string literals, and static variables are all objects are allocated on the stack (as opposed to the heap) are usually allocated at load time. When I looked at all the static allocations in my program, I saw this:

class Bar
{
    ...
private:
    static std::string m_name;
    static std::string m_value;
}

Do you see any problem?  I didn’t – at first. The problem is that std::string is a STL object that has a constructor.  Statics are usually allocated at load time. Which means their constructors are likely being called during load time BEFORE the main() is hit. We assume that the corresponding destructors are called after main exits. Apparently, the CRT memory tracking starts right as the program starts but is supposed to ignore any CRT allocations (ones made by the standard C libraries).  Apparently these static allocations by STL objects are captured.  Since we report leaks BEFORE we hit the program exit, we haven’t actually hit those ‘hidden’ destructors for those static strings yet.  Those allocations are still reported as active – and hence ‘leaked’.  I only caught this because I’d initialized those strings with string literals.  The leak dumped the memory and I recognized the strings. So a good way to test if this is happening to you is set your object data to something unique and make sure those unique values appear in the leak report memory dump.

Yet another clue is that the value of an uninitialized string consists of a number that appears to be a memory address (while the addresses change each run, other allocations in that run have roughly the same addresses), and a bunch of zeros. This likely corresponds to a pointer to the character buffer, and an int that indicates size. This is further confirmed by the fact the leaks are 8 bytes each in x86 build, and 16 bytes on a x64 build. This corresponds to the sizeof(int)+sizeof(void*) sizes on the respective builds.

Solution:
The solution is to ignore what is (likely) a false report, or change the static std::string objects to non-static or static std::string* pointers. Then make an appropriate initializer/destructor that creates/destroys those string objects. This means the memory is allocated/deallocated on the heap in the scope of the program run. The other thing you do is file a bug with Microsoft and put a few comments on the documentation website that indicates the statics are still leaking and they haven’t really fixed it yet. 🙂

Links:
MS article about CRT memory debug operations that says this bug had been fixed (but it has not as of VS2012)

http://msdn.microsoft.com/en-us/library/x98tx3cf.aspx

When the less-obvious solution is best

When the less-obvious solution is best

Sometimes more really isn’t the answer.

Case and point – dishes at your house.  I saw this picture and college days of yore flooded back:

At one bachelor flat I lived at – our sink often looked like this.  We had 4 guys living in one house, and dishes weren’t a high-priority item.  So, they tended to pile up and then nobody seemed to remember who dirtied what and an endless cycle of blame and avoidance resulted.

The solution: throw out all the extra dishes.  Instead, there was *exactly 1* set of silverware, plate, and glass for each person.  Further, each one was color-coded.  Red plate/cup/silverware was Bob’s.  Green was Fred’s, etc.  That meant if you wanted to eat something, then you had to have washed your stuff to do so.  It also meant if you left your dirty dishes around – the culprit was immediately identifiable.  The first benefit was that with only 4 plates in the house, a ‘pile’ of them never got very big.  The second was that very quickly the innocents were vindicated, and slobs were identified. (We still had a full set of ‘nice’ dishes for when people came over for a dinner – but they were in a separate cupboard.  Anyone seen using those individually would be berated furiously).   The system worked like a charm and there was nary an argument about dishes for the rest of the time I lived there.

I think this is a corollary to many a home-owner’s observation that your possessions naturally expand to fit the space you have.   Or maybe Apple’s design philosophy that sometimes its the limitations you put on the usage of the device that actually allow easier and higher functionality.

Great time at the beach

Great time at the beach

I hadn’t taken much vacation this year (3 days to be exact), and had been grinding on a lot of code the last few months.  So, I dug through our list of corporate discounts and found a great place on the Oregon coast I could stay at for cheap.  So, I took 2 days off and went to Pacific City, OR and got this great view from my room every night.

I feel I got quite lucky because the weather at the coast isn’t usually very clear in the fall – but I got two near-perfect days.  Guess someone was looking out for me.  🙂

Siggraph 2011 Trip Report – Vancouver, BC

Siggraph 2011 Trip Report – Vancouver, BC

Summary: 
Very good trip, but not a huge ‘groundbreaking’ year of new technologies.

  • Courses were really good and had a lot of good information from top-notch game developers/houses.  Most of the courses included a review/update on ‘what is the current state of the art’ which is good for those who aren’t following those niches closely on a regular basis.
  • Voxels and voxel methods are kind of back in rage again – but for things like visibility, ambient occlusion, and AI pathfinding.
  • Lots of industry leaders questioning direction and disparate energies trying to figure out where the market and graphics are going.  API developers and HW vendors desperately trying to find out what the next platform of choice will be (laptop/mobile phone/pad/etc) so they can be positioned to capitalize on it when it becomes clear.  Nobody seems to know.
  • MLAA was the star tech of the show as far as people showing up with implementations and praise of it.
  • Many talks stressed that fact fast content creation and rapid artist pipelines are *the* driver of technologies and tools; and most coding/engine requirements are subservient to the needs of the content pipeline.
  • While the show floor was very much smaller than the California years, 3D printing and 3D capture was big.   You could get your face scanned at one booth and 3D print it at another one in just an hour or so.
  • Intel’s booth was really well attended with lots of interest in onloading techniques.  I had a number of good talks with developers – some of which wanted to know if we’d be bringing Larrabee back. J

 

Sunday:
Course
– Destruction and Dynamics for Film and Game Production
This all afternoon session started with a great overview of current destruction techniques.  There was a great and thorough overview of the current ways to break up a mesh (Voronoi, Boolean, FEM convex decomposition, and tetrahedral creation), the methods and problems of collision detection after the breakup, and constraint solving.  The focus now is not so much on the math of the constraint solvers, but on how to use them in intuitive and fast ways for the artists.

Takeaways:

  • Use the best freely/commercially already available solver for your purpose out there.  Constraint solvers are now very clever and (all things considered) computationally cheap. 
  • Don’t futz with the solvers to get what you want – use tons of constraints to get the effects you want.  Many movie models (like the earthquake scenes of the movie 2012) used literally thousands of constraints per building.  By using constraints, you can scale the effect up and get nearly the same look/feel as when you did it on the low-poly version.  Fiddling with the solver just gets you more and more headaches as low-poly pre-vis ones will collapse very different than high-poly versions and vice-versa.
  • Treat the solvers as black boxes and use different activation systems to achieve desired effects. It is easy and intuitive for artists to use these below methods and nobody needs to experiment with the super-fiddly parameters of the constraint solvers for each kind of breaking they want
    • Dynamic activation = when you break an impulse threshold, you break the constraint holding it in place
    • ‘Glue objects’ that simply hold sets of other constraints together to get ‘clumping’ effects. When the set has a higher impulse than the individual pieces
    • Cascaded activation = you don’t even consider objects as breakable until it becomes an edge piece or is touching an edge piece.   Gives nice ‘collapsing wall’ effects.

Monday:
Course – Advances in Real-Time Rendering – Parts I & II
http://advances.realtimerendering.com/

  • Went to an all-day course with presentations of new real-time techniques by such big-wigs as CryEngine3, Halo R&D team, Little Big Planet, Disney and many others.  It appears that voxel techniques are making a comeback for Halo and Little Big Planet guys.
  • Upcoming Halo title had a fascinating technique for occlusion via voxels.   They collaborated with Umbru software to develop a new kind of voxel-based visibility and portal system when their existing watershed algorithm started breaking down with the new outdoor content
    • Use a BSP tree of voxel areas to descritize the scene into a hierarchy of voxels and connect the polygon geometry with voxels.  Then use that unique structure to generate portals automatically, do fast runtime visibility, and AI pathfinding
    • They also changed their engine’s threading system to be job based instead of a more manual load-balancing that didn’t scale.  They found novel ways to avoid synchronizing the game engine between threads and dramatically reduce memory copies.
  • For their new Cars 2 game, Disney used a unique uniform-grid system to lay out light probes for their spherical harmonic lighting system. Lead to some great optimizations and easy authoring.
  •  Little Big Planet was using voxels for spherical harmonics and ambient occlusion lighting.  They also used a single particle system with really clever screen-space blending in Little Big Planet 2 to generate all their special water/explosion/fire effects.
  • Cryengine 3 guys talked about how they used previous frames with re-projection to get fast occlusion culling and local reflection.  They also used a lazy technique of updating cascaded shadow maps with further out maps getting updated much less often.
  • Frostbite used a rhombus shaped boken technique which has more passes, but is separable and therefore runable on multiple threads.  They also used cascaded maps for culling, not just shadowing.
  • Treyarch talked about techniques they used on Call of Duty Black ops.  They only allowed one dynamic light per surface, but unlimited lightmaps/lightgrids and environment probles per object to always maintain a 60fps goal but with great lighting effects.
  •  A talk by a new developer in ‘stealth mode’ who talked about Real-Time Image Quilting – which allowed realtime texture quilting without seams for expansive environments.
  •  The Gears of War 3 guys talked about overcoming lighting constraints on the PS3 by combining the light vectors into a single combined vector and doing tricks to extract it in the shader
  • Final talk was by a fellow who did super-accurate, highly realistic facial rendering with subsurface scattering.  His technique was an improvement on the nVidia one in many ways (speed/quality), but still not realitime.  Still, was interesting conversation about the difficulties of rendering realistic skin.

–         

Tuesday
Course- Beyond Programmable Shading I + II

All day course that talked about the direction and emerging graphics trends.  I alternated between this and looking at posters during the day.

  • Raja Koduri from Apple did a great talk on how power is managed in current devices.  He basically echoed what Intel has been saying: power isn’t scaling with Moore’s law – and how simple programming mistakes can save or kill a battery.  He talked about the need for dialog between hardware and software folks to develop API’s for power usage feedback and for current API’s that might need to be re-worked for low-power futures.
  • There was talk about allowing the actual scheduling methods used internally to graphics processors might need to be exposed in the future for greater programmability and what possibilities that might open.
  • There was a talk about how nVidia Optix SIMD-izes it’s ray tracing pipeline
  • Decent talk on current state of the art on bokeh, motion blur and DOF effects.
  • There was a good talk by Intel’s Marco Salvi on their OIT methodology along with other recent efforts.
  • At the end, there was a SUPER open-panel discussion with speakers about where DirectX/OpenGL/OpenCL were going.  The consensus is that there is no consensus.  Things are in a real flux right now with radically competing demands. Console devs want absolute control (open the gfx api command buffer to us!), mobile developers that want low-power/long life, desktop devs still wanting the best quality, and the uncertainty of the ‘platform of the future’ being really unknown (will we all be on iPads/consoles/laptops/etc).  The only thing they seem to agree on is that DirectX/OpenGL are basically still the best solutions we have unless we want to really bifurcate the API landscape – and it’s unclear whether we want to do that.  There’s a lot of opportunity for new technologies and new movers to make a big impact.

Poster sessions was pretty good with lots of entrants.  Nothing ground breaking, but there were several topics that I now want to look up. (don’t have notes with me today).

 

Wednesday:

  • Spent good part of the morning wandering the show floor, checking out some of the art and book displays, and seeing the show reels in the digital theater.  The afternoon was my turn in booth duty.
  • Show floor and hiring area was noticeably smaller than the California venue.  Big things this year: 3D printers were really big with lots of people doing super-cool live demos.  3D image capture was also big with some great facial/body capture systems.  Lots of 2-3 year computer art degree programs and schools were really pushing their schools.  I worry that there is going to be a glut of artists in about 3-4 years when these masses of graduates get out.

 

Thursday:

A grab bag day of talks as the all-day courses were mostly over.

Hiding Complexity (occlusion culling):

  • For Allen Wake’s almost all outdoor scenes, he used a method of culling shadows that looked from objects to their visible lights.  He’d then use scissor rects for each cascaded shadow layer to avoid filling pixels where they weren’t needed and not even render certain cascades if they were empty.  He also used temporal cues/temporal bounding boxes to avoid recalculating cascades if they didn’t change much
  • Latest Need for Speed – the classic SIMD tale.  They realized a 40x speedup by just throwing out their fancy data structures and simply SIMD-izing their culling system and arranging the data by the track layout.
  • Killzone 3 culling – many details, but basically they use a quick sw rasterizer distributed on many PS3 SPU’s to determine list of objects to draw on a 640x360p depth buffer in 16 pixel high strips.  In the visibility step, the assume all objects visible last frame are still visible, then test individual objects against the depth buffer in parallel.  Do simple sphere reject first, then bounding box test if sphere test passes.  For determining good occluders automatically, they use the low-poly physics mesh of the object.  They pick objects that have a fairly large bounding box, and who’s coverage area is near it’s bounding box area.  (actual coverage area is stored in meta-data).

Smokin’ Fluids:

  • Mostly non-realtime smoke simulation talks.  Best talk was on capturing thin features in smoke simulations. They used a moderately low voxel field with particles to simulate.  Then they did a clever pass over the data to ‘condition’ the voxel data to maintain superb fine details.  Turned out to be almost 10x faster than traditional method and didn’t suffer from the problems of simulations looking very different at higher and higher simulation densities.

Discrete differential geometry

  • Some pretty heady talks on discrete laplacians and circular arc structures.  Most of it was above my head – but was an interesting talk about how they could identify whether a curvy building could be made up of manufacturable, uniform pieces (like a geodesic dome)

Filtering Techniques for AA

  • Great talk on AA techniques which really turned into the ‘Why you should convert your game to MLAA today” cheering session.  Presentations started with different presentations on how people implemented MLAA in God of War III, on Xbox, and on PS3 for various games.
  • Good talk on FXAA and SRAA as alternatives from nVidia – but over half the room left by this point.
  • One guy from nVidia got up and said if they could give you MSAA or the like for nearly free if they’d convert back.  The speaker said, “No, I’d still keep MLAA on the CPU and then use the GPU for something else”.
Fixing more Battlefield 2 problems

Fixing more Battlefield 2 problems

Battlefield: Bad Company 2 on my PC was causing me MAJOR grief.  I’d get into a game, and about 5 minutes in, I’d get dropped and ‘connection to EA server lost’ message.  UTTERLY annoying.  Appears tons of other people have the same problems.  Various solutions from turning firewalls off/etc – but that didn’t work for me.

Changing your router’s UDP and TCP endpoint filtering to “Endpoint Independent” certainly helped – but it would still drop after about 10 minutes.

What seems to have worked is this.  Go to your steam client and:

Steam Library -> rightclick BC2 -> properties -> Set launch options… -> “-pingInterval 50000”
Paste the “-pingInterval 50000” phrase without the quotation marks and that’s it.

That seems to have solved the problem for me; but this was pretty much utter fail on EA/Dice’s part. The amount of hate-forum mail that’s gone on for over 2 years relating to this problem is amazing.

More Battlefield 2 adventures: PnkBsrtrB.exe handshaking failed

More Battlefield 2 adventures: PnkBsrtrB.exe handshaking failed

So, when playing the AIX mod for Battlefield 2, I kept getting kicked after about 2-3 minutes of play when punkbuster(!) complained that:

PnkBsrtrB.exe handshaking failed ; PnkBsrtrB.exe heartbeats stopped ; PB init failure ; Packet loss

The basic problem boiled down to the fact that when Punkbuster installed, it created the outgoing rules in windows firewall, but it did not create the INCOMING rules.  I simply added incoming rules for:

c:windowssysWOW32PnkBsrtrA.exe
c:windowssysWOW32PnkBsrtrB.exe

And set them to ‘allow’.  That turns out to fix the problem.

I still can’t believe how horrible the solutions to prevent cheating were back in the day.

Booting from a 4.0TB raid drive in Windows 7

Booting from a 4.0TB raid drive in Windows 7

Woot!  I just set up my system the last few nights with *three* 2TB drives all connected in RAID 5 setup into a single, 4.0 TB partition – AND I am using that huge bit bucket as my primary boot drive.  Now I have speed from the data striping and redundancy from the RAID.   It’s quite nice so far, but I’m still getting all my old software installed.   However, up until about a month ago – this wasn’t possible with the standard Intel storage controller that comes with almost all the new 2nd generation core systems (or with most RAID controllers).  But getting this magical land to work involves a lot of new information and a good bit of bleeding-edge fighting.

What you need to know:
First off, getting a RAID-ed boot partition larger than 2.2TB involves several pieces that need to work together.  If you only have the first 2, then you can still boot from a single-drive larger than 2.2TB, but you won’t be able to use it in a RAID setup with large partitions.

  1. A motherboard with EFI support
  2. An OS that can boot in EFI mode and supports GPT partitions
  3. A raid controller that can support creating and using boot partitions larger than 2.2TB.

First off, you need a motherboard with an EFI (Extensible Firmware Interface) bios.  EFI bios is the successor to the 20+ year old CMOS bios.  You might remember or have seen CMOS when that awful text screen came up when you hit the power and it would count memory or say ‘Press [del] to enter setup’.  Well, Intel with a lot of other companies finally, happily, heavenly, triumphantly are putting the nails in what had become a painful bit of computing legacy by throwing out CMOS and inventing EFI.  Apple has had EFI bios in their machines for a few years now, and some of the first-generation Core i3/i5/i7 motherboards had EFI support too – but it was spotty and very few things seemed to take advantage of it.  Which brings us to #2 – needing an OS that supports EFI booting and GPT.

If you have a motherboard that supports booting in EFI mode, you still need an OS that supports it and knows how to boot from partitions created with GPT (GUID Partitioning Table). I had an EFI board on my old system, but you couldn’t actually boot in EFI mode unless you used an early Linux distribution with EFI support or until about Windows Vista SP1/Win7 range.  Not only that, but EFI booting only worked on 64-bit versions of the Vista/Win7.  If you’d bought the 32-bit version – then you were out of luck.  The Vista route was also painful because unless you had an install disk that had the EFI boot files on it, you couldn’t actually install Vista in EFI mode.  Early versions of the Vista install CD’s (at least the one I had) didn’t have the EFI boot files.  I never learned if someone who had one of those early disks could return it and get an install disk with EFI boot support.  The reason you needed this is because if you couldn’t boot in EFI mode, you were stuck with the 2.2TB CMOS MBR boot limit.  If you weren’t in EFI mode during the install process, Vista/Win7 will refuse to create a boot partition larger than 2.2TB because it knew a MBR system couldn’t boot from it.  Unfortunately, you couldn’t override this in the install.  GPT booting was usually only supported when the BIOS booted in EFI mode.  So if your OS install disk couldn’t boot to EFI, or wasn’t able to make GPT boot partitions in CMOS mode, you were still out of luck.  As you’ll see, booting in EFI mode is actually more than just having a disk with the EFI files on it.  It’s a two-step approach, but those details will come later.

Finally, Windows 7 64-bit comes along and seems to solve our problems.   The OS install disk has EFI boot support on it.  The install disk is able to make bootable GPT partitions larger than 2.2tb.  Awesome.  But until this last month, you weren’t able to get your Intel RAID controller to make a bootable RAID set larger than 2.2tb because Intel hadn’t finished writing the EFI bios support for it.  You have to check the motherboard manufacturer’s website for a BIOS update – but you should now see new BIOS’s with the new large-partition RAID booting support.  That was the final key to the puzzle.

So, without further adieu, here is how one does this with the following equipment:

  • Asus P8P67 rev 3.1 motherboard (socket 1155 with 8 SATA ports and embedded Intel matrix storage controller) with BIOS flashed to version 1704 or higher
  • Three 2.0TB Hitachi drives all plugged into the Intel SATA ports
  • Windows 7 64-bit install DVD
  1. Back up everything
    The way I did it IS a destructive process and your drives will get erased.  There is no way that I know that you can migrate from a CMOS partitioning system to GPT.  There might be, but fiddling with this sort of stuff in a RAID setup is voodoo and you’ve been warned.
  2. Get and install the BIOS patch
    Be sure you are backed up at this point.  Asus bios flashing seems to always forget my RAID setup when I flash the bios.  In other words, poof – they’re gone.
    I went to Asus and found the newest BIOS patch. In my case it had the very clearly named:
    P8P67 (REV 3.1) 1704 BIOS – 2.2TB or larger HDD can be supported under RAID mode.
    I downloaded the patch file and put it on a USB key.  I then rebooted the machine and entered the BIOS setup.  Asus has a really nice built-in BIOS flashing utility with their new EFI BIOS.  I was able to point it at the USB key, find the image, flash the BIOS, and reboot. If the flashing forgot your RAID drives, then you’ll likely be greeted by a ‘no boot device found’ error.  You cannot safely re-create them if this happened.  The data on them is gone.
  3. Reboot your machine, and create your raid set
    With your BIOS patched, you should now be able to enter the Intel storage manager portion of the boot-up cycle via CTRL-I and create a large RAID set as bootable.  In my case, I selected: create a new set, selected the 3 drives I had plugged in, set the RAID configuration to RAID 5, 64k blocks, and made it bootable.  Save the changes and reboot.
    I find it extremely helpful and safer to shut off the machine, and physically unplug any extra drives you don’t want to get accidentally erased when manipulating RAID setups.  When you create the RAID set, you simply select the drives by serial number.  If you accidentally include a drive that (let say was your backup drive), the moment you add it to the RAID set – the data is gone. Be careful.
  4. Insert your EFI bootable Linux/Win7/Vista DVD, or USB key
    This is where you’ll need to consult your individual motherboard’s docs – and where things can get a little hairy.  For EFI booting on my ASUS board, I have to insert the EFI bootable media (in my case it was the Windows 7 x64 install disk), then reboot/turn the machine on.  Then I had to:

    1. press [DEL] to enter the CMOS setup during boot (with my Windows 7 disk in the drive)
    2. Go to the ‘boot’ menu in the CMOS
    3. Scroll down to list all the bootable devices (CD-Rom/Volume0 RAID set/etc)
    4. I saw the CD-Rom drive I wanted listed at the top, but that’s NOT the one you want.
    5. Keep scrolling down, and at the bottom you’ll see the device listed AGAIN but with the word EFI printed at the start of it.  THAT’S the one to choose.  It tells the system to boot from that device in EFI mode.  If you do NOT see your device listed a second time with the letters EFI in front of it, then that means the BIOS has not been able to find the key EFI boot files on the media you’re using.  In my case, my old Vista x64 disk wouldn’t allow me the option.  Why Asus doesn’t let you just manually set the mode to EFI only booting – I’ll never know.  I think it’s stupid they don’t let you, my old Intel DP35DP board let you do that…
    6. Select the device with the EFI bootable disk in it and tell the bios to boot.  You’ll notice that the fonts are different on startup, and that the cursor will do a funny indented thing during the boot cycle.  This tells you it’s in EFI booting mode.
  5. Boot the Windows install disk and create your partitions
    You should see the windows installer start just as normal.  When you get to the partitioning menu, you can select the auto-create option, and that should work.  However, this step is the MOST finicky and the place where you’ll find if you really are in EFI mode or not.  I chose the manual creation of the partitions and told it to make me the biggest one.  Sure enough, it said 3.7TB.  By default, you will get 2 other 100MB partitions created by windows for recovery purposes.  You can live without them, and they do not show up as drive letters in your system after it’s up.  Instructions on how to create your system without those extra partitions are here. The important thing is that you make SURE the partition is the right size at this point.  If it won’t create a 2.2+TB partition, or if it says it did, but the size is 2.2TB, then you’re not really in EFI mode or the Vista disk you have isn’t EFI bootable.  After you see that beautiful, full-sized partition – you might get a warning about the partition being potentially unbootable due to it’s size -but you can safely ignore it.  Just make sure that it actually created the partition.
  6. Finish the OS install as normal
    From here on out, follow the Windows install as normal.  On reboot, the motherboard will detect the GPT partition and properly boot in EFI mode automatically from here on out.  You’ll never need to do the selective EFI bios boot again.
  7. Install the Intel matrix storage manager software
    This really excellent software should be on your system.  If you have a UPS device on your system, I highly recommend turning the buffering on.  It allows for the drive read/writes to also be buffered in system memory.  It is dangerous without a UPS because a power loss means anything in the system buffer is lost, but delivers a noticeable speed improvement if you do have a UPS. Also, make sure to ‘Initialize’ the drives.  When you’re just fresh installed with windows, this is instantaneous.  Finally, this software is great because it’ll report any SMART errors your drives start throwing.  If a drive starts dying, it’ll warn you (hopefully) long before it finally goes bad.  I’ve seen this software work exactly as advertized when one of my own drives started failing in a previous setup.  It also gives you information about which drive is failing so you know which one to pull out and shows you status of any rebuilds when you swap in a new one – which can happen WHILE you’re actively using the system.  How cool is that?
  8. Done!  When you get into windows, you should have one large C: drive with all your space and no extra ‘boot’ drives!

I know many Linux afficianatos cringe at big bit buckets like this.   Put your swap on another partition!  Make a partition for your programs files!  Make a tiny one for your boot partition!  This is all well and good if you like micro-managing.  I hate micro-managing.  I hate extra drive letters.  We’re in the f-ing 2000’s now people!  3TB drives are $150, and 2TB drives are $65.  RAID comes built into boards for free.  Storage is a commodity.
I use Windows for most of my daily game playing/tv watching/etc and don’t want to futz with the headaches of multiple partitions.  I want to play games, watch tv, and surf the web.  I don’t want to have to worry about how big to make a partition for my programs files or data files or worry about how big my boot partition is on windows.  What if I over or under-guess?  I get to either re-install or test my luck with a partition resizer or have some programs one place, and others somewhere else.  Or the fun of every. single. time. I install a program, I have to select a custom install and pick a different drive letter.  Windows patches often automatically install in the system32 directory – what if a new service pack is too big to fit on my tiny boot partition?  It makes no difference for security if my boot is on a different partition than my data or programs.  Creating a separate partition on the same drive just for the swap file is no gain.  Unless it’s on another physical drive, you’re not gaining any speed in Windows.  And even then, I have 16GB of memory in my machine (and you can too – it’s ridiculously cheap at $60/8GB of high-quality DDR3 ram), I never swap anymore.

In short, I never have to worry about my space or what I’m putting where.  We’re not in the 70’s anymore where drives were real investments.  They’re cheap, replaceable commodities now.  I have a RAID 5 which means I’m getting the best of both speed and redundancy all in one place, and it’s details are taken care of by the hardware.  What more could one ask for.