Matt's Homepage

I was working on a good write-up on how to use SQLite in C# (since the most popular package install for SQLite on C# is kinda broken in Visual Studio 2010), but thought this might be a good data point for folks.

So, the short answer?  Yes, you can build SQLite databases on Windows, copy them across to your Mac, and then use them in iPhone applications without any issues.
You should open the database with the proper character formatting  ( sqlite3_open([dbPath UTF8String], &database)),
but other than that the SQL files can be just copied straight across the devices from Win 7->Mac->iPhone and the same select/delete/etc commands work like a champ.

How do I know?  I did it last night. 🙂

So, your machine is bluescreen-ing on a semi-regular basis.  It’s annoying the @#$% out of you, but you can’t find anything in the system logs that indicates what’s causing it.  Maybe (like in my case) the computer in question is your DVR box and sometime during the night Media Center is waking up, trying to update a program guide, and then blue-screening.  Nothing helpful is left in the logs, but you did get a minidump file.  If you get a minidump, my friend, you are in business!

1. Make sure you have a minidump file with your bluescreen.  You should see a numbered file with the .dmp extension with the date/time for the bluescreen located in  C:\Windows\Minidump
2. Download a handy free tool called BlueScreenView by Nirsoft.  This handy tool will automatically decipher a minidump file and you can verify that it matched what you saw on the blue-screen.  It won’t give you everything you need, but it will tell you if you have the right mini-dump for the crash you saw. It also shows you the codes thrown so you don’t have to write them down by hand at the bluescreen.  You’ll note that often BlueScreenView reports a source of the error (ntkrnl.exe in my case) but this is usually NOT the real root cause.  As we’ll soon find out, the high-level source it cites isn’t always the real problem, but was a module loaded BY that source or the module in which the source was loaded.
3. Do these one-time setup steps.  In order to make sense of the minidumps, you need some tools provided by Microsoft:
   1. Download and install the Debugging Tools for Windows pack.  Make sure it gives you the right version for your OS (win7 x64, vista x32, etc).  This pack contains the kernel debugging tools you’ll need.
   2. windbg.exe will likely be installed in c:\program files\Debugging Tools for Windows (x64) (or whatever x32/x64 you have)
   3. Open a command prompt as administrator, CD to the windbg.exe directory
   4. run:
      windbg.exe -IA
      windbg will start up, and inform you that it is now the registered file association handler for all dump files. Close windbg.exe
   5. Restart windbg, and go to file->Symbol File Path
   6. Enter:
      SRV*C:\Development\SymCache*http://msdl.microsoft.com/download/symbols
      You can set the local directory ('C:\Development\symcache' in my case) to whatever you want, but everything following the rest must be exact.  This instructs windbg to load the needed symbols from Microsoft’s internet site (release modules usually don’t have symbols, and letting you recompile your own kernel by giving the source out isn’t something MS usually lets you do. :)) Whenever you debug something and windbg needs the symbols, it checks your cache location first and downloads the needed symbols if they are not found and stores them in the cache.  So the more you debug the more symbols you build up and faster future debugging will go.  Exit windbg and save the settings.
4. Open windbg.exe (again), and do a file->open dump and open the minidump in c:\windows\minidump that corresponds to the bluescreen you’re trying to debug.  You might need to be administrator when starting windbg.
5. Windbg will automatically start downloading symbols, and doing some basic analysis.  It may look like it’s done/just sitting there sometimes, but don’t do anything until you see it’s ‘diagnosis’.  Usually looking like this:
   Use !analyze -v to get detailed debugging information.
BugCheck 9F, {3, fffffa800af7f440, fffff80000b9c4d8, fffffa800745f860}
Probably caused by : usbhub.sys
6. But don’t take this as the final word on the crash source and send nasty letters to the usbhub.sys driver writer!  Type !analyze -v as it suggest, and you’ll likely get a more detailed analysis, like this:
DRIVER_POWER_STATE_FAILURE (9f)
A driver is causing an inconsistent power state.
Arguments:
Arg1: 0000000000000003, A device object has been blocking an Irp for too long a time
Arg2: fffffa800af7f440, Physical Device Object of the stack
Arg3: fffff80000b9c4d8, Functional Device Object of the stack
Arg4: fffffa800745f860, The blocked IRP
Debugging Details:
------------------
DRVPOWERSTATE_SUBCODE: 3 IMAGE_NAME: usbhub.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 4a5bcc2d
MODULE_NAME: usbhub
FAULTING_MODULE: fffff8800767a000 usbhub
CUSTOMER_CRASH_COUNT: 1
DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
BUGCHECK_STR: 0x9F
PROCESS_NAME: System
CURRENT_IRQL: 2


STACK_TEXT:
fffff800`00b9c488 fffff800`02ef3273 : 00000000`0000009f 00000000`00000003 fffffa80`0af7f440 fffff800`00b9c4d8 : nt!KeBugCheckEx
fffff800`00b9c490 fffff800`02e9029e : fffff800`00b9c5c0 fffff800`00b9c5c0 00000000`00000001 00000000`00000000 : nt! ?? ::FNODOBFM::`string'+0x292b0
fffff800`00b9c530 fffff800`02e8fdd6 : fffff800`03034700 00000000`00146bde 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0x66
fffff800`00b9c5a0 fffff800`02e904be : 00000030`9c591870 fffff800`00b9cc18 00000000`00146bde fffff800`03002e48 : nt!KiProcessExpiredTimerList+0xc6
fffff800`00b9cbf0 fffff800`02e8fcb7 : 00000010`31b602c1 00000010`00146bde 00000010`31b602f2 00000000`000000de : nt!KiTimerExpiration+0x1be
fffff800`00b9cc90 fffff800`02e8ceea : fffff800`02ffee80 fffff800`0300cc40 00000000`00000002 fffff880`00000000 : nt!KiRetireDpcList+0x277
fffff800`00b9cd40 00000000`00000000 : fffff800`00b9d000 fffff800`00b97000 fffff800`00b9cd00 00000000`00000000 : nt!KiIdleLoop+0x5a
STACK_COMMAND: kb
FOLLOWUP_NAME: MachineOwner
FAILURE_BUCKET_ID: X64_0x9F_3_AiCharger_IMAGE_usbhub.sys
BUCKET_ID: X64_0x9F_3_AiCharger_IMAGE_usbhub.sys

Now we see the whole story.  We see that in the usbhub.sys device driver, something listed in it’s ‘DPC’ table failed to respond in time to some request the usbhub.sys made.  That process was put on the timer expiration list which threw the bluescreen.  Since usbhub.sys is a hub with many things plugged into it, odds are good that the DPC list is the list of device drivers for each device plugged into the hub, a list of events that need handling, or a list of devices themselves.  When we look at the ‘failure bucket’ we see the AiCharger_IMAGE_usbhub.sys device was the source of the failure.  Odds are good the usbhub.sys is loading ‘images’ that contain the device’s driver for each of the devices plugged into the hub; and the one that failed in this case has the name AiCharger.   If I look in my Device Manager in Windows, I find a driver called AiCharger.sys – under the USB devices.  Ah ha!  A quick Google reveals this is a driver that enables smart/high-speed USB charging of iPhone/iPod devices on my Asus motherboard. If I go one step further, I can speculate that the bug is in the portion of the driver that is supposed to respond to sleep/wake/power events and that somehow the call to wake up the iPhone I have plugged in isn’t responding.  Dang – Asus owes me a donut for doing all the work for them.

So, now you know who’s really responsible.  You send a bug note to Asus with the dump results and un-install the AiCharger tool/stop leaving your iPhone connected at night to the machine when it’s asleep until they get a fix for AiCharger.  You also find out that someone else already had the same problem…
There are many other debugging commands you can also use, and those are all outlined here.  Hopefully this will help YOU out the next time some crazy bluescreen you can’t figure out; and you won’t be re-installing the OS to get rid of it.

Protips: 99% of the time, bluescreens are usually a driver and not something in the actual Windows system.  Especially if they are repeatable.  Always get the latest drivers first.
When the crashes are wake/sleep/resume/power related, often you should go to the device driver in the Device Manager and uncheck any ‘allow system to turn off the power of this device’ as a second step if the latest driver doesn’t solve it.  This prevents Windows from making calls into possibly faulty driver code.  Power mangament issues are very common with drivers still.
If you get dumps and the crashes are different places every time or random in timing – then you might have bad memory or a bad motherboard that’s corrupting things.  Check heat sinks or temps and possibly change ram/mb’s.

Other resources:
-The official Microsoft list of bluescreen failure codes with documentation on each one:
http://msdn.microsoft.com/en-us/library/ff542347%28v=VS.85%29.aspx

-Another list of the various bluescreen failure codes and their plaintext sub-code descriptions with some notes from external folks:
http://www.faultwire.com/solutions_index/fatal_error-1.html#IssueList

-Microsoft Answers forum that has really responsive and informative threads on just about every blue-screen investigation ever done.  These guys chew up minidumps all day and can help you track down just about anything that’s going on (if just searching the forum doesn’t do it for you automatically):
http://social.answers.microsoft.com/Forums/en-US/w7repair/threads

-Another Microsoft forum that seems to do a fair amount of this kind of debug work:
http://social.technet.microsoft.com/Forums/en/w7itproperf/threads

You might already know this, but this is for those of you that want to compile extra-fast on your multi-core beast.  Bet you didn’t know that by default, most versions of Visual Studio do NOT use multi-core compiling.  So, to turn it on, do this in visual studio:

Tools > Options > Projects and Solutions > Build and Run > maximum number of parallel project builds

Set this to the number of cores you have (or the number of cores you have -1 if you want to do things on your desktop while compiling extra-big things).

To see if it’s working, when you compile, in the compiler output window at the bottom you should see each line prefixed by a number like this:
1>blahblah
4>blahblah
3>blahblah

Those prefix numbers tell you which ‘core’ the message is coming from.  I find this speeds up your compile times dramatically – especially on large projects.  Give it a try!

Interesting article from the Intel guys doing TBB.

Arch Robinson just removed almost ALL the volatile keywords from Intel Thread Building Blocks.  Why?  For several reasons, but mostly because he claims that overall it slows your code, probably does not actually solve the underlying ordering problems if your code needs to be portable (a REAL concern on today’s writing of games/apps for x86, Xbox, PS3, and iPhone devices!),  and likely isn’t doing what you think it’s doing anyway.  Here’s a pertinent example:

Sometimes programmers think of volatile as turning off optimization of volatile accesses. That’s largely true in practice. But that’s only the volatile accesses, not the non-volatile ones. Consider this fragment:

    volatile int Ready; 

    int Message[100];

    void foo( int i ) {

        Message[i/10] = 42;

        Ready = 1;

    }

It’s trying to do something very reasonable in multi-threaded programming: write a message and then send it to another thread. The other thread will wait until Ready becomes non-zero and then read Message. Try compiling this with “gcc -O2 -S” using gcc 4.0, or icc. Both will do the store to Ready first, so it can be overlapped with the computation of i/10. The reordering is not a compiler bug. It’s an aggressive optimizer doing its job.

You might think the solution is to mark all your memory references volatile. That’s just plain silly. As the earlier quotes say, it will just slow down your code. Worst yet, it might not fix the problem. Even if the compiler does not reorder the references, the hardware might. x86 hardware will not reorder it. Neither will an Itanium(TM) processor, because Itanium compilers insert memory fences for volatile stores. That’s a clever Itanium extension. But chips like Power(TM) will reorder. What you really need for ordering are memory fences, also called memory barriers.

So what’s the solution for multi-threaded programming? Use a library or language extension hat implements the atomic and fence semantics. When used as intended, the operations in the library will insert the right fences. Some examples:

* POSIX threads
* Windows(TM) threads
* OpenMP
* TBB

So, when is volatile actually necessary?  It turns out there are only 3 portable cases volatile is actually needed:

• marking a local variable in the scope of a setjmp so that the variable does not rollback after a longjmp.
• memory that is modified by an external agent or appears to be because of a screwy memory mapping
• signal handler mischief

And now you know, and knowing is half the battle.

That’s the assembly instruction you need to unlock a secret ‘debug mode’ on AMD processors since the Athelon.  While you need to be in ring 0 to execute it; it did bring up some interesting possibilities of using the special debugging mode for reverse-engineering operation of the chip, accessing possible new features, or presenting a chink in the security armor.  So far, the security problems don’t seem to be probable, but if they cause undocumented resets/etc – they might be.

Anyway – interesting article.  Original posting here.

If you get this message while converting some code from compiling with the Intel Compiler over to using Visual Studio, then I have a solution for you:

error LNK2019: unresolved external symbol _vmlsLn4 referenced in function “BLAHBLAH”
fatal error LNK1120: 1 unresolved externals

Problem:
The Short Vector Math Library (SVML) which has the vmlsLn4() function is implemented in  svml_disp.lib.  Now this lib is usually only for the Intel Compiler’s use in vectorizing code but if someone used any of the SVML instructions, you can do the below trick to fix it.  See this article for more details about using SVML.

Solution:
1. Install the latest Intel Compiler (with Visual Studio links if it asks).  If that doesn’t come with the library you need, install Install Intel Parallel Studio as well. Between the two, you’ll get the svml_disp.lib you need.

2.  At the top of the .h/.cpp file that uses , declare this external:
extern “C” { __m128d vmldLn2(__m128d a); }

3. Go to your project settings, and for the library includes, add:
svml_disp.lib

4. In the ‘Additional Library Directories” box, add a path to wherever you installed the Intel compiler’s lib directory.  The default location is:
C:Program Files (x86)IntelCompiler11.167libia32

Hit F7 in Visual Studio and you should build like a champ and run just fine.

Researchers at Queen Mary, University of London and Royal Holloway have discovered that bees learn to fly the shortest possible route between flowers even if they discover the flowers in a different order. This ‘Travelling Salesman Problem’ often takes supercomputers days to solve, but Bees are now the first animals proven to do it. Computer-controlled artificial flowers were used to track the bee’s path and found they quickly learned the shortest route. Since a bee’s brain is only the size of a pinhead, researchers are hoping to identify the neural circuitry required and use that understanding to construct their own systems that rival the computational power of existing machines

Fascinating. A humbling reminder that despite the fact we consider computers near godlike in their abilities – there are solutions even the brightest minds can’t duplicate with a Turning-based machine. Or even reproduce at all.

Has often made me wonder if some day we’ll have a quantum or other non-Turing ‘co-processor’ with our current machines that is based on an architecture more suited to solving these types of difficult problems.

I go to The New Old Thing a fair amount. It’s a blog written by a veteran system level guy at Microsoft. He puts some fun articles up – and I had to respond to this one. Here was his original article:

A car park in Birmingham switches from English to German in times of stress. Over a decade ago, a colleague of mine noticed an error message on the screen at the exit to the parking garage at the Seattle-Tacoma International Airport. The way the airport works, you pick up a ticket as you enter, and you pay your parking fee at vending machines stationed around the parking garage, and at the exit, you insert the (paid) ticket into the machine, which verifies that you paid your parking fee and opens the gate.

When my colleague pulled up to the machine, it had an error message. In German. Fortunately, my colleague knows German, and he recognized the error message as a Windows 95 serial port conflict resolution dialog. While he was trying to figure out how to click Abbrechen on a machine with no mouse or keyboard, an attendant walked up, took his ticket, and opened the gate.

A conversation ensued about what went wrong and the right way to fix it. Here was one guys response:

yuhong2: Well, as I remember, the dialog in question relates to multiple DOS virtual machines trying to access a serial port at the same time. There was several ways to virtualize a hardware device across multiple DOS VMs, and one of them was to allow only one DOS VM to access a device at a time. If two of them tried to access at the same time, the only option was to pop up a conflict resolution dialog like this one, and Win3.x and Win9x had built-in support for doing this. More information on all this stuff can be found in the old DDK docs (like the Windows Server 2003 SP1 DDK) that had the VxD documentation.

At which point I needed to reply with the following, and got some serious ‘Amen brothers’:

@yuhong2 – your answer is totally plausible, but shows what a lot of us do (me included) – in the face of a bug like this – we just keep jumping down the engineering rat/rabbit hole without stopping to ask a more fundamental question of whether you’re even on the right track with your architecture.  I’ve learned to stop looking down the hole and to try and smell them coming before I start. Trust me – there IS no bottom. And more importantly, when you’re spending your time looking off the cliff, you’re not looking at your goal anymore.

After being a software engineer for 10 years on major projects – I’ve learned this one lesson about code usability: if you’re using more than 3-4 sentences your mom can’t understand to describe a fix or architecture; something is likely gone, or will be going, very wrong soon.

While I completely understand what yuhong2 is saying and it’s very plausible and intelligent, if it’s true, it shows me that someone made a terrible choice of platforms/architecture when choosing to solve this problem of a parking meter. I’ve worked with more than a few tech leads that come up with horrendously complicated algorithms and architectures to solve problems that could be solved MUCH more easily. And you know what? The easier solution, while perhaps not the fastest or cool looking, is almost always the fastest and best long-term. Why? Because those complex architectures have even more complex problems. i.e. yohung2’s answer.
As the project grows, it just gets worse and worse – not better. Until you need a phD to figure out why some part of your threaded, interconnected data structures are hanging or getting corrupted once every 30 days. This should be a moment to stop and ask yourself what you’re really doing. Sure, certain problems really do require complex solutions (i.e. 3D graphics, threaded applications and drivers – some of what I do), but know your tools and the strengths and limitations of them (the hardware platform, the language you’re using, the software/os stack, etc). Yes, you darn well better have a good toolbox of the latest algorithms, languages and OS info, but your toolbox will do nothing but get you into trouble if you don’t know how and when (or when NOT) to really use those tools. If yuhong2 is right, you see how your complex solution just backed you into a corner and shows you someone probably picked the wrong stack or algorithmic approach when just having a very simple box that checked times would have worked.

I have worked long enough to also know that many times platform choices are out of your control as an engineer – budgeting and marketing often limit you. But if you’re forced to use a platform – design so-as to avoid the limitations of that platform – don’t try to force them into doing what you want. As a rule of thumb, always stick with the simplest solution that completely solves the problem first, then you’ll be re-writing to solve performance and feature issues instead of core this-thing-doesn’t-even-work-yet problems. At worst, I’ll have a working system that’s slow and I can then optimize and re-design the parts that need it. I won’t have wasted time optimizing for things that may not have needed any help. Now the caveat is that you really must know what you’re doing and why your making those choices for simplicity. It’s just as bad to stupidly pick an architecture that’s too simple for the problem and gets you just as much in a corner. But I’ll argue a working, slow system will always sell and ship before one that’s 10 months late and MIGHT be faster. Simplicity also makes the code more maintainable and much more extensible long term as there’s less inherent inertia in the code to move about.

Software is like using clay to make art. Sure you can beat and force and manipulate it to create very complex structures – but you start making more and more complex problems for yourself. i.e. if you try to build a car engine from clay. If you work *with* the clay/code’s natural strengths and not force it to do what it isn’t good at, however, you end up with beautiful thing that’s simple, works well for its intended use, and is very easy to maintain (a clay bowl).

Another guy summarized even better:

For any sufficiently complicated system, there will always be failure modes where the average Joe has but two options: call an expert to fix it, or arrange his life to not use that system. In the case of the parking garage, Joe’s only rational response is to use a different exit lane.

Computer interface design goes wrong when the programmer refuses to admit that the system is in such a state. The parking garage’s system should have displayed “Closed – use a different lane”, with the error message shown on a maintenance screen.

This echo’s some of the great wisdom I’ve been learning from this great book:

Believe it should be on every programmer’s shelf.  I’ll do some more postings as I get through it.