After taking several IAT's, I found myself consistently receiving scores that indicated a small automatic association consistent with the more common cultural prejudices (associating women with liberal arts and men with science for example). While this may reflect something about my unconscious thought, I was also somewhat skeptical since the tests always presented these pairings before the other pairings. This seemed to favor the associations presented first, perhaps due to the primacy effect.
Correlation, however, does not imply causation, so using open source versions of the IAT found online I attempted to test this hypothesis. In order to assess the IAT rather than my own subconscious associations, I tried to measure associations between unrelated artificial concepts such as even or odd numbers with consonants or vowels. Given that I am no more likely to associate 2 with Q and 3 with A rather than the opposite, any associations "revealed" by the test are merely a product of an error in the test (either systematic or random). I repeated this procedure with several artificial associations and consistently found that the test reported a slight automatic association consistent with the pairing presented first. This indicates a small systematic bias in IAT.
I certainly don't claim that this invalidates the test though. It only indicates a small systematic error: a moderate or greater preference as reported by the test scores is unlikely to be caused by such a bias in the test. There were also a number of shortcomings in my method. The sample size was extremely small: a single person. Additionally, my own participation may have made the results susceptible to the experimenter expectancy effect.