It does get confusing because there are similarities between Fair Isle colorwork and other types, especially other types of European stranded color knitting which evolved at the same time. I've even seen traditional Fair Isle patterns that did not use the color changes between the foreground and/or background patterns. Fair Isle does use the X and O patterns pretty exclusively and they often intersperse these with smaller patterns called peeries. I've never seen a Fair Isle garment that only used two colors.
Fair Isle Chart (Look at the large X at how the background colors change symmetrically)
Not a Fair Isle Chart (Estonian) even though multiple colors are used
Fair Isle Tams with lots of beautiful subtle color gradations
Not Fair Isle tams - although multi-colored each chart pattern section has only 2 colors
Stranded Color Knitting has as an in-depth annotated bibliography but I'll briefly recommend 4 books.
Sheila McGregor's Traditional Fair Isle Knitting (very inexpensive Dover reprint)
Carol Rasmussen Noble's Knitting Fair Isle Mittens and Gloves (easy to find although some chart errors in the patterns)
Ann Feitelson's The Art of Fair Isle Knitting (probably out-of-print by now??)
Alice Starmore's Book of Fair Isle Knitting (excellent book on the subject but you'll most likely have to sell one of your children to get a copy)