Unattractive or unlikeable (David's word) or disagreeable or unpleasant or ill-natured: the opposite of amiable (friendly, pleasant). The word "unamiable" does exist, though it is not very common nowadays.

The main point here is "but", which in this case means "apart from", "except", "besides": its first meaning in Webster's 1828 dictionary.
http://1828.mshaffer.com/d/word/but

So "few but unamiable traits" means "few traits apart from unamiable ones": in other words, nearly all the traits any other observer would have seen in the child were unamiable; there were few to be observed besides those. To any other observer apart from his/her mother, the child would have seemed almost entirely unamiable; most of the traits such an observer would have seen in him/her were unattractive and therefore few (if any) were attractive or pleasant.


--------------------------------------------------
Note added at 17 hrs (2012-07-16 09:12:02 GMT)
--------------------------------------------------

Just to add to what I have replied to David about "unamiable", it is quite true that this word is not listed in moden dictionaries (at least the ones I have looked at), but it was once a recognised word. As I say, it is defined (as "not raising love") in Dr Samuel Johnson's A Dictionary of the English Language, first published in 1755, with eighteenth-century usage examples from Joseph Addison in The Spectator and the poet John Philips.
http://archive.org/stream/dictionaryofengl02johnuoft#page/n9...

Johnson's work was regarded as the pre-eminent English dictionary until the first OED was completed in 1928. So I think his inclusion of "unamiable" is sufficient to establish this as a real word, albeit now archaic. When Nathaniel Hawthorne wrote this passage (from The Scarlet Letter) in the mid-nineteenth century, it was still in use, though it is very rarely found in modern English. But it is certainly not a non-word.

In any case, the meaning of "unamiable" is fairly obvious; the difficulty here is the correct interpretation of "but".

--------------------------------------------------
Note added at 6 days (2012-07-22 15:02:52 GMT)
--------------------------------------------------

By coincidence, I have just come across the word "unamiable" in David Copperfield by Charles Dickens, which I am currently re-reading. The passage refers to the odious Miss Murdstone:

"I do not doubt that she had a choice pleasure in exhibiting what she called her self-command, and her firmness, and her strength of mind, and her common sense, and the whole diabolical catalogue of her unamiable qualities, on such an occasion." (chapter 9)