Spent 2-3 hours today trying to speed up parsing with memoization. Got the syllable lexer memoized. It was actually slower.
The word parser/divider isn't done in a functional style, so I found it hard to memoize. It might stand to gain more, since it has more Python loops, etc., but it started looking like a time pit, so I quit. I left the code in there: Syllables.Word._FastSplitRest(), but I made sure compile() never calls it. Might come back to it later, but I doubt it, unless speed really becomes an issue.
It still bugs me, though: I know we are parsing some of the same tails over, and over, and over. But at this point, it's waste a few cycles, or waste a lot of hours.
If this becomes an issue, it'd be wise to profile first, I think. I was really sure the syllable lexer would be easy to memoize, and it was. I was really sure it would give me 10x the speed, but it did the opposite. Python can be that way, I've heard: your gut (your c and asm?) instinct about what's fast and what's not isn't always right.