Flashcard Scoring / SRS

[mikelove]

- cards which have been 'forced excluded' out of the pool
- (may as well search for 'forced included' too, if those are specifically marked in some way)
- cards which are not yet learned (according to whatever definition of 'learned' the user has set) but have been tested at least once
- cards which were marked wrong for the most recent X tests (user can specify how many)

All good suggestions - the first two were actually included in one development version but we took them out for reasons I can't quite remember now, but it would make sense to reintroduce them as part of our whole general "pool" overhaul. Thanks.

 

[HW60]

Very well, here's a quickie outline for the newer system:

• Initialize each card with "score" and "easiness factor" values of 100.
• If a score change is permitted (the score hasn't been changed too recently), then:
  • Correct answer: Divide the card's current "easiness factor" by the "easiness divisor" configured in Tweak Parameters, and multiply the result by the card's current score.

I am very sorry boring you, but I actually stopped reading here after looking at the Tweak Parameters and getting lost there. My suggestion: What about making a very simple example with real numbers, starting with values of 100 for score and easiness factor, using the default Tweak Parameters, assuming that a score change is permitted, and as an example assuming three correct answers in a row and a wrong answer afterwards? That would lead to 5 scores starting with 100 and everybody would know, at what day the tests will be repeated using repetition-spaced card selection.

Maybe the User Manual too would become much more readable if some very easy examples would show what the rules really mean.

 

[mikelove]

I am very sorry boring you, but I actually stopped reading here after looking at the Tweak Parameters and getting lost there. My suggestion: What about making a very simple example with real numbers, starting with values of 100 for score and easiness factor, using the default Tweak Parameters, assuming that a score change is permitted, and as an example assuming three correct answers in a row and a wrong answer afterwards? That would lead to 5 scores starting with 100 and everybody would know, at what day the tests will be repeated using repetition-spaced card selection.

Good point - here's a version with the default Tweak Parameters values for the Spaced Repetition profile:

• Initialize each card with "score" and "easiness factor" values of 100.
• If a score change is permitted (the card hasn't already been answered the same way (correct or incorrect) within the last day), then:
  • Correct answer:
    • Divide the card's current "easiness factor" by 40, and multiply the result by the card's current score; subtract the original score to get the "score increase."
    • If the answer was rated 4 ("barely remembered"), multiply this "score increase" by 90%; if it was rated 6 ("remembered perfectly"), multiply it by 110%.
    • If the card is being reviewed later than scheduled, divide the actual interval by the scheduled interval, take the square root of that, and multiply the "score increase" by that. If the card is being reviewed early, don't change its score at all.
    • Add the "score increase" to the previous score. Check to see that the card's score is now at least as 200 if "barely remembered" / quality 4, 400 if "remembered" / quality 5, or 600 if "remembered perfectly" / quality 6; if not, raise it to that score.
  • Incorrect answer: Drop the card's score back to 100.
• Ensure that the new score is between 100 and 10000, and raise / lower it if needed to put it within that range.
• Finally, adjust the "easiness factor" based on the answer quality, -12/-10/-8/-4/0/4 for 1/2/3/4/5/6 respectively. Ensure that the new "easiness factor" is between 50 and 200, and raise / lower it if needed. Skip this step if the card was answered correctly ahead of schedule in a repetition-spaced test.

So in your proposed example:

• Initial: Score 100, EF 100
• First Correct Answer (quality 4): Score 235 ((100 * (100 / 40) - 100) * 90% + 100), EF 96 (-4)
• Second Correct Answer (quality 5): Score 564 ((235 * (96 / 40) - 235) + 235), EF 96 (+0)
• Third Correct Answer (quality 6): Score 1433 ((564 * (96 / 40) - 564) * 110% + 564), EF 100 (+4)
• Incorrect Answer (quality 1): Score 100 (reset), EF 88 (-12)

 

[HW60]

So in your proposed example:

• Initial: Score 100, EF 100
• First Correct Answer (quality 4): Score 200 (initial), EF 96 (-4)
• Second Correct Answer (quality 5): Score 480 ((200 * (96 / 40) - 200) + 200), EF 96 (+0)
• Third Correct Answer (quality 6): Score 1219 ((480 * (96 / 40) - 480) * 110% + 480), EF 100 (+4)
• Incorrect Answer (quality 1): Score 100 (reset), EF 88 (-12)

With a first answer quality of 4, the "score increase" seems to be (100/40)*100-100 = 150, barely remembered makes 150*90% = 135, so the new score is 235, which is greater than 200 and therefore needs not be adjusted. What did I get wrong?

 

[mikelove]

You're right, actually - forgot about that but I've updated the example now. Thanks!

 

[Charles]

Michael - Thanks for the explanation and the example. I sort of understand it now, but I don't think I will ever have the energy to change the default settings. For me, the flashcards work great without touching the scoring system parameters.

 

[HW60]

I made a quick and dirty Excel file with 2 tables "Score" and "Tweak" to test what the different parameters in Flashcards / Advanced Settings really do. It should be very easy to use: enter figures only in yellow cells, from 1 to 6 in line 1 of "Score", positive numbers (not zero!) in line 3, and y or n for yes or no in the yellow cell of line 4 (line count in column 1). The yellow cells in table "Tweak" can be changed to test setting parameters of Pleco: how do scores, easiness factors and time to learn develop. I hope that the number of errors I made in these tables is not too big - any ideas for improvements are welcome.

 

[rcloud]

Mike,

I'm of those who had alway used Pleco for my flashcards and then when the iPhone version was released migrated over to Anki in order to continue to be able use flashcards on my phone. Once I made the switch I realized it would be extremely difficult to ever go back, although whenever your release a new version I always go and check out the flashcards.

Here are some things that I've come to love about Anki that I'd like to see implemented in Pleco.

#1 & most importantly: The ability to add media- I have a lot of sentences that utilize recordings of natives speaking an entire sentence. Being able to use SRS for listening practice was a revelation for me and is now a large part of my flashcard deck. Also, part of my deck consists of food ingredients and dishes, without pictures in my deck these would become almost useless as learning the name of a dish isn't very helpful without knowing what it is.

#2 & almost just as important: Statistics- The more the better. Keeping track of my progress is extremely important. Someone already mentioned the HanziStats plugin and it's a feature that I love. Although, even more importantly is keeping track of review history. I know it may seem silly, but being able to see a graph with my review count and time keeps me motivated as I know if I don't do my reviews I'll always see that dip to zero on my history graph.

#3- Ability to easily add sentences - I love being able to add sentences to my deck right on my phone when I encounter them.

#4- Faster scoring of cards - In Anki, there is a default answer, based on your history with that particular fact, that guesses how well you know the fact. I have found this to be spot on almost all of the time except for when I have forgotten a fact. In Anki I have it set up so that different area of the screen correspond to different scorings, so I can set up tap zones for default and incorrect which allow me to fly through my reviews. The current system that you have looks much prettier, but I have found it takes longer and even small differences in time add up when you are doing hundreds of reviews a day.

#5- Syncing- A nice feature, but I'm not sure it's as important as you think. Most people I know that use Anki only review on their phone. So while it is really nice to add cards from the computer and sync them, I don't think that AnkiOnline is the main reason people use Anki.

#6- Accept Custom Fonts - I love being able to use the KaiTi font in my flashcards.

#7- Edit cards on the phone - With the dictionaries built into Pleco, I wouldn't have to worry about incorrect definitions for words, but for sentences this is important as well as for adding notes to cards (especially important for adding mnemonic devices).

And one last feature specifically for SRS and one that isn't included in Anki, but I'd love to see...

#8- Local notifications when a set number of cards are due to be reviewed

I love Pleco and find it indispensable, but for me these things would have to be included in order to convince me to switch back. For now, I use Anki for SRS and then when I need clarification on a definition I copy the words or sentences and switch to Pleco (via an Activator swipe) which looks up the word or the sentence in the pasteboard. I have found this to be the best of both worlds as it essentially integrates the two apps.

 

[mikelove]

Thanks for the thoughtful feedback on this.

#1 & most importantly: The ability to add media- I have a lot of sentences that utilize recordings of natives speaking an entire sentence. Being able to use SRS for listening practice was a revelation for me and is now a large part of my flashcard deck. Also, part of my deck consists of food ingredients and dishes, without pictures in my deck these would become almost useless as learning the name of a dish isn't very helpful without knowing what it is.

Oft-requested and should hopefully be coming in 2.3 - would actually be pretty easy if we didn't have to also come up with an interface to manage / import / export / copy between cards / etc the media you'd created.

#2 & almost just as important: Statistics- The more the better. Keeping track of my progress is extremely important. Someone already mentioned the HanziStats plugin and it's a feature that I love. Although, even more importantly is keeping track of review history. I know it may seem silly, but being able to see a graph with my review count and time keeps me motivated as I know if I don't do my reviews I'll always see that dip to zero on my history graph.

By "review count" do you mean the number of cards reviewed each day? We're not tracking the statistics needed for that right now for space-saving reasons, but we could easily start doing so I suppose.

#3- Ability to easily add sentences - I love being able to add sentences to my deck right on my phone when I encounter them.

That's actually already possible - you just go into Organize Cards and tap on "New Card"; you can even set up a shortcut to do this with a button tap-hold from the main dictionary screen. Is there something Anki does to make this particularly streamlined or were you just not aware that Pleco had this feature?

#4- Faster scoring of cards - In Anki, there is a default answer, based on your history with that particular fact, that guesses how well you know the fact. I have found this to be spot on almost all of the time except for when I have forgotten a fact. In Anki I have it set up so that different area of the screen correspond to different scorings, so I can set up tap zones for default and incorrect which allow me to fly through my reviews. The current system that you have looks much prettier, but I have found it takes longer and even small differences in time add up when you are doing hundreds of reviews a day.

This one I'm a little wary of - if a card is always coming up with anything other than a 6 / highest possible score, doesn't that suggest that the system should be showing it to you more often? We already do have an option to show all 6 score choices in lieu of correct / incorrect buttons followed by 3 choices each, and I suppose we could add an option to hide two of the incorrect choices (so you'd get a 1x4 instead of a 2x3 grid of buttons to tap on), but I'm worried that changing the order of buttons for each card would create more problems than it would solve.

#5- Syncing- A nice feature, but I'm not sure it's as important as you think. Most people I know that use Anki only review on their phone. So while it is really nice to add cards from the computer and sync them, I don't think that AnkiOnline is the main reason people use Anki.

It's actually more important for us because a lot of people like to create new cards while reading documents on their iPads, then review that vocabulary on their iPhones; that sort of system is very awkward to set up without online sync.

#6- Accept Custom Fonts - I love being able to use the KaiTi font in my flashcards.

Does Anki support this on iPhone? And with fonts that you supply yourself rather than fonts that are built into the app? And you're installing the font on the iPhone instead of, say, generating images of characters with the font on your desktop and copying those over to the iPhone? Not sure how he's doing that - I guess he must have implemented his own copy of FreeType (as we occasionally threaten to do).

#7- Edit cards on the phone - With the dictionaries built into Pleco, I wouldn't have to worry about incorrect definitions for words, but for sentences this is important as well as for adding notes to cards (especially important for adding mnemonic devices).

Already supported with custom cards - see #3.

#8- Local notifications when a set number of cards are due to be reviewed

That certainly might be doable - it would be annoying to have that come up whenever even one new card was due for review, but maybe if you set it to remind you to review at a specific time every day (or once the # of cards due reached a certain threshold) it could work.

 

[rcloud]

Thanks for the thoughtful feedback on this.

You're welcome, I was worried as I was writing that up that it would come off as Anki is king and Pleco has no features, which was not my intention at all.

Oft-requested and should hopefully be coming in 2.3 - would actually be pretty easy if we didn't have to also come up with an interface to manage / import / export / copy between cards / etc the media you'd created.

Great to hear! This and the statistics issues are the only true 'deal-breakers' for me.

By "review count" do you mean the number of cards reviewed each day? We're not tracking the statistics needed for that right now for space-saving reasons, but we could easily start doing so I suppose.

Yes, in Anki the number of cards reviewed each day is listed and also shown as a graph along with one for the time spent reviewing. Before I started using Anki I never would have thought I'd ever care about such a feature, but like I said now it is a huge motivation for me. On those days where I don't feel like studying and think to myself "it doesn't matter if I skip one day", I remind myself that I have set a goal to study every day and the graphs hold me accountable.

That's actually already possible - you just go into Organize Cards and tap on "New Card"; you can even set up a shortcut to do this with a button tap-hold from the main dictionary screen. Is there something Anki does to make this particularly streamlined or were you just not aware that Pleco had this feature?

I apologize, I completely missed that feature. Not only do you have it, but now that I look at it your implementation is currently better as in Anki I have to deal with HTML color codes for characters.

This one I'm a little wary of - if a card is always coming up with anything other than a 6 / highest possible score, doesn't that suggest that the system should be showing it to you more often? We already do have an option to show all 6 score choices in lieu of correct / incorrect buttons followed by 3 choices each, and I suppose we could add an option to hide two of the incorrect choices (so you'd get a 1x4 instead of a 2x3 grid of buttons to tap on), but I'm worried that changing the order of buttons for each card would create more problems than it would solve.

I guess this come down to a philosophical difference in grading between the two apps. As in Anki the scoring is incorrect and then three correct scores based on how well you know the fact. So Anki can follow your progress with each card and guess how well you should know it based on the confidence of your correct answers. Hiding the two incorrect choices would make it easier for Anki users to transition to Pleco.

Does Anki support this on iPhone? And with fonts that you supply yourself rather than fonts that are built into the app? And you're installing the font on the iPhone instead of, say, generating images of characters with the font on your desktop and copying those over to the iPhone? Not sure how he's doing that - I guess he must have implemented his own copy of FreeType (as we occasionally threaten to do).

Currently, whatever font you have set in your desktop version of Anki is what AnkiMobile looks to use. So, for those who JB its just a matter of installing the font of your choice. However, Anki does support embedding fonts in SVG, it just isn't as complete a solution as the sheer size of Chinese fonts means converting a .ttf to .svg results in a large file, but works for those who don't jailbreak.

That certainly might be doable - it would be annoying to have that come up whenever even one new card was due for review, but maybe if you set it to remind you to review at a specific time every day (or once the # of cards due reached a certain threshold) it could work.

I definitely agree with you on notifications for one card, which would be extremely annoying. What I was thinking was, like you said, the user sets a certain amount of cards and once that number is reached they receive a notification.

 

[numble]

rcloud mentions a lot of the things that I've been too lazy to write about Anki.

A couple of differences for me:

Going through cards on the desktop is definitely faster than on the phone--just set your fingers to 1234 and your thumb on space, and you're rapidly going through flashcards. Anki was pretty slow on my 3G, but it is blazing fast on my iPhone 4--I went through over 550 cards yesterday, in probably the same time I used to do 2-300, so I might just be doing it even more on the iPhone now.

The official non-jailbroken version of Anki for iPhone is not as feature-rich as the jailbroken one. It doesn't allow you to add cards yet, and I don't think it offers any statistical information or notifications. I'm not sure if the fonts travel over either (I don't think they do).

 

[mikelove]

Yes, in Anki the number of cards reviewed each day is listed and also shown as a graph along with one for the time spent reviewing.

Good, that's thankfully a pretty easy statistic to record if we just start doing it.

I apologize, I completely missed that feature. Not only do you have it, but now that I look at it your implementation is currently better as in Anki I have to deal with HTML color codes for characters.

Great!

I guess this come down to a philosophical difference in grading between the two apps. As in Anki the scoring is incorrect and then three correct scores based on how well you know the fact. So Anki can follow your progress with each card and guess how well you should know it based on the confidence of your correct answers. Hiding the two incorrect choices would make it easier for Anki users to transition to Pleco.

Pleco could certainly do that part too - we're already tracking the previous answer history and could easily infer what answer you're most likely to choose next - but I'm still not quite understanding the utility of it; if you want to choose something other than your default answer, isn't it going to take you longer to figure out which of the other buttons you're supposed to tap than it would otherwise? And isn't there also a bit of lag required to see which answer it thinks is the default and decide whether that's the correct one?

I'm also worried that it might act as a disincentive to actually think about how well you remembered a card before rating it - If 4/5/6 are always in the same three places the process can be largely automatic, you think about how well you remembered the card and tap on the button that corresponds to that, but if you're sailing through a session, you encounter a card that defaults to 6 "remembered perfectly" and actually have a little trouble remembering it this time, you might still go with that default 6 rather than taking the time to pick 4.

Currently, whatever font you have set in your desktop version of Anki is what AnkiMobile looks to use. So, for those who JB its just a matter of installing the font of your choice. However, Anki does support embedding fonts in SVG, it just isn't as complete a solution as the sheer size of Chinese fonts means converting a .ttf to .svg results in a large file, but works for those who don't jailbreak.

Ah, that makes more sense. We actually do a little tiny bit of SVG rendering too with some very rare characters that aren't in the iPhone's fonts but are in the stroke order diagram database, but if we wanted to support custom fonts it would probably be no more trouble to just implement FreeType (and we'd get the added benefit of it being fully operational on non-jailbroken devices too).

Going through cards on the desktop is definitely faster than on the phone--just set your fingers to 1234 and your thumb on space, and you're rapidly going through flashcards. Anki was pretty slow on my 3G, but it is blazing fast on my iPhone 4--I went through over 550 cards yesterday, in probably the same time I used to do 2-300, so I might just be doing it even more on the iPhone now.

On that note, I'm very much looking forward to seeing the rumored Android-running Sony PSP successor - it would be the first Pleco device since the Tapwave Zodiac that actually had good, solid, physical control buttons sized and positioned in such a way as to allow you to comfortably use them to run through hundreds of flashcards. Of course, given Sony's past history in this area the buttons may be undersized and feel thin and stiff, but there's also at least a chance they could get it right and a year from now you could be blissfully zooming through flashcards on one of those.

 

[doctorj]

Hi Mike,

I have been using another SRS flashcard program (Flashcards Deluxe) on my iPod touch for a while and have been really pleased with it. It has worked really well for textbook vocab lists which can do a single bulk upload.

Recently i have started using Pleco flashcard /SRS system in parallel for the ease of which you can enter and start revising random vocabulary that i come across in daily life. I am pleased with the software so far but there are a couple of things that would be ideal from my perspective.

I would like there to be an option of giving progress in terms of due dates or current interval instead of score. Currently a score of 600 seems meaningless, whereas an interval of 2 days, or 30 days makes much more sense.
On a similarly note, I would love to be able to see what effect each answer would have on the score/ interval i.e if the score is currently 400 then a 6 different scores under the 6 different correct/incorrect options.

That's all for now.

 

[mikelove]

I would like there to be an option of giving progress in terms of due dates or current interval instead of score. Currently a score of 600 seems meaningless, whereas an interval of 2 days, or 30 days makes much more sense.

That's definitely coming - it's only in the last year or two that SRS has gotten to be so utterly dominant in flashcard apps that we can start to plug it at the expense of other systems.

On a similarly note, I would love to be able to see what effect each answer would have on the score/ interval i.e if the score is currently 400 then a 6 different scores under the 6 different correct/incorrect options.

That one I'm not quite so wild about because some of the future algorithm upgrades we're considering would actually vary the interval after you tested the card - this is a ways off (2013, probably), but we've started thinking about ways that we might automatically compensate for a glut / scarcity of cards due for review by changing some cards' intervals after their last test. The math behind SRS doesn't have to be adhered to too rigidly for it to be effective - the difference between a 6- and an 8-day interval is pretty small as far as one's brain is concerned - and the added flexibility would better accommodate people with irregular study habits.

 

[insighter]

Another question to raise here, though only partially scoring-related - are there any other batch functions or search fields you'd like to see? Some particular transformation you'd like to have automated, maybe on the score / easiness factor or maybe even on the card text? (remove something, add something, change the score a certain way if certain conditions are met, attempt to auto-convert characters to Pinyin, bring up a list of any cards where the Pinyin doesn't appear to match the characters, etc)

I may have mentioned something like this before, but I think an SRS system that took into account multiple scores for each part of the character (pinyin, tones, writing) would be great. It's very possible to keep testing how to write a character over and over again when you suddenly figure out you have forgotten something very basic, like the character's pinyin. Your varying test modes based on card scores option is a good start, but it is limited by the fact that once score has increased on a particular card to a level where you move out a certain test mode (pinyin, recognition, tones) onto something more advanced (testing writing) you cannot use the algorithm of SRS to make sure you don't forgot your earlier knowledge of tones, pinyin, etc.

 

[HW60]

One of my biggest problems with SRS is the huge amount of cards I have to test after a pause of 2 or 3 weeks (holidays, sickness or else) - there is no way and no system to catch up in a moderate time, and SRS (and Pleco) provide no support for such a normal event. An easy way to reduce the number of cards after a pause would be to change the calender date of the device, going back the appropriate number of days as if the pause had not happened. Unfortunately changing the date normally causes some other problems ...

Maybe a batch change of all flashcards can do the same job: change the date of creation of all cards and all subseqent tests by a fixed number of days (i.e. 14 or 21 days in the example), and the amount of cards to be tested the next time is the same as if the pause had never happened. Of course cards just learned and tested once shortly before the interruption maybe forgotten after three weeks, but that would be the normal development and cannot be helped.

 

[mikelove]

I may have mentioned something like this before, but I think an SRS system that took into account multiple scores for each part of the character (pinyin, tones, writing) would be great. It's very possible to keep testing how to write a character over and over again when you suddenly figure out you have forgotten something very basic, like the character's pinyin. Your varying test modes based on card scores option is a good start, but it is limited by the fact that once score has increased on a particular card to a level where you move out a certain test mode (pinyin, recognition, tones) onto something more advanced (testing writing) you cannot use the algorithm of SRS to make sure you don't forgot your earlier knowledge of tones, pinyin, etc.

That's something we've thought about, yes - letting you essentially test multiple "profiles" at the same time.

One of my biggest problems with SRS is the huge amount of cards I have to test after a pause of 2 or 3 weeks (holidays, sickness or else) - there is no way and no system to catch up in a moderate time, and SRS (and Pleco) provide no support for such a normal event. An easy way to reduce the number of cards after a pause would be to change the calender date of the device, going back the appropriate number of days as if the pause had not happened. Unfortunately changing the date normally causes some other problems

Maybe a batch change of all flashcards can do the same job: change the date of creation of all cards and all subseqent tests by a fixed number of days (i.e. 14 or 21 days in the example), and the amount of cards to be tested the next time is the same as if the pause had never happened. Of course cards just learned and tested once shortly before the interruption maybe forgotten after three weeks, but that would be the normal development and cannot be helped.

Already available; "Search Cards" for cards with "# reviewed" > 0, then tap Edit, Batch, choose "Increase by" under Score / Change Score, set "Amount" to 1400, and tap "Change Score Now." That'll increase all of their study intervals by 14 days. (if you remember them correctly after that long then they "deserve" to have their next interval calculated on that basis, if not then they'll go back to 1 day regardless) You could also set "points per day" under Card Selection to a smaller number (say 50), then gradually increase it again as you clear up the backlog.

But as I said, we're thinking about ways we might deviate from "standard" SRS a bit to deal with situations like this more effectively - we already prioritize the cards that are the farthest past-due when you do a rep-spaced test after a long time, but we probably ought to either a) prompt you before a long test to indicate how many cards you feel like testing now (and select which cards to test / bury other cards intelligently based on that), or b) keep going until you indicate that you're finished, selecting appropriate cards along the way. (we kind of do this now with "Endless" "Weighted" tests, but they're a clumsy way to do it and will probably disappear altogether in 2.3/2.4)

 

[scykei]

Is it possible to change into different "user accounts" that have different sets of scores? Like when someone else wants to try the flashcards and I don't want them to mess up my score.

 

[mikelove]

Is it possible to change into different "user accounts" that have different sets of scores? Like when someone else wants to try the flashcards and I don't want them to mess up my score.

Sure - create a new profile (Flashcard Testing / Profile / tap on "Manage" at the bottom left corner of the screen / tap on "New"), and on the screen where you name it, choose "(create new)" under scorefile. This will give you a new flashcard test profile with its own private set of scores, totally independent of yours.

 

[scykei]

Ohh that's cool! I never figured out how to use all of the options though. Is there a list of the settings showing what each of these profiles apply?