Flashcard card selection order for cards that are due

Cookiemole

Member
In the card selection settings, it seems like there is an option to randomize the order of the cards that are due. If it's not selected, Pleco defaults to ordering the cards in order of due date. I would like to ask for a third ordering option: reverse due date. This means, of the cards that are due, the ones with the most recent due dates show up first.

I'd like to argue in favor of this feature by presenting the following use case example. Between the HSK and a few other lists, I have about 10k flashcards. For the first year of using this program, I was pretty diligent about doing all the flashcards that would show up in my inbox. But in the last few months I've gotten lazy and now I have a couple thousand cards that are due. To cut this down to bite size chunks, I can enable certain categories at a time, but even if I only enable HSK6 I still end up with 700 cards in my inbox, which is still overwhelming. Let's say I have some free time today and fire up the flashcard app. Let's also say I only feel like doing 50 cards today. If I fire up the app tomorrow I won't be able to get reps on the 50 cards that I do today. Instead, because the number of due cards is around 700, I'll be presented with a completely different set of cards because those are considered to be more overdue. The next time I see the 50 cards from today could be a month from now, depending on how lazy I am and how long it takes me to get through the backlog. I'll be denied a sense of progress, and the feeling that I'm reinforcing what I'm learning today.

If memories fade like exponential decay, like the SRS models claim, then the cards that are months overdue have mostly decayed already, and the leftover residue is only decaying slowly. There is no need to do earlier cards first; I would argue that it is more urgent to catch the cards that have just become due, because you see more gains if you are reinforcing a card that is fresher in your memory.

In a practical sense, this would divide up the backlog into bite-size chunks without the need to fiddle with categories or filters. Even with a backlog of thousands of cards, the set of cards that are being "worked on" are the ones that have most recently been seen. If I do 50 cards today, the first 50 I see tomorrow will mostly be the ones I worked on today. Older cards would stay in cold storage, with minimal decay rates.

With this in place, I would argue that there won't even be the need to limit the number of unknown cards in your hand, because there won't be any pressure to finish the whole set of due cards.
 

mikelove

皇帝
Staff member
Thanks for the feedback.

The strategy we're tentatively going with for this in 4.0 is to prioritize cards not by due date but rather by, essentially, (time past due date) / (scheduled interval) (probably not that exactly, might be a logarithm or two in there) - in other words, we'd still favor cards with earlier due dates, but cards with short intervals would be much more likely to come up first. This is based on the expectation that cards with long intervals are less likely to fade from being delayed a few days than cards with short intervals. In (probably) 4.1, once we've gathered enough data (from people who opt into sharing data with us) to come up with a decent model for Pleco forgetting curves (hard to go by past work because we've got some novel factors at play like seeing words outside of a flashcard test [which we'll now factor in] / different test types / etc), we'll likely tweak that to prioritize words by the likelihood of forgetting them if you wait another day to review them, but when we do that it'll likely still pretty much focus on cards with shorter intervals, since everything we know about memorization theory suggests those ought to be the priority among cards you're still trying to save.

For situations like yours, where you've got a large accumulated backlog and can't really just work through it in a day or two, rather than try to solve that problem with a priority algorithm we're planning to have a dedicated "you have lots of overdue cards, how would you like to fix this" command on the New Test screen, and one of the options we're planning to offer through that is an option to essentially 'give up' on cards that are way overdue and introduce them again as new cards at whatever pace you've set out for that. (they'll probably be treated as something slightly different than new cards, actually, 'suspended' status or some such - introduced within your quota of new cards per day but with a faster advance back to long-term review if you get them right) There'll probably also be a less dramatic 'distribute these reviews evenly over the next N days' command if you'd like the card to keep its current status (and build on its current review interval on a correct answer).

I'd generally be against adding a pure 'reverse due date' ordering option because it doesn't really do anything to address the underlying backlog - you might have a sense of forward progress of sorts by reviewing recent cards, but that pile of overdue old cards will only keep growing and you'll never be motivated to do anything about it. Whereas an explicit command to get rid of the overdue cards will ensure the problem is addressed intelligently and you get back to a nice reasonable count of cards due.
 

Cookiemole

Member
Thanks for the lengthy reply! I haven't spent as much time as you thinking about these things, but I'll report my own thoughts and observations on the off-chance that you guys find it useful. My complaint now is actually pretty nit-picky, because I'm able to mostly get what I want by playing with time filters.

Regarding 'reverse due date' and the backlog problem, I would argue that as recent cards are promoted to higher scores, fewer and fewer of them will become due, so the program will naturally reach further back in time. In this way, the backlog would eventually be cleared. However, I do see the point you are making that when cards are more overdue, but not yet forgotten, they should be assigned more urgency because you want to catch them before they are forgotten.

I'm not sure I would want to use an additional mechanic for dealing with a longstanding backlog. To me, a more unified approach that uses the priority algorithm for everything seems to be cleaner. Rather than a 'give up' mechanic that cuts down on the number of due cards, personally I would prefer a system where there is no pressure to complete all the due cards, and if there's a day where I only feel like doing five minutes of flashcards, I'm still making progress.

Ultimately, the goal is to learn a set of cards using the least number of reps possible. That means maximizing the probability that the user responds to each card with "I got this one right". Cards that are almost but not quite forgotten should be accorded the highest priority, and cards that are so old that they've been forgotten should be consigned to the lowest priority. I think fresh cards that have just become due should be somewhere in between.

I like the concept of the 'overdue factor', because this becomes a measure of how likely a card is to be forgotten in a way that accounts for card intervals. Cards with shorter intervals have an overdue factor that grows more quickly, and it sounds like that is why the algorithm will favor them. However, returning to the argument I was making earlier, I don't think the priority should monotonically increase with increasing overdue factor. I think at some point it should start dropping because cards eventually become stale. I would rather focus on an almost-forgotten card than one that is almost certainly forgotten.

So, if priority is a function of overdue factor, I am proposing that it be a function that increases initially, reaches a maximum (maybe around 5 or 6 intervals overdue), and then starts dropping. I think cards that are way overdue should be lower than fresh cards, so the function should drop to below its initial value. Viewing things this way, my initial suggestion of 'reverse due date' is similar to priority being a monotonically decreasing function of overdue factor without the initial peak. I now see that is not optimal.

Also, on a different note, it would be nice if there was an option for cards that are manually added to "skip the line" and enter the set of the due cards. Psychologically, for me anyway, I find it more interesting to study recent words that I have encountered naturally, and to get some reps on them before I forget them. Words encountered this way might not be more important than those in the HSK lists, but they're more fun to study, so I feel like the thousands of due cards from word lists can wait.
 
Last edited:

mikelove

皇帝
Staff member
That priority 'bump' makes a certain amount of sense, I suppose, but I don't know where I'd feel confident drawing that line absent more data. My suspicion is that much of the literature about SRS (such as it is) wouldn't actually hold up in the less controlled environment of living / working in China, as so many of our users do - if you're being flooded with Chinese vocabulary outside of our flashcard app then it may well be that for many cards your recall rate has a pretty high floor and there's still a benefit to reviewing a card even a few weeks late.

Also, even if you dismiss the possibility of a boost to recall from looking at a card a month late, there's also the benefit that the card is still in our system and hasn't been given up on completely. Say that you have a 2/3 chance of getting a word correct even a month after its scheduled review date; assuming you do in fact get it right, the benefit might be small but it also didn't waste very much time, but more usefully, in the 1/3 chance you get it wrong, *now we know that* and can help you learn it again. Letting cards that have fallen behind remain behind more-or-less indefinitely means that even if you do forget them you won't find that out until you actually try to use them.

Also, as far as 'giving up,' I think there's a powerful psychological benefit to *not* having a large backlog, and also to not allowing yourself to become accustomed to having one. Flushing your review backlog (or whatever we end up calling this) is giving yourself a fresh start; now you only have 50 cards to review and after reviewing those 50 you're done for the day and don't have to feel like you're giving up. I don't think that SRS and 'study however many cards you feel like' are really compatible approaches; allowing the flexibility to only study X cards per day is something I'm reluctantly acceding to in 4.0, but I think we're doing our users a disservice if we don't at least make a token effort to keep them on schedule.

So making the decision to push back some cards explicit accomplishes three very important things: it keeps you feeling current / motivated, it keeps you tied to the SRS concept of studying cards regularly (it forgives but doesn't forget, as it were), and it allows us to factor in that these are now suspended / buried cards which we might want to bring back later when you have more time (rather than having them live forever in purgatory at the end of a 4000-card backlog).

The closest equivalent to what you're proposing among things we're considering would be an option to basically hide the interval portion of this altogether - it'll still be the same thing internally, more-or-less, still reviewing cards at increasing intervals, but on the surface it'll look more like our current 'weighted' algorithm in that you'll see however many cards you ask for and we won't pressure you or even give you any information about due dates. And by not claiming that it's SRS we can get away with having it behave less like SRS without feeling like we're lying to our users. We aren't actually keeping 'weighted' around, since ignoring the last reviewed time for a card is rather silly - there's no benefit to randomly studying the same card 3 days in a row if you keep getting it right - but you'll be getting that same behavior of our system picking cards that it thinks might be a good idea for you to study now without any sense of things being due or overdue.

Option to automatically add manually created cards to the pool makes a lot of sense - we do have a feature to force add cards now but it's not automatic.

(I should add that most of the actual code for the new flashcard system is already written - we're doing tests here quite happily now - but we've designed our new scheduler to be unbelievably flexible, both for refining our plans now and for revamping them more heavily in the future, and a big part of what we're working on right at this moment is refining our interval calculation algorithms and the settings that control them; that and the UI skin (waiting on the reveal of iOS 11 in June) are the two biggest remaining tasks now)
 
Last edited:

Shun

状元
Great discussion. @mikelove: Have you considered making the list of upcoming cards dynamically visible to the user, like a Play Queue in music players? The user could check what kind of cards are coming up, and if there are too many new cards, or too many repeated/"stale" cards they already know, make changes to the list of upcoming cards? Pleco could group those cards into chunks like 'the group of cards awaiting repetition from yesterday's studying session", or "the new cards scheduled for today", which could be moved closer or further back in time in just the way the user feels is right. These chunks could overlap, of course, since it's SRS, but they would have to be introduced in that order.

Generally speaking, I doubt that a non-deep learning, mathematically programmed system is always able to make the right card selection choices that best match a user's (emotional) inclination (his desire to learn is there, Pleco just can't tell what part of the vocabulary is the most active/most interesting to the user at that point in time), so the user's input, that is, a very efficient and intutitive interface with the user, may be crucial for maximizing a user's success, as well as an ability to change the list of upcoming cards within a session (even if this will actually open up a new session internally).

The ideal SRS for me also has to repeat a new card after two hours, then 8 hours, then 24 hours, then two days, then four days, and so on... Any mistake reduces the interval back to 2 hours, but then the interval increases more quickly. All of these cards ready for repetition could go into a Upcoming Cards chunk named (cards repeated right once, card repeated right twice), then the user could leave them as-is or order them based on what they think is best from their experience.

Personally, I agree with Cookiemole that a large backlog doesn't necessarily have any negative psychological effect. It should be clear to every Chinese learner that the road ahead is a long one, so if Pleco reflects that, it could even be motivating to have all those cards in the pipeline.

To move from the abstract to the concrete, I would need to know even more about the currently planned system to make specific wishes, though I think almost any system can be turned into reality, since it's just about ordering flashcards.
 

mikelove

皇帝
Staff member
I'd generally prefer not to override the order but rather just the ratio, since we're doing more intelligent test type scheduling now (trying to ensure each card gets a good mix of types - in a random 'alternating' system like we have now, 1/3 of your cards are going to get the same test type twice and 1/9 three times in a row) and combining that with an enjoyably random selection of types (not seeing 50 fill-in-the-blanks in a row before your first multi-choice) requires that we have at least a little bit of flexibility about the card order. We are planning to support the much-requested 'study all of this list in a set order' mode, but the main use cases for that are situations where you probably want everything self-scored anyway.

So I don't know if an actual queue makes sense (obviously don't want to show people the actual cards :)), but some ability to adjust the ratio of what you're about to see might. Maybe we'd just show you 3 counts for each profile, new / learning / learned, with the only options being to tweak the counts of those. Maybe add a 4th 'recovering' or some such for recently incorrect cards. We do also let you study multiple profiles simultaneously so you could potentially add some further wiggle room by mixing and matching those. I'd lean towards more intelligible classes like that over 'cards that I've answered correctly twice in a row' etc since I have a hard time thinking users will have different feelings about their twice-correct versus thrice-correct cards.

But a lot of how this all develops will depend on user feedback; we've built a pretty incredible core system now, and the goal for the initial 4.0 release is to expose as much of its awesomeness in as few options as we can get away with and then add stuff later based on how users respond to it. The current system is a decade old, but was never designed or intended to last a decade - this one actually is :)

(an example of futureproofing: we now not only log every flashcard answer [in detail], we also log every change you make to your test settings, so that if there's some factor that later turns out to be important like the order of your correct answer score rating buttons we can go back 3 years and find out exactly what those were like when you originally tested such-and-such card)
 

Cookiemole

Member
Doing SRS feels like I'm juggling lots of balls. These balls are constantly falling, and I'm constantly catching them and throwing them back up in the air. Fortunately, the balls are magical ones that fall slower and slower the more times I toss them back up. Eventually, the balls in circulation slow down to the point where I have enough breathing room to grab some new balls and throw them up too.

I think my usage pattern might be similar to some other users in that I'll go through periods where I am gung-ho and I don't mind catching and tossing at an incredibly high rate. Other times, though, I get slow or stop entirely and find out that I'm can't keep up with all the balls that I injected into the system during my more manic periods. Lots of balls end up hitting the ground because they're coming down at too high of a rate.

Now, if there are both balls up in the air and on the ground, what's the optimal order to attack the balls? I think what I was suggesting was to focus my attention first on balls that are about to hit the ground and to not worry about the balls already on the ground because they're not going anywhere anyway. When I can get the balls up in the air to slow down enough and things are less frantic, I'll worry about picking up the ones on the ground. However, the current flashcard version tells me to pick up the balls on the ground first because they've been waiting around for longer. But since I'm moving more slowly now compared to by gung-ho period, this means I'm throwing more balls up into the air before I'm ready and creating a frantic situation that I can't keep up with. The balls are already coming down too fast for me to deal with, and now I'm throwing more up in the air and making them fall at an even faster rate. Also, I'm saving fewer balls that are about to hit the ground because I'm diverting my attention elsewhere. Thirdly, the balls that I've just picked up and tossed up in the air are just going to hit the ground again because instead of focusing on them I'm busy picking more balls up off the ground.

What I would rather do is create a ordering function that tells me to deal with falling balls first and shifts grounded balls to a later position in the priority stack. When the balls in the air start falling more slowly I'll naturally get around to dealing with the ones on the ground, even if I'm moving at a slow pace. No matter how slowly I'm moving, eventually the balls in the air are going to slow down to the point where I have some breathing space to pick up the balls on the ground.

What exactly constitutes a ball that is about to hit the ground? I guestimated earlier that it would be one that was 5 or 6 interval periods overdue. But I would imagine a program implementing this concept would make this a configurable parameter.

The 'giving up' mechanic seems to be very similar in practice to deprioritization, but instead of leaving the balls on the ground you clear them off the field so you don't have to see them anymore, and then only bring them back when you're ready. But whether you sequence them later in the priority stack or you put them away for the time being, the result is essentially the same because you are dealing with fallen balls at a later point in time.

Anyways, I'm just relating my own impression and observations. I'll be perfectly happy using the program whatever shape or form it takes!
 

mikelove

皇帝
Staff member
Thanks - seems like we mostly agree here. The 'giving up' mechanic also appeals to me because it's easier to understand - people don't necessarily realize that they're getting the same effect with a reversed priority order, even if they are - so even if in the hands of an SRS expert the two systems end up pretty similar, being open / explicit about removing cards from study is much clearer for a novice user.
 
Top