New Scoring Algorithm for 2.0.2

[mikelove]

So I've been thinking a bit about ways to further streamline / improve the low-level flashcard system algorithms in 2.0.2, and it seems like we could really use an alternative to the Automated scoring system - not some fancy rules-based thing that requires a big complicated new UI, but just a different Automated algorithm (though we'd keep the old one around too, of course).

It's still very tentative, but here are some basic concepts:

Time-interval-based score changes - a major flaw in the current Automated algorithm is that the scores jump up far too quickly if you're a very heavy user of flashcards - with "once per day" turned off, a card's score could jump from 100 to 100,000 in a matter of days, and even with it turned on, if you see a card frequently enough to have it come up pretty much every day you're going to see cards essentially vanish from testing entirely after a week or two.

So in this new system, a score increase would be weighted based on the amount of time that's elapsed since the score last increased - if you just tested a card yesterday, the score might not go up by more than 10 or 20%, and if you saw it again the same day the score would barely go up at all. (but would go up a tiny bit, so we're not ignoring that information altogether like we do with "once per day" enabled) Likewise, decreases would be more dramatic if you hadn't seen a card in a while - a card with a score of 5000 that you'd just answered correctly yesterday might only drop to 3000 or 4000 when answered incorrectly, but if you hadn't seen that card in a couple of months the score could drop all the way back to 100. (this would also apply to difficulty factor changes)

Less dramatic frequency differences - while the variations in scores with the current algorithm are quite reasonable for repetition-spaced tests (see a card after 1 day, then 4, then 8 or 9, then 17, etc eventually going all the way up to 100 days or more), for frequency-adjusted tests they can be a bit too aggressive - brand new cards should show up a lot more often than older ones, but not 500 or 1000x more often. So either the scale of score increases would be lower under this new system, or the algorithm for frequency-adjusted tests would simply be tweaked a bit to lower the difference between card frequencies - a card with a score of 1000 might only come up 1/2 or 1/3 as often as a card with a score of 100.

More sensible difficulty changes - the decrease in difficulty factor for a wrong answer seems like it may be a bit too dramatic in the current system, so in the new algorithm, along with scaling the difficulty by time interval as discussed above, we'd probably also reduce the difficulty decrease for an incorrect answer (and increase it for a correct answer) - having the difficulty hit its minimum value after two or three wrong answers as it does now seems a bit too limiting, and taking 4 or 5 correct answers to make up for one wrong answer seems a bit off as well.

More customizability - along with publishing the details of the algorithm as we do now, we'd provide a screen that would let you fine-tune the constant values it used, giving you more exact control than with the current Aggressiveness setting - if you preferred the very big difficulty drops on incorrect answers in the old algorithm you'd be able to bring those back through this screen.

Anyway, that's what I'm thinking of so far. The reason this is a 2.0.2 rather than a 2.1 thing is that I really want to get it in the iPhone version and that'll be out before 2.1 - of course a nice added benefit is that Palm users get to enjoy it too, and since pretty much all of the hard work for this would be on the cross-platform engine side of things (where it doesn't really take any extra effort to support Palm other than a bit more testing) there's little reason not to make it available for Palm folks. UI-intensive features like rules-based scoring / a better multi-choice flashcard interface / better Manage Cards / etc will pretty much all come after we drop Palm support, but hopefully a lot of the under-the-hood changes like this can be done pre-2.1 and hence be Palm-compatible.

 

[ldolse]

That all sounds pretty good, wondering if anything would help out intermittent studiers? I like all the repetition based stuff conceptually, but when I have a hiatus of several weeks (or months, hate to say it) having all the cards back up for review is always dispiriting...

I can see the that perhaps the time based interval stuff tweaks could help here. If it's been a while since you've studied but you nail a card then perhaps the score should be increased by an even greater factor based on the time since the last session. That way when I see 大 or 人 come up yet again Pleco can give it a score of 50,000 or whatever so that it takes 6-12 months or so to pop up on me again (or at least till the newer cards have been studied).

The amount of time a student takes to actually answer a card could also be used as an input to weigh the scoring... Cards I know well are answered within 1-2 seconds, while ones I don't know take longer.

 

[marsch]

Not sure how the following would fit into your new design:
what bothers me most about the current repetition spacing system is that every now and then I get a huge number of old, but well known cards, because their time for review has come. Chances are, I won't manage to get through all cards that are due for review. Of course, that's no problem for the cards with the high score, but I'd also miss some of those with a low score.
So what I do now: set the score limit to say 2000, do all those cards, then do the rest.
Would be nice if within the repetition spacing session cards could either be sorted by score, or presented to me in an order like in a "frequency adjust" setting. That way I might still miss a few low score cards, but most likely I'll have reviewed those before I see the higher score ones.
Just looking for a way to review new cards not only more frequently (which rep spacing does), but also rather at the beginning of a session - so I could finish a session early and still not miss something "urgent".

 

[mfcb]

... I get a huge number of old, but well known cards, because their time for review has come. ...

know that problem too, but my approach is just contrary to yours, i make the limit >800 and get rid of the well-known cards first, because usually i just have to click them through, and the other cards with lower scores i will see anyway more often, if its today or tomorrow doesn't matter.


mike, heavy user of flash-cards is maybe not the right word (except you meant my body weight, hehe)... i guess its the application, my mistake was maybe that i used the automated scoring for "writing practice" that way it sure must have failed. anyway, the new Time-interval-based score changes sound very good. and with the additional fine-tuning screen we can find many new ways to create highly sophisticated scoring problems :lol:

 

[sthubbar]

Mike, it sounds great Thanks for supporting this on the Palm. I'm still holding onto the hope that there will be a big rally of other Palm users that can maybe convince you to keep supporting the Palm just a little longer.

 

[mikelove]

ldolse - interesting idea, I was more thinking about how this algorithm would help for frequency-adjusted than repetition-spaced study (in general I like the former better, both for intermittent users like yourself and for people "cramming" who don't want to be told that there are no more cards for them to study today) but I suppose it could be very helpful for this problem too.

Actually one very simple change that might help a lot with this would be to do score increases in rep-spaced relative to the *actual* interval since the card was last reviewed, rather than the *target* interval. So if a card has a score of 600 but you haven't actually looked at it in 15 days, and you remember it correctly, and the difficulty factor is such that the score would normally double after a correct answer, instead of increasing the card's score from 600 to 1200, we'd increase it to 3000. That would even work with the "old" automated scoring system, actually - really not much more than an extra checkbox and a line or two of additional code.

I'm a little uneasy about factoring in the time taken to answer a card because of the possibility of people getting interrupted, or of having their answer delayed for reasons that have nothing to do with how well they remember the card (e.g. a free-answer test where you're handwriting in a character and have to try two or three times before it's recognized) - we'd have to add an extra "override" button or something like that and that could get kind of confusing for people.

marsch - good idea about offering the option to sort rep-spaced cards by score, I think someone might have mentioned this a long time ago but I'm not seeing it in our feature suggestions file.

mfcb - good point, sorting scores in both ascending and descending order would make sense. We should probably make it easier to copy things like scoring settings between profiles, I'm sure you're not the only person who's accidentally created a new profile with Automated after previously using Manual.

 

[goog1e]

I'm a very heavy user of flashcards and my main daily sessions are about an hour a day (~200 cards) with SRS. I use the flashcards for probably another hour total with scoring turned off and set to "random" to learn my daily new cards (avg 10) and review cards from the last few days before dumping them into my main review.

For me, I like to hear/see a card every day (sometimes several times a day) for several days before I get the 4/9/17 (actually I set it to Aggressiveness 2 so I get 3/?/? jumps). You could argue that at this point I'm not reviewing the card but learning it, so that behavior is outside the scope of SRS -- that's why I review in unscored mode for so long before adding to Main review category. But after I "learn" a card, the jumps I see do feel a bit too great. I'm very interested in trying the time-interval-based scoring. I was going to ask about adding the "delay initial score increase until the card has been right several times" option from manual scoring to automatic scoring but the time-interval might work out.

Just in the past week or so I've begun paying attention to the difficulty scores of my cards. Seems like I can't get a card to score below 66 even if I give it a "1" -- and like 60+% of my cards have a 66? Many others are 77 and 90... I don't know if this is an issue with my setup (aggressiveness settings, scoring everything with 3s,4s) but it would be nice to get some score variation to further rank cards.

I'd be interested if any SRS pros that have experience with Supermemo, Mnemosyne, Anki etc could weigh in with suggestions while the issue is on the drawing board.

And since you mentioned flashcard management, it would be really nice to have a "merge categories" option and "flatten categories" (where all sub-categories are removed and cards are directly placed under the parent).

 

[mikelove]

With an aggressiveness of 2 the difficulty is capped at a minimum of 66 - any lower than that and the scores essentially wouldn't increase at all after a correct answer, which would be bad since you could have a card stuck forever at a given score (unless you gave it a bunch of 6es). Though it is pretty easy for a card to fall to that point, certainly.

Interesting point about seeing the card several times a day at first - the initial increase after the first correct answer would be tough to calculate in this system, since there'd be no time interval since the last review (only since the card was created, which is effectively meaningless for our purposes since people can create new cards long before they get around to studying them), and certainly there's a strong argument for not increasing the score too much after the first correct answer, though at the same time we also need to draw a distinction between new cards for words people already know and new cards for words they don't - perhaps the difference between a 4/5/6 score would be factored in more strongly early on.

 

[dhp]

Hey Mike,

It's my fist post so the first thing I wanted to say is thanks for the great job you're doing with pleco! I'm looking forward to the iPod Touch version as I would be really be happy to dump my windows mobile pda - the only reason I got it was pleco

Just wanted to add my 2 cents' worth to this discussion... I don't know if you've heard about SuperMemo - a program that's been around for 20 years now and is based on spaced repetitions. Their website has a lot of articles on how it (supposedly) all works. What I really like about their algorithm is that it's personalized - it adapts to the user. I remember that you could include their library into your own software to use their algorithm, but that was years ago and I haven't checked back with them if it's still available... or how much the licensing fee would be.

Here's their website: http://www.supermemo.com/

Hope you can find some inspiration there,

Last but not least - happy new year of the ox!

Daniel

 

[mikelove]

dhp - thanks! I'm glad to hear you're liking Pleco so far.

The "Automated" algorithm in 2.0 is actually similar to the algorithms used in SuperMemo and other similar programs like Mnemosyne - not quite as adaptable or powerful as the newer SuperMemo ones, but reflecting the same basic principles of adjusting the repetition frequency of individual cards based on how well you've remembered them in the past.

The problem with any SuperMemo-like algorithm, however, is that it requires you to review cards on their schedule rather than your own - if like ldolse you occasionally stop studying for a week or two, you can have thousands of cards pile up while you're gone, or if you simply feel like spending an hour reviewing Chinese vocabulary it can be frustrating to have the system tell you there are no more cards due for study today. There are ways of tweaking it to work around both of those problems, but doing so makes the algorithm less effective since you're not studying cards at the mathematically-determined correct intervals.

There's also the problem, and this has bugged me about SuperMemo for a while now, that the algorithm can't factor in times when you see a vocabulary word outside of the software - if you hear a word a lot for a couple of weeks, you might remember it better than you naturally would otherwise, so it would see its repetition interval grow more than it should (and continue to be high, even after you'd answered it incorrectly, on account of your previous history of correct answers) and the overall system might interpret this as meaning you were learning words faster than you actually were because you were getting extra practice with some of them outside of it.

I'm not exactly an expert in this cognitive science stuff, though, so I may be way off on some of this, but in general I'm feeling lately that it's more important that a system be easy to understand and capable of accommodating a wide variety of learning strategies than that it use a lot of complicated learning algorithms - it should help you study in whatever way you want to rather than insisting that you use a particular strategy / review a particular number of cards per day.

 

[ldolse]

ldolse - interesting idea, I was more thinking about how this algorithm would help for frequency-adjusted than repetition-spaced study (in general I like the former better, both for intermittent users like yourself and for people "cramming" who don't want to be told that there are no more cards for them to study today) but I suppose it could be very helpful for this problem too.

I guess I don't neccessarily understand the pros and cons and exact behaviors of each of the scoring/test methodologies. I'll try out automated and frequency-adjust and see if I like that any more over repetition spaced....

one very simple change that might help a lot with this would be to do score increases in rep-spaced relative to the *actual* interval since the card was last reviewed, rather than the *target* interval. So if a card has a score of 600 but you haven't actually looked at it in 15 days, and you remember it correctly, and the difficulty factor is such that the score would normally double after a correct answer, instead of increasing the card's score from 600 to 1200, we'd increase it to 3000.

That sounds good, pretty much what I was thinking of.

I'm a little uneasy about factoring in the time taken to answer a card because of the possibility of people getting interrupted, or of having their answer delayed for reasons that have nothing to do with how well they remember the card (e.g. a free-answer test where you're handwriting in a character and have to try two or three times before it's recognized) - we'd have to add an extra "override" button or something like that and that could get kind of confusing for people.

I wasn't neccessarily thinking that someone should be penalized for a slow answer. If you get the answer right then the normal score increases apply, but when it comes to the time factored score increase discussed above, I think that the time taken to answer the card could come in handy. A card answered quickly is one that the time factored score increase definitely applies to, but it often won't apply to one answered slowly. I think it's a minor nit to get a lower score than I could have on a card I know well, it just means it will come up again sooner. On the flipside it's more of a disservice to give I card I really need to study more a much higher score because I managed to get it right after a bit of hemming and hawing.

 

[mikelove]

ldolse - frequency-adjusted basically just means that the card scores translate to their probability of coming up in a session instead of the interval between appearances - in repetition-spaced, a card with a score of 800 might come up every 8 days while a card with a score of 200 might come up every two days, while in frequency-adjusted, a card with a score of 800 would be four times as likely to appear in any given session as a card with a score of 200. You configure a specific number of cards to test (or "Endless," in which case the system keeps testing you until you tell it to stop) and it chooses which cards to show you based on those frequencies.

The biggest issue with that approach is that it's subject to random variations - just by bad luck you might study a new card once and not see it come up in a session for a week or two. It all should even out in the end, but if you're on a tight schedule of tests / quizzes / etc you might not have time to wait for it to do so. Which makes me think we might want to consider a hybrid approach - something which, say, prioritizes cards that are "due" for study according to their rep-spaced intervals first, but then lets you keep going with frequency-adjusted after you run out of those.

Good point about time-based increases just for very fast answers - I still think this would mainly be useful in self-scored sessions, but it might have enough benefit in those to be worth considering as an option at least.

 

[sthubbar]

Yes, the idea of time based adjusting of difficulty is interesting. It could do something like if it takes 30 seconds or more, the score doesn't get adjusted because the system assumes you got up to do something else or were otherwise distracted, or maybe it could prompt in those cases with a button, to confirm that it really took that long to remember the answer.

FYI, my current repetition spacing system is manually adjusting the scores. I have it set to increase the score by 60% if 1 correct in a row and decrease the score by 60% if 1 wrong in a row. I know the decrease is more than the increase, and that is what I want. It closely follow what I was doing with Pleco 1.0.

The thing that would be nice for me to be able to do is to use difficulty to adjust these parameters. Right now difficulty is not factored at all. My initial thinking is that if I took the above values and then took into account the difficulty to calculate the increase/decrease. So for the easier cards, the increase would be greater and decrease would be less and for harder cards the opposite.

I guess the suggestion here is that in manual scoring it would be nice to be able to also factor in difficulty.

 

[mikelove]

sthubbar - That will happen with rules-based scoring, I think, something we're contemplating for 2.1 or 3.0 - it would let you define an arbitrary set of formulas / actions for card score changes, which could affect difficulty and score and probably factor in those plus some other factors like time intervals.

 

[dhp]

thanks! I'm glad to hear you're liking Pleco so far.

Liking it is not the right word here, Mike, I'm loving it! It took me years to finally realize that no Besta dictionary is going to be what I'm looking for - for one simple reason which is: they're made by native Chinese speakers for native Chinese speakers. Therefore from my point of few they s**k... are not that good that is :wink:
The ABC dictionary is awesome (and it's a great loss to the world that the author - John de Francis just passed away a few weeks ago), and then there's the monolingual dictionary - something that I've been waiting for for quite a while as well. Pleco rocks! Thank you again!

The "Automated" algorithm in 2.0 is actually similar to the algorithms used in SuperMemo and other similar programs like Mnemosyne - not quite as adaptable or powerful as the newer SuperMemo ones, but reflecting the same basic principles of adjusting the repetition frequency of individual cards based on how well you've remembered them in the past.

I'm not going to try to talk you out of working on your own spaced-repetition algorithm here, as I'm in no position to tell you what do you and what not. But I certainly want to stir up a bit of a discussion here.

I've done some homework lately and discovered that apart from SuperMemo and Mnemosyne there's another piece of software that has drawn my attention: Anki. It's basically based on the same algorithm,
but the advantage it has is that it's not only multi-platform (Windows/OS X/Debian/iPhone/Windows Mobile and a few more versions are all available) as Mnemosyne is, it's also open-source (=you can compile/modify it yourself) and - what's most important to me - you can synchronize between different devices over the internet. You could be working on your database at home on your Mac, but do your repetitions on your work PC or on your commute back home on your mobile device... do your reading in the library and add new vocabulary to your flashcards and then review them on your home Mac once you're back in the evening. The possibilities are countless.

So my big question is - what if... which is a big if... we could at least export our pleco flashcards to use them with Anki? Or do incremental exports - only whatever we add to pleco's flashcards since last export gets exported. Or maybe one day we could synchronize the databases? Having them on different platforms would give the users much more flexibility with their studying.

The problem with any SuperMemo-like algorithm, however, is that it requires you to review cards on their schedule rather than your own - if like ldolse you occasionally stop studying for a week or two...

Haven't done that much homework yet... not sure how Anki or Mnemosyne solve that problem. SuperMemo has some new functionality here... but I haven't used it in a while and don't remember anymore what it does exactly.

There's also the problem, and this has bugged me about SuperMemo for a while now, that the algorithm can't factor in times when you see a vocabulary word outside of the software...

I don't think that any software can solve that problem, but on the other hand that will happen with only some of the words so on the average the influence shouldn't be that big.... I'd say that the "fuzziness" of the algorithm would probably accomodate for that... which is a a rough guess only, I do admit that I have no real data to support either of my points here.

I'm not exactly an expert in this cognitive science stuff, though, so I may be way off on some of this, but in general I'm feeling lately that it's more important that a system be easy to understand and capable of accommodating a wide variety of learning strategies than that it use a lot of complicated learning algorithms - it should help you study in whatever way you want to rather than insisting that you use a particular strategy / review a particular number of cards per day.

I think that that's where the point is - none of us are experts here, it's just what we think feels right. Please don't get me wrong here, but if someone is an expert (=Mr. Wozniak - the author of SM's algorithm), why not use their expertise and concentrate your own effort on other functionality of pleco that nobody else can solve for us? And (hopefully nobody gets angry at me for saying that) you're our GOD here. At least your MY GOD as I don't remember much from my programming classes I took back at university a decade ago. Whatever needs to be improved in pleco - if you don't do it - nobody else will.

These are just my humble opinions, please do not take them too personal, but I do hope that you can consider them and tell us what you think.

Thank you for your time and again - for the great job you're doing with Pleco.

Daniel

 

[HW60]

Pleco 2.0 was a great help for me to learn quite a lot of new words in the last months. I do not want to spend too much time with changing and testing algorithms (but learn more words ...). Repeating of "old" hanzi and new shengzi is of equal importance. I would prefer a very personal method to skip or repeat words for the near future: just two buttons SKIP and REPEAT after Pleco shows the results of my answer. With SKIP I can postpone the next appearance of a word for some time period (a week or so), with REPEAT I would like to see the word again in the near future. Pressing the 答-button in the bottom right does the same as before. I would hope not to see the cards for a while that are boring me because I know them (but the algorithm does not know that), and would appreciate to see cards which for some reason have a very big score and do not show up since weeks, though I forgot them in the meantime.

The time-interval-based score changes may just do that - without testing it seems to be difficult to predict. It should cover three problems: a regular increase of the number of flashcards (e.g. 10 per day), a new start with a big number of flashcards (e.g. 1000 after changing the scoring file) with subsequent increase of new flashcards, and no increase of flashcards at all. When I switched my scoring file in December because I was unhappy with the flashcards presented to me, it took quite a while until Pleco behaved as expected. With the above somewhat brute force method Pleco would (probably) do what I want ...

 

[mikelove]

dhp - exporting flashcard data is a tricky issue copyright-wise; we can remap them to free dictionary entries when they're exported (as we do now) but using licensed dictionary data in flashcard exports can be a problem with some dictionaries. Improved synchronization in general is certainly something we'd like to work on, though, likely with a significant online component to it. One problem at the moment is that nobody's really settled on a consistent system for keeping track of score / learning data - everybody tracks different things, so there'd be no good way to move that sort of data between Pleco / Anki / SuperMemo / Mnemosyme / etc. Since Anki and Mnemosyne are open-source, I suppose it might be possible for us to come up with an "emulation mode" for them which would import / export the exact same score data (thanks to our use of "scorefiles" we're not really tied down to any one scoring system) and change / interpret it the same way, though I'm not sure whether that would really be worth the effort.

As far as SuperMemo and expertise, SM may be a great system for some people but I really think for Pleco we need something a lot more flexible - the mathematical brilliance of the algorithm isn't even really as important as making it easy-to-use and highly customizable. A lot of the possible combinations of settings in Pleco 2.0 don't actually make a whole lot of sense together, so a big goal for this new algorithm would be to have it work better for a wider range of study methods with little or no customization - hence ideas like varying the score change based on how recently the card was last tested, which would hopefully save people from having to manually tweak the aggressiveness / points-per-day settings so much. Licensing / embedding SM in Pleco would be lovely for people who like and are willing to stick with their approach, but for people studying more casually or people who prefer to decide for themselves how many cards they're going to study today it could seem frustrating and inflexible.

Flashcards weren't even part of my original vision for Pleco - I've never been a big fan of them myself - but as they've become an increasingly popular / essential part of it over the years I've come to appreciate that they're really a very interesting design problem, and one that nobody's quite figured out yet. It's not that nobody else can solve these problems, but somebody else could come along and design a better Chinese dictionary too - as long as we can solve a problem in a unique or interesting way I think it's well worth taking a crack at it.

HW60 - good idea about encapsulating those score changes in simple buttons; you can sort-of do that now with the optional Score button but putting it front and center could certainly help, both for ease of access and ease of understanding. The "Limit Unlearned" option in 2.0.1 is designed to help somewhat with the 1000-new-cards-at-once problem, though more could certainly be done on that.

 

[sych]

The problem with any SuperMemo-like algorithm, however, is that it requires you to review cards on their schedule rather than your own - if like ldolse you occasionally stop studying for a week or two, you can have thousands of cards pile up while you're gone, or if you simply feel like spending an hour reviewing Chinese vocabulary it can be frustrating to have the system tell you there are no more cards due for study today. There are ways of tweaking it to work around both of those problems, but doing so makes the algorithm less effective since you're not studying cards at the mathematically-determined correct intervals.

With the addition of the "difficulty" factor in Pleco 2.x, I've ceased seeing this as a problem. If you're getting the word right more often in the flashcards session because you're getting outside practice, then it just means that the flashcards program needs to test you on it less. This is automatically accomplished through the scores and difficulties. If you stop getting the outside practice, you won't get the flashcards right so often, and the score and difficulty will adjust accordingly.

So to summarise, I think that seeing a word outside of the flashcards system just makes it "easier", and the system already accommodates that in a way that I'm happy with.

 

[sych]

I wouldn't mind seeing something like a "forgetting factor" available. I believe that in the newer versions of the SM algorithms, there's an extra factor in the algorithm which basically takes into account the fact that you're never actually going to remember absolutely everything, and allows you to slip up a bit.

The currently flashcards algorithm does its damn best to make you recall 100%. If you just can't remember everything perfectly, it'll keep bugging you perpetually until you do. Sometimes, I'd be happy with 90% or 95% percent recall across my vocab... but currently, using repetition spacing, it doesn't "let" me forget anything. The SM guys, I believe, think that this is "bad" and we have to be "allowed" to forget sometimes. Personally, I'd just like the option of being able to slack off occasionally, so if I could tell the system I don't mind being a bit hazy on the most difficult vocab, I'd do that.

I think I could probably accomplish my "slacking off" goal if I switched to frequency-adjust, but I'm not a big fan of that mode. I prefer the system telling me how many cards I "should" study on a given day, and have it manage the intervals precisely rather than by random number adjustments.

 

[mikelove]

Something like that is a possibility, but I'm not sure if it makes sense to do it automatically - maybe a "bury" button instead, something that let you tell Pleco that you wanted to stop working on a specific card for now. With an automated system, along with having a bit of that same random uncertainty as frequency-adjusted, we'd have to come up with a way for people to tell Pleco to emphasize certain cards (and always insist on studying them), which means a whole lot of extra configuration work on the part of our users - a bury button is a lot easier for people to get into.