Experimental Database-Improved Palm Version

mikelove

皇帝
Staff member
Posted here:

http://www.pleco.com/beta/pd2b5-palmdbtest.zip (Palm OS only)

This incorporates a number of improvements to the flashcard and user dictionary database system on Palm, which should both improve stability and reduce file size. It also adds a Stop button to the flashcard import screen, along with improving the import screen's responsiveness to Palm OS system events, so you should be able to stop in the middle of imports (it may take a few seconds to wind down) and you should no longer have to wait several minutes after an import finishes for your Palm to become responsive again (or have it soft reset on you). It also includes some newly-accelerated code for the inner loops of two big flashcard functions (Manage Flashcards searches and new sessions), which should make those run significantly faster - simple searches like All Flashcards or a Random (as opposed to Frequency-Adjusted or Repetition-Spaced) flashcard session should be almost instantaneous, even with 10,000+ cards.

There's also another experimental version included in this archive, named Pleco 2-vfs.prc, which in addition to the other changes stores flashcard databases directly in a Palm's internal flash memory instead of as a regular Palm database. (this only works on Palms using flash memory for file storage, i.e. those running Palm OS 5.4 or later) This can result in significant performance improvements, especially for write-intensive operations (like flashcard imports), but has not been as well-tested as the regular database system. It'll only create the flashcard database in that internal store if it doesn't find one anywhere else, so make sure to remove your existing flashcard database before launching it. To access files in that internal flash memory, install FileZ and enable its "Show Hidden Volumes" preference option and the internal store will appear like any other Palm volume - you'll see the flashcards listed there as PlecoFlash.pqb. (it's a standard SQLite format file, so you can open it in a SQLite-management utility like this, something not possible with regular Palm databases) If this works well for people we'll make it an option in the official version, along with providing a utility to move flashcard databases between the two locations.

(incidentally, a quick way to check whether a flashcard database is corrupted is to open it up in a utility like that and execute the command "pragma integrity_check;" - won't catch every possible type of corruption, but we've never yet encountered a corrupted database that didn't result in an error in that check)

To try out one of these experimental versions, simply install it over the regular Beta 5 application and it'll replace it. The version will appear in the about box as 2.0b5(pf) for the improved version without the new internal memory storage, and 2.0b5(pf-vfs) for the improved version with it. There's a small chance that the changes we've made might cause corruption in old flashcard databases, so we strongly recommend backing up your flashcards / user dictionaries before installing this.
 

mikelove

皇帝
Staff member
Update for people having problems on the TX (or other flash memory Palm models)

We've done some experiments with our defective (faulty flash memory = frequently corrupted databases) TX creating/accessing flashcard databases off of an SD card instead of (either form of) internal storage, and it actually seems to work pretty well - imports are a bit slow starting off but get much faster after the first few dozen cards. And the databases don't seem to get corrupted at all. So we're now recommending this as a tentative workaround for people with corrupted-database issues.

The reason I'm posting this here is that it's much easier to try out SD-card-based flashcard storage using this new experimental version than the old one, and it should be significantly faster / more stable too (since the code for the internal flash storage feature also affects SD cards). To try it out, install/run the internal storage experimental version (the one with -vfs after its name) with no flashcard database installed; that will create a file called PlecoFlash.pqb in that hidden internal drive. Use FIleZ as described in the original post to view that internal drive, and move PlecoFlash.pqb from there to your SD card (making sure there's now only that one flashcard file in any location). Launch Pleco and it should use the flashcard file off of the card; you can then proceed to import your flashcards again and see if they now come through uncorrupted. After the import finishes, go back onto the card with FileZ and verify that that PlecoFlash.pqb file has gotten bigger (confirming that you didn't accidentally leave a flashcard file in another location which Pleco found instead)

Small note on this: make sure NOT to rename PlecoFlash.pqb to anything with a longer name (same goes for any other flashcard / user dict database) - the names were carefully made as short as they were so that the journal files (which SQLite creates automatically by adding -journal to the end of the file name) would not have their names truncated (and hence be inaccessible / not found by SQLite when reopened).
 

ipsi

状元
Interesting. I was going to download this last night, but got free textbooks instead. And then found HanziSmatter... I'll probably grab it when I get home tonight.

EDIT: If storing the file on the SD Card, must it be placed in /PALM/Launcher? Or can it go anywhere?
 

mikelove

皇帝
Staff member
It can go in any directory that Pleco normally searches for files. We just found a bug in this that can cause frequency-adjusted sessions to be screwed up (basically they end up always showing the same cards every time), so I'd suggest avoiding those while trying this out.
 
Thanks Mike, this appears to have solved the problem...there are still some discrepancies in my extremely long lists (4006 cards turned into 3767 cards), but I bet that it is just duplicates.

I was trying to find that "pragma integrity_check;" or a place in FileZ to input a command, but couldn't
 

ipsi

状元
I'm honestly not sure if this is any faster for me with regards to imports despite using the VFS version. Regardless, I can now actually make a selection on the Flashcard Duplicates screen, which is very nice! :). I've now fixed, I think, the majority of the duplicates in my NPCR database (and I'll upload a new one soon). A couple of things strike me about that, but I'll post those in the regular feedback thread.

Had a quick look at the SQLite DB - very straight-forward :D.

EDIT: Starting a flashcard session is much, much faster.
 

mikelove

皇帝
Staff member
Fleminator - glad to hear it! Integrity_check only works if you copy the database to your desktop and open it in a SQLite database viewer program like the one I mentioned - another standalone one is here, it uses a somewhat out-of-date version of SQLite but works fine as a viewer program at least.

ipsi - the speedup actually seems to be a lot greater on the TX than on Treos for some reason - perhaps the TX uses slightly faster flash memory. Glad it's improved in other ways at least.
 

ipsi

状元
Got some strange issues with the lists - try searching for 面 in a dictionary, as that shows it for me. What happens is that the top item in the list seems to take up like 1.2 times as much space as normal, so the other entries get pushed down and it all looks kinda weird.

Dictionary lookup also feels a fraction slower, but that could just be my imagination.
 
One thing I'm discovering that I don't like about having the flashcard database on my SD card is that if my card is accidently ejected or the palm doesn't recognize it for some reason, then Pleco creates a new score file on the internal-hidden memory, which I then have to go and delete...Its only happened once so far, but it also has been less than 24 hours.
 

mikelove

皇帝
Staff member
ipsi - the list thing was a transient bug in the particular build that version was based on, we were trying to fix the weird behavior where the spacing between items gets a bit narrower at the bottom of a list; fixed now. I can't think of any reason why dictionary lookup would be slower unless you'd created some user dictionary databases where there previously weren't any.

Fleminator - two things should make this better in the next release: first, it now opens whatever flashcard file was created earliest, rather than the one that was created latest (dumb mistake on our part to have it using the latter), so once you reinsert the card Pleco should go right back to using the one on the card. Also, we've whipped up a nice little User Database Mover utility which you can use to easily delete those accidentally-created files (along with moving them between locations (including conversion between pqb and pdb) and marking them as "backup-only" so Pleco won't automatically open them).
 

sych

榜眼
Hi Mike,

Even though the database operations seem to be a whole lot faster (a welcome improvement), I'm finding my device (Treo 680) to be a fair bit more unstable under B5(pf) than under plain-old B5. When installing, I simply replaced the existing Beta 5 Pleco2.prc with the new "pf" one, I did not export or re-import databases, etc. Was that correct?

If I want to revert to the original B5, can I simply copy the original PRC file back over the top?

To elaborate, my phone is crashing (soft resetting) whilst using Pleco, even sometimes crashing on the main dictionary screen when I press the Home, Messages or Phone buttons to exit. I didn't notice this under the original B5. Also, I did find that on the original B5 I was sometimes getting crashes when pulling up or exiting from Instant Access... it seems a bit worse on B5(pf). I am also getting more frequent crashes when not in Pleco, e.g. when trying to make a call, receiving an SMS, etc. Crashes were nowhere near this frequent before installing B5(pf).

FYI I have enabled instant access with "Experimental" mode (I find "Experimental" is the only mode that can reliably pull text out of Blazer and from MMS messages), and I'm also using CJKOS. I have not yet had the chance to find out if my crashes go away if I turn off instant access - I might try that today.
 

mikelove

皇帝
Staff member
That was the correct procedure, yes. Have you tried deleting your flashcards / user dictionaries and then installing the pf-vfs version? That might help matters - the biggest cause of soft-reset-on-exit errors that we've seen has been an issue with the Palm OS's flash memory database system not getting along well with our flashcard files, which the vfs version is designed to avoid (by circumventing that Palm OS system altogether and writing data directly to flash memory). Reinstalling the original PRC should help, though since most of the changes from the pf version will also be in Beta 6 it would probably be a good idea to try to get things working with the new system if possible.
 

sych

榜眼
Thanks Mike.

I'll persevere with the pf version for now - in fact, I've had 2 days now without a crash, so I don't know what's going on. A few days ago it was a crash every hour or so. If I find any pattern or if it starts crashing again, I'll let you know.
 
Top