Live OCR Crosshair mode

tarakara

秀才
Hi,

1. How do I enable Crosshair mode on Android for Live OCR?
I use it all the time on my iPhone but can't find the option on Android.

2. Is there any difference between Pleco Live OCR on iPhone and Android?
I pointed at the same text and iPhone captures more accurately than Samsung S10.

Btw, Google Translate and Naver Papago OCR is amazingly accurate.
Both manages to capture Chinese subtitles on talk show videos where the text is blended on multiple different scenes and a complete article on HSK reading website accurately. Would be great if it can be done in Pleco OCR too.
 
Last edited:

mikelove

皇帝
Staff member
There's a crosshairs icon in the toolbar you can tap on to toggle into / out of crosshairs mode.

As far as capture accuracy, no, it's the same algorithm. Does it work more consistently with still images? Does the video in the camera generally seem to be clear and in focus? Have you tried the "New OCR" option in Settings / OCR?
 

tarakara

秀才
There's a crosshairs icon in the toolbar you can tap on to toggle into / out of crosshairs mode.

As far as capture accuracy, no, it's the same algorithm. Does it work more consistently with still images? Does the video in the camera generally seem to be clear and in focus? Have you tried the "New OCR" option in Settings / OCR?

On iOS, with crosshair pointed at the text, characters appears centered on the crosshair and it only captures text on the horizontal line so text on the next line or above won't be captured.

On Android, with crosshair pointed at the same text, characters appears only on the left side of the crosshair instead of centered like iOS and sometimes it captures text outside of the crosshair horizontal line.
With iOS crosshair, it usually captures text instantly while Android crosshair mode keeps showing random characters.

For the New OCR on both iOS and Android, accuracy is slightly better.
But with New OCR enabled, on iOS the settings icon to change modes on Live OCR is gone so there's only box mode and on Android, the crosshair appears to be fixed inside the box.

I tried Still OCR but it doesn't work as well as Live OCR. There's also a bug on Android Still OCR. After taking a portrait photo or uploading a portrait image, the Still OCR image is shown in landscape while the toolbar is shown in portrait mode.

The video in the camera is very clear and in focus.
I tried Google Translate and Naver Papago with the same phone and it captures Chinese subtitled text blended in with changing scenes on videos and long hskreading articles shown on Dell monitor very accurately and pretty much instantly.
 

mikelove

皇帝
Staff member
With New OCR on Android, the crosshair is fixed inside the box, but that's just confining the area it searches for text in to speed up recognition a bit. If you make that box as large as the largest word you might want to fit inside the box, does it then work reliably? (crosshair in non-"New" OCR is kind of glitchy and to be honest once we added the New OCR option we didn't see much reason to keep working on it, on the theory that anybody who wants that functionality can just use New OCR)

For still OCR, I'm not seeing this bug here - maybe the orientation is not getting reported to the app correctly? Are you opening the photo from the file picker or are you getting it into Pleco some other way like sharing it from another app? Does it work better if you open it from the file picker?

As far as clearer recognition + text detection, this is also in the works - the "New OCR" was actually in large part about laying a foundation to support machine-learning-based OCR (which is at least an order of magnitude slower than what we use now) while still offering it 'live,' basically by doing a lot more analysis before recognition so that we could get away with actually recognizing text less often, but we put the rest of that work on hold while we were waiting for phones to get fast enough; they've hit that point now, we just need to go back and finish the job of getting newer OCR libraries into Pleco.
 

tarakara

秀才
The New OCR accuracy is better as characters shows up more often compared to old OCR where characters keeps changing but it's still not very accurate for certain scenarios.

For example, it can do quite well on https://hskreading.com/have-time-to-play/ but on longer articles like https://hskreading.com/civilized-society/ most characters can't be captured or inaccurate even with box enlarged while Google and Naver translate manage to capture all characters with 100% recognition.

New OCR requires the phone to be very close to the monitor screen to capture characters compared to Google and Naver translate which can be taken from afar so more text can be captured.

The New OCR also doesn't work accurately on Chinese subtitles and text in videos.

I noticed in landscape mode, the top toolbar is not at the extreme top position on iPhone and the bottom toolbar is very large on both OS so it blocks most of the capturable area.

For Still OCR, I used Pleco built-in take picture and open image gallery option in portrait on Samsung phone and tablet but it always turn up as landscape image and portrait toolbar. No change even after settings reset. No problems on the iPhone.
 
Last edited:

mikelove

皇帝
Staff member
Yeah, in general text on computer monitors isn't a use case we spend all that much time optimizing around either - I know some people still do like to use a phone for that, but I feel like nowadays most people would rather either use a mouseover dictionary on their computer or read the article on their phone. If you open that article on your Android phone you should be able to get very good accuracy out of it with our Screen Reader function, or you can just share it with Pleco Reader and tap-lookup characters in it directly.

We would like to do better on subtitles, but that requires the algorithmic improvements I mentioned; this may also have the side effect of improving recognition on monitors, but I'd rather focus our attention on supporting article reading in other ways. (including a Mac app and possibly also a PC app)

The still OCR issue might be something particular to your phone and its camera app, since I still can't reproduce it here - could you let me know the full model number (which variant of the S10 is this) and your Android version / firmware version?
 

tarakara

秀才
The articles earlier are just to test OCR. As you mentioned, Web extensions, Screen or Document reader works well enough for articles on plain text.

Main use case is capturing Chinese text blended in on Videos and Images on PC monitor and Phone Tablet screens where Web extensions or Screen reader wouldn't work.

Another use case is Chinese text on apps etc. and while Screen reader cast works ok for computerized fonts but it's much quicker to use another phone to capture the screen while browsing the app and currently Pleco OCR isn't accurate enough for text blended in on Videos and Images. Handwritten text also isn't accurate.

Clipboard reader works well so it's possible to paste from Google translate. It does requires quite a few clicks though if you have to translate multiple times at once.

The S10 model is SM-G973F on Android 12.
 

mikelove

皇帝
Staff member
Thanks. Actually Macs and iPhones now have OCR built into their image viewers, so if you drag over Chinese text in an image it'll treat it as selectable and let you copy/paste it like any other text; I expect this will be coming to Windows and Android as well in the not too distant future. Videos are dicier though.
 
Top