Screen OCR + Screen Reader

mikelove

皇帝
Staff member
#1
New Android features we were originally planning for our 3.3 update but have now bumped up to 3.2.13 and will have out in beta by the end of the week: Screen OCR + Screen Reader. Both designed to let you instantly look up text anywhere on your Android phone via a floating button; the former using Android 5's new MediaProjection APIs to take a screenshot and OCR it, the latter using Android's Accessibility APIs to capture a copy of all of the text on the screen and present that in a reader window.

With either one you're basically tapping on a button to capture the current screen and then tapping on the place in the text you want to start reading; OCR gives you a more seamless transition but will occasionally get a character wrong, while Reader takes you a bit more out of your current context (though we're trying to match up character locations as closely as the Accessibility APIs will allow) but is more accurate.

Screen OCR will unfortunately require Android 5 or later, but will be free for anybody who owns our OCR add-on. Screen Reader will be free for everyone (even if you haven't bought a single add-on) and will theoretically work as early as Android 4.1 - we've successfully extracted text from WeChat on 4.1 at any rate - but inconsistent accessibility support in earlier Android versions (doesn't really work in the old Browser app, e.g.) may mean it's mostly useful on the latest ones. (and the green characters in OCR can be hidden, as in our other OCR modes - we're just making them visible here so you can see what the app is doing)

There's no direct equivalent to either of these APIs on iOS, but between iOS 9's integration of "Share" into the text selection menu and the new App Extension we're working on, we're going to have a much faster way to look up text on iOS shortly as well.

device-2015-07-06-003847.png device-2015-07-06-003906.png device-2015-07-06-003919.png
 
#5
Just wanted to try this out but I don't know how to get the button to appear.
I have the right version. Do I need to set any option for this to work.

Galaxy Note4 Android 5.0.1

Thanks.
 
#7
This Screen Reader feature is so awesome.

But I can't seem to get it to work with Anki or Firefox. Would be awesome if I can use anki deck with Pleco's ABC dictionary lookup.
The screen reader works if I go to the card edit screen on Anki, but doesn't recognize characters on the card review screen.

Right now i'm using Android 4.1.2.
Do you know what version (if any) will allow Anki/Firefox to work with this feature?
I don't want to upgrade firmware unless I know this feature will work with these apps.

Or are there any workarounds? (ie. using different fonts, etc.)

thanks
 
Last edited:

mikelove

皇帝
Staff member
#8
Thanks!

You'd need Android 4.4 to get Anki working - there was no accessibility support in older versions of Android's embedded web browser (which is what Anki uses to render stuff).

Firefox I'm less sure about, haven't tested that much yet - Chrome should work even on 4.1 though.
 
#9
Both designed to let you instantly look up text anywhere on your Android phone via a floating button
I don't have an Android phone so I can't see this in action. Just to verify: the floating button would always be present regardless of what application your using? This button is super useful when you need it, but I can see how it might be distracting to have it on the screen when you don't. Would there be a way to call up the button via a shortcut, special button press, etc. (perhaps using a program like Tasker?)

There's no direct equivalent to either of these APIs on iOS, but between iOS 9's integration of "Share" into the text selection menu and the new App Extension we're working on, we're going to have a much faster way to look up text on iOS shortly as well.
I'm super jealous of this Android functionality - will the semi-equivalent functionality be coming to iOS anytime soon? :)
 

mikelove

皇帝
Staff member
#10
There's also an option to trigger it with a notification now, or a shortcut intent.

Actually working on a couple of App Extensions; along with that "Share" one we think we've come up with a way to do clipboard monitoring via a Today widget (actually somewhat more nicely than on Android, due to iOS' greater user control over notification display and its wider range of supported UI elements). What we're basically working on now is the "take advantage of all of the cool stuff Apple has added in the last 2 years" update, akin in a lot of ways to what we were aiming for with last summer's Android ones.
 
#11
we think we've come up with a way to do clipboard monitoring via a Today widget
Will the first step in the workflow for all this new functionality require manual text selection by the end user? I use a handful of apps that don't provide text selection controls. It's very frustrating but I don't think there's much that can be done about it.
 

mikelove

皇帝
Staff member
#12
Yes - do those apps offer any other way to copy text?

This too seems to be no better on Android, FWIW - quite a lot of apps are intentionally opaque even to the accessibility framework we use for Screen Reader, Amazon Kindle for example.
 
#13
Yes - do those apps offer any other way to copy text?
Nope. They are mostly news apps or news aggregation apps. I just think it's lazy programming. In this specific case the Android screen reader functionality sounds ideal, but I can usually access the web content directly, or if I'm really desperate take a screen shot and use Pleco OCR.
 
#14
Why there is no button - quick select all?
Cause when i ocr some image with pleco screen reader, i have to press second from right arrow to select all recognized chars,
why there is no "select all green" ?

(i need it for copying)
 

mikelove

皇帝
Staff member
#15
Haven't had any requests for it - we can look into doing it with a long press. We did just add support for that to the non-OCR Screen Reader recently, so with that one at least, if you long press one of the white text boxes you can copy either that box's text or the entire screen's text to the clipboard.
 
#16
Yes, it is good option (ocr copying all green text).
Because now, when i have some book in pdf or images, i have to run it through desktop OCR, like Abby fine reader,
put this file on smartphone and open in pleco file reader.
But with that option i could just open pdf or image, run ocr screen reader and copy all text to file.

Also good option would be to automatically create file and convert to simplified or traditional, cause now i have to use another app to convert chars.
It would completely eliminate need for using PC computer to ocr.
 

mikelove

皇帝
Staff member
#17
Well a desktop OCR will probably always be more accurate than ours, but in other respects that makes sense, yes.

Simplified<>Traditional conversion is IMHO not worthwhile unless it's done intelligently and we don't have a good enough algorithm for it yet (too many one-to-many mappings).
 
#19
I dont understand why simp to trad have to be done intelligently?
There is chrome extension like
https://chrome.google.com/webstore/detail/new-tong-wen-tang/ldmgbgaoglmaiblpnphffibpbfchjaeg
and it works - converts simp -<> trad.
More than one traditional character is converted to the same simplified character (some information is lost here).
So if you have this one simplified character, how do you know which traditional character it should be?
Wikipedia has some examples:
https://en.wikipedia.org/wiki/Ambiguities_in_Chinese_character_simplification
 
#20
Since my phone is in my hands, cant it in some way connect from hand to brain using some nerves, and check with brain if this conversion will be ok?


Ok, but cant it be done by ckecking in some words database which character configuration should it be?