iPad 3 / Still OCR

[alanmd]

Re: iPad 3

I'm sure in a few months there will be an iPad stand for scanning documents.

Seems likely, though the angles may be a bit tricky - may need a separate app that does perspective correction. [...] awkward angles - the iPhone is really a much better device for document scanning.

I don't quite understand this- granted the iPad is more awkward to hold for taking photos, but if the camera is held over the centre of the page, why are there any more perspective problems? With both devices you should be centering the camera over the page, not the device.

 

[alanmd]

Re: iPad 3

I just hope apps that don't support retina display on this new iPad won't look so terrible as it did on the iPhone 4. I don't expect many apps and games to be updated for this new capability considering how insanely large they would need the textures to be. :S

They will look, worst case, the same as they did before. They just stand out because other apps look so great- if the app is doing something useful just ignore he resolution and use it.

IPad 3 text looks incredible, and any app that uses native routines to draw text benefits from this. Also apps that get the iPad to scale down images (instead of optimizing for the iPad 1 or 2 resolution) look pretty good out of the box. For me the killer feature is the readability of on-screen text- Chinese especially looks just amazing.

 

[alanmd]

Re: iPad 3

True, but it's tough to keep an iPad in your pocket.

For this reason I end up using iPhone Pleco about ten times as much as iPad Pleco. I love the layout of Pleco on the iPad, but The iPhone is always there, and even when I am reading or studying on the iPad, I often still look up words on the iPad.

What I really would like (pipe dream time, although this would be possible with a bit of code jail broken) is a unified clipboard across the two devices, and/or context menu integration (i.e. highlight some text on the iPad and select "lookup on Pleco iPhone")

Update: To partly answer my own question, I found an app that sort of achieves this called Cloud Copy, at least it makes copying text between iPad, iPhone, Mac, PC, and (apparently soon Android) a bit less painful. You have to run the app (and annoyingly wait for it to log in) after selecting text on the iPhone or iPad, but you can then just paste on the other devices. This is probably the best that iOS lets you do...

 

[radioman]

Re: iPad 3

I also use my iPhone / iPad Touch more than my iPad. Flashcards o'plenty, dictionary lookups, etc. But the iPad can render large docs (so, from a practical perspective, I cannot abandon it for reading"

What I really would like (pipe dream time, although this would be possible with a bit of code jail broken) is a unified clipboard across the two devices, and/or context menu integration (i.e. highlight some text on the iPad and select "lookup on Pleco iPhone")

Maybe a year ago I looked at this exact thing. I wanted to be able to send commands from the iPhone to the iPad (which would be running Pleco with the clipboard function). However, it was messy, not convenient. Software needed to be running on both devices and you needed to initialize it every time you wanted the devices to interact. I think I was having issues with the background software continuing to run, as well as some crashes. I have not looked at it lately, the the function is something I would find useful. I'll probably reinvestigate it.

With that said, it would be interesting to know how many people have both an iPhone and an iPad, with both running Pleco. I know I am.

 

[radioman]

Re: iPad 3

Below is also something that I looked at. This might be the easiest way to take a PDF document with many pages and organize it (at least the way Pleco is currently arranged). Export to a PNG file and make sure your page numbering starts at 1. (Photo Library does not show photo names，so thats not so useful). Then you could at least go to Block Reader > Image Files > Select a folder > find the exact page > open the page.

The problem right now is that you cannot "Select a Folder". At least for me, nothing happens.

Is that specific issue that is addressed under 2.2.11?

also now the app only looks in the root folder for pictures. I wanted to organize it into a different folder, but when i did those pictures do not appear in the list.

Have you turned on "Multi-level file move"? There's a bug relating to that - fixed in the submitted-almost-a-week-ago 2.2.11 update.

i have turned on multi-level file move. i moved the pictures in to pleco through itunes. then went to my ipad->OCR->Block Recognizer->Image File at which time the popup list comes up. They do show up, but when i go to the file browser and move them into another folder that I created and go back to OCR->Block Recognizer->Image File the popup list is empty.

 

[mikelove]

Re: iPad 3

I don't quite understand this- granted the iPad is more awkward to hold for taking photos, but if the camera is held over the centre of the page, why are there any more perspective problems? With both devices you should be centering the camera over the page, not the device.

I was operating on the assumption that people would keep their iPad resting on the table because it's too heavy to hold up for a long period - you could still line them up neatly with a document stand, but I daresay a lot of people would want to have the paper flat on the table and the iPad angled with its bottom touching the table.

IPad 3 text looks incredible, and any app that uses native routines to draw text benefits from this. Also apps that get the iPad to scale down images (instead of optimizing for the iPad 1 or 2 resolution) look pretty good out of the box. For me the killer feature is the readability of on-screen text- Chinese especially looks just amazing.

Oh yes - the swell of rumors concerning the iPad Retina display were what prompted us to finally go out and license some new Chinese fonts; on a low-resolution display HeiTi is probably the best bet (well, that or a highly-optimized SongTi bitmap font like we used to use on Windows Mobile - then again, a lot of those displays were higher-resolution than the non-Retina iPhone/iPad) but on a Retina Display more calligraphic-looking fonts are no problem at all.

What I really would like (pipe dream time, although this would be possible with a bit of code jail broken) is a unified clipboard across the two devices, and/or context menu integration (i.e. highlight some text on the iPad and select "lookup on Pleco iPhone")

Update: To partly answer my own question, I found an app that sort of achieves this called Cloud Copy, at least it makes copying text between iPad, iPhone, Mac, PC, and (apparently soon Android) a bit less painful. You have to run the app (and annoyingly wait for it to log in) after selecting text on the iPhone or iPad, but you can then just paste on the other devices. This is probably the best that iOS lets you do...

Maybe a year ago I looked at this exact thing. I wanted to be able to send commands from the iPhone to the iPad (which would be running Pleco with the clipboard function). However, it was messy, not convenient. Software needed to be running on both devices and you needed to initialize it every time you wanted the devices to interact. I think I was having issues with the background software continuing to run, as well as some crashes. I have not looked at it lately, the the function is something I would find useful. I'll probably reinvestigate it.

Well in theory we could implement something like this in Pleco, but it would take quite a bit of work - iCloud sync is really designed for less frequently updated data, and even small changes can take a minute or two to propagate, which means it's not really practical for this. And Apple's GameKit system requires too much session-specific setup and isn't really as reliable as we'd like. So we'd probably need to connect both devices to a central server (no peer-to-peer due to firewalls) and forward data between them through that; not too difficult to set up that server using EC2 or Linode or somesuch, but a lot of work to develop / test the code on both sides, plus now we've got an extra server to pay for and maintain.

The problem right now is that you cannot "Select a Folder". At least for me, nothing happens.

Is that specific issue that is addressed under 2.2.11?

If you were using "Multi-level file move," then yes - that should be fixed in 2.2.11.

 

[radioman]

Re: iPad 3

I am thinking along the lines of Bluetooth connectivity or direct communication via local network.

Well in theory we could implement something like this in Pleco, but it would take quite a bit of work - iCloud sync is really designed for less frequently updated data, and even small changes can take a minute or two to propagate, which means it's not really practical for this. And Apple's GameKit system requires too much session-specific setup and isn't really as reliable as we'd like. So we'd probably need to connect both devices to a central server (no peer-to-peer due to firewalls) and forward data between them through that; not too difficult to set up that server using EC2 or Linode or somesuch, but a lot of work to develop / test the code on both sides, plus now we've got an extra server to pay for and maintain.

 

[mikelove]

Re: iPad 3

I am thinking along the lines of Bluetooth connectivity or direct communication via local network.

Direct Bluetooth connectivity on iOS is very ugly, sadly, and a local network is never something you want to have to provide tech support for - if somebody's smartphone isn't on their home WiFi network that's not something that I want to be responsible for helping them to fix So a server-based approach is about the only thing I'd trust for a shipping product.

 

[radioman]

Re: iPad 3

Utlizing the new software load, I now can use folders. However, I am running into an interesting challenge.

I am preparing the files and folders off line, arranged as follows:

Main directory = Document1/
with subdirectories, i.e.:
/Document1/pages 01-10/abc.png
/Document1/pages 11-20/def.png

1) I do not think I can drag and drop a tree structure into iTunes. Is that correct?
2) If I do drag and drop via programs like DiskAid, the tree appears there. No problem. As well, I can go to the file Document Reader view and see the folders. But if I go to access via the OCR block reader, It will not show the directory because it appears that it only shows directories if there is an actual graphic in the directory.

That is, in my scenario with only subdirectories in the main directory (no .png files), the entire tree is not shown, even though second level has many png files.

Is there some sort of check Pleco does in OCR Block Read Mode such that it will not display the directory under those conditions?

If you were using "Multi-level file move," then yes - that should be fixed in 2.2.11.

 

[tesnusxenos]

Re: iPad 3

I have the new ipad (3) the Handwriting feature works on my Iphone but there is no button 4 it when I instal it on my new ipad anyone else having this problem?

 

[mikelove]

Re: iPad 3

1) I do not think I can drag and drop a tree structure into iTunes. Is that correct?

Yes, but if you pack it into a .zip archive you can copy that over and it should unpack into the correct structure in Pleco when you tap on it.

2) If I do drag and drop via programs like DiskAid, the tree appears there. No problem. As well, I can go to the file Document Reader view and see the folders. But if I go to access via the OCR block reader, It will not show the directory because it appears that it only shows directories if there is an actual graphic in the directory.

That is, in my scenario with only subdirectories in the main directory (no .png files), the entire tree is not shown, even though second level has many png files.

Is there some sort of check Pleco does in OCR Block Read Mode such that it will not display the directory under those conditions?

Sounds like it might be a leftover of the "multi-level file move" bug - honestly if we'd known that feature was this popular we'd probably have put more effort into it We'll investigate this and fix it in 2.2.12 if there is one (or 2.3.0 if not).

I have the new ipad (3) the Handwriting feature works on my Iphone but there is no button 4 it when I instal it on my new ipad anyone else having this problem?

Try going into the Pleco "Add-ons" tab and tapping "Restore iTunes Purchases" - that should bring it back.

 

[radioman]

Re: iPad 3

Wanted to ping the board here to see if there was any chance of a alpha or beta for the PDF / Photo PDF function in the not to distant future. Still very interested.

Some further observations on the feature - I have found that actually exporting out PDFs @ 300 dots per inch allows much more accurate OCR than screen shots (not surprising). So getting high resolution PNGs from exporting all the pages of a PDF, or if there was a way to do the same via some PDF toolkit, seems that that is definitely the way to go.

I continue to be amazed at the accuracy and speed of the OCR. And every time I think there is no real need to have an on the fly OCR experience, I am forever reminded of it's necessity whenever the OCRed text that has not been redacted is brought in. It seems that, unless the text is just flat and clean, with no pictures or shading or font variations, the OCR invariably has a problem. This does not seem to be a just Pleco issue, but can easily reproduced by a number of other OCR software products. Sure you could go through a document using Adobe Acrobat and redact prior to OCR. But even then you do not get the feedback provided by Pleco with regard to character matching via the green overlay.

Three more things -

- I think it would be useful when positioning the box to allow a two-finger automatic positioning of the box. For instance, if an OCRed photo is being evaluated, I would like to be able to create a box that uses my finger position to define the OCR area - the upper left/lower right or lower left/upper right. maybe with some sort of short (1 second?) delay before the green box would be repositioned.

- I brought this up before, and the idea was met with some opposition due to other interface challenges that would occur. But in seeing the benefit of the redaction, I still think it has merit. Specifically, if you have a box on the screen, you should be able to slide the box with your finger. This is especially useful for OCRing areas where there is a repeated collection of text (e.g., questions and answers bundled together). From the GUI perspective, perhaps this could work if the user presses and holds inside the box in an area with no text. The box would turn orange and then could be moved. Tap the box or outside the box (or maybe just release your finger) to put it in normal mode.

- Any way to allow 3 character searches on Plecoforums? Typing "pdf ocr" brings up nothing. I have run into this more than once. I guess it makes sense not to be able to search on "a" or "of" by themselves, but with 3 characters, I would think the search term should be acceptable and useful.

 

[mikelove]

Re: iPad 3

Wanted to ping the board here to see if there was any chance of a alpha or beta for the PDF / Photo PDF function in the not to distant future. Still very interested.

It'll be among the last features we implement for 2.3, since it's (relatively) self-contained and doesn't run much risk of creating new bugs in other areas. So it may not even be in the first 2.3 beta. I know you're eager for it but from a project management standpoint it just doesn't make sense to do it early.

- I think it would be useful when positioning the box to allow a two-finger automatic positioning of the box. For instance, if an OCRed photo is being evaluated, I would like to be able to create a box that uses my finger position to define the OCR area - the upper left/lower right or lower left/upper right. maybe with some sort of short (1 second?) delay before the green box would be repositioned.

Could do some tests, but I'm wary of the ergonomics - even ignoring the potential for hand discomfort, it's very hard to position accurately with a multi-finger gesture; broad movements are about all that they're good for.

- I brought this up before, and the idea was met with some opposition due to other interface challenges that would occur. But in seeing the benefit of the redaction, I still think it has merit. Specifically, if you have a box on the screen, you should be able to slide the box with your finger. This is especially useful for OCRing areas where there is a repeated collection of text (e.g., questions and answers bundled together). From the GUI perspective, perhaps this could work if the user presses and holds inside the box in an area with no text. The box would turn orange and then could be moved. Tap the box or outside the box (or maybe just release your finger) to put it in normal mode.

Only really workable in Block Recognizer mode, though I don't see any big issue with it in that - the press-and-hold-in-an-area-with-no-text gesture is too iffy, though, we'd probably need a dedicated button or something.

- Any way to allow 3 character searches on Plecoforums? Typing "pdf ocr" brings up nothing. I have run into this more than once. I guess it makes sense not to be able to search on "a" or "of" by themselves, but with 3 characters, I would think the search term should be acceptable and useful.

Seems to be a server-level setting - we can investigate it further after we finish our big server move.

 

[radioman]

Re: iPad 3

Could do some tests, but I'm wary of the ergonomics - even ignoring the potential for hand discomfort, it's very hard to position accurately with a multi-finger gesture; broad movements are about all that they're good for.

I don't disagree about trying to move your hand in alkward positions to put a box around some text. The challenge I was trying to address with this one is that if there is a location on a OCRed photo that needs to be highlighted, then I want to avoid having to search for the green box.

For example, In the case where the green box completely surrounds the photo and I am zoomed in a bit, I would like to avoid having to zoom out to see that the box is is either full size of the photo or in some remote region of the page, and then zoom back in again. And I do not see this as a subtle gesture thing.

Scenario 1 -Two-Hand Gesture - One such case might be trying to make a 5 inch diagonal box on the screen. In this situation, you might use your two thumbs, one upper left, one lower right. Even if the positioning was a little off, the user could just adjust it a bit as the box would have been newly positioned right on the screen with no need to zoom out. Irrespective, I figure this approach would still be quicker and less effort than the zoom-out/look-around/zoom-to-new-position method, specifically in the cases where you have the exact area you want to draw the box already right on screen.

Scenarion 2 - One-Hand Gesture, where a box is roughly outlined. I could see a scenario where I am looking at the text on the screen and I want to set the box in approximately quadrant B of the viewable screen. My thumb and index finger could span that pretty easy, and it would be a rough approximation such that I do not have to expend too much energy adjusting the box. So I just quickly set the box and then I can now use the guides to easily fine tune the box. In most cases I suspect there would be some box fine-tuning anyway.

Alternative Approach - I think the above is practical and makes sense. But perhaps an alternative implementation might be having a button that would just center the box to the current view, 90% height, and 90% width (or maybe a user settable percentage). In this way, the user could reposition the box without having to unnecessarily zoom out to set the box around an area that is already present within the current screen view.

 

[character]

Re: iPad 3

@radioman, would most of what you want be addressed if the box handles were always shown (at the edges of the screen if one was zoomed into the box), and tapping and holding on one of them would immediately resize the box to make that a corner of the box which one could resize further at that point?

For awareness of where you are in the page, perhaps a simple 'green outline within a white box' display of that could be shown in a corner when the 'picture' menu is tapped.

---

I'm not sure this is a good idea or not, but I was thinking about a new mode/feature where one could block off parts of the image to prevent them from being OCR'd.

 

[mikelove]

Re: iPad 3

I don't disagree about trying to move your hand in alkward positions to put a box around some text. The challenge I was trying to address with this one is that if there is a location on a OCRed photo that needs to be highlighted, then I want to avoid having to search for the green box.

Ah, that makes much more sense. So it's sort of a "summon OCR box to where I am" gesture then.

On your other idea about moving the box - how about if we just add a little move cursor outside of the box somewhere? You'd drag that around to move it - a green crosshairs above and to the left of the top left corner of the box, say.

For awareness of where you are in the page, perhaps a simple 'green outline within a white box' display of that could be shown in a corner when the 'picture' menu is tapped.

Not sure if this is a widespread enough issue for us to need to address - is the issue really that you don't know where you are on the page or is it simply that you've lost the green box? radioman's solution would seem to help for the latter while yours would help for the former.

I'm not sure this is a good idea or not, but I was thinking about a new mode/feature where one could block off parts of the image to prevent them from being OCR'd.

Not really feasible with our current OCR engine - it just does rectangles, so we could either divide the non-blocked-off space into a bunch of rectangles and recognize those (problematic if you have characters crossing the line or rectangles small enough to confuse it) or recognize the whole thing but just not display results from the blocked-off area (which I'm not quite sure of the benefit of).

 

[radioman]

Re: iPad 3

On your other idea about moving the box - how about if we just add a little move cursor outside of the box somewhere? You'd drag that around to move it - a green crosshairs above and to the left of the top left corner of the box, say.

I get the feeling that being able to somehow activate and drag the box would be really intuitive. I don't see having a separate control that is located elsewhere would accomplish that, especially given that the current positioning controls use direct engagement with the green box (intuitive to set). Perhaps if there was some tag on the side of the box, that might work as it would be like using a handle dragging around the box.

Not sure if this is a widespread enough issue for us to need to address - is the issue really that you don't know where you are on the page or is it simply that you've lost the green box? radioman's solution would seem to help for the latter while yours would help for the former.

I don't think you were implying this, but for clarification, in my scenario, I know the green box is (usually around the perimeter), I just don't want to have to zoom out to find it.

@radioman, would most of what you want be addressed if the box handles were always shown (at the edges of the screen if one was zoomed into the box), and tapping and holding on one of them would immediately resize the box to make that a corner of the box which one could resize further at that point?

For awareness of where you are in the page, perhaps a simple 'green outline within a white box' display of that could be shown in a corner when the 'picture' menu is tapped.

Perhaps the kind of tap and hold @character suggests might provide additional flexibility in situations where part of the box is already visible. Also, showing a "position" box could have merit in cases where the document is very large. However, my original idea was to get a quick way to solve the problem of an OCR result "gone bad". And usually in those situations I am staring at the exact text I want to read - but can't - due to the OCR defined area being beyond the borders of the viewable area. Therefore, it seemed to me me that making it as easier being able to draw a box around the area of interest in quick and dirty fashion made sense.

 

[mikelove]

Re: iPad 3

I get the feeling that being able to somehow activate and drag the box would be really intuitive. I don't see having a separate control that is located elsewhere would accomplish that, especially given that the current positioning controls use direct engagement with the green box (intuitive to set). Perhaps if there was some tag on the side of the box, that might work as it would be like using a handle dragging around the box.

Well any activation gesture has very poor discoverability, that's my basic problem with it - a button to go into "move mode" is easy to see but means a more cluttered interface in an app that's already legendary for its cluttered interfaces, and a gesture means we either need a note to explain the gesture or a popup or some other kludge to make people aware of it. A drag handle located outside of the box's frame but colored / styled to look like the box has a pretty clear function and is very easy to find.

I don't think you were implying this, but for clarification, in my scenario, I know the green box is (usually around the perimeter), I just don't want to have to zoom out to find it.

That was what I figured, yes - I'm basically trying to determine whether being able to "summon" the box with a two-finger gesture would solve both of your problems.

 

[character]

Re: iPad 3

I'm concerned with the responsiveness of the existing UI (tap to look up a definition, drag the page around) if there's a two-touch resizing gesture or dragging gesture added without it being a special mode which could be activated/deactivated somehow. The slower the loops of read->look up definition->read and read->move page->read are, the harder it is to maintain flow.

Well any activation gesture has very poor discoverability, that's my basic problem with it - a button to go into "move mode" is easy to see but means a more cluttered interface in an app that's already legendary for its cluttered interfaces, and a gesture means we either need a note to explain the gesture or a popup or some other kludge to make people aware of it.

But there's room for a button of some sort on the picture menu. I do like the idea of a button to bring the selection box to about the size of the current view (or the page, if the page is smaller than the view), as then the box could be further resized. This would work well in conjunction with the existing zoom/pan functionality (zoom/pan to desired location and then resize the box to that size).

A drag handle located outside of the box's frame but colored / styled to look like the box has a pretty clear function and is very easy to find.

Are you talking about a drag handle which is not attached to the box, but is more like a mouse, i.e. move the handle over here and the box moves over there? That wouldn't be a good fit for the direct interaction model of iOS.

 

[mikelove]

Re: iPad 3

I'm concerned with the responsiveness of the existing UI (tap to look up a definition, drag the page around) if there's a two-touch resizing gesture or dragging gesture added without it being a special mode which could be activated/deactivated somehow. The slower the loops of read->look up definition->read and read->move page->read are, the harder it is to maintain flow.

I'm not quite sure how it would hurt responsiveness - all we do is register with the system to detect two-tap gestures and do something differently when we actually do detect one of them. Nothing will change if there's only one finger on the screen.

But there's room for a button of some sort on the picture menu. I do like the idea of a button to bring the selection box to about the size of the current view (or the page, if the page is smaller than the view), as then the box could be further resized. This would work well in conjunction with the existing zoom/pan functionality (zoom/pan to desired location and then resize the box to that size).

That might be another good approach, yes - a simple "center box on screen" button. The problem is that if we put it in the picture menu not enough people are going to find it (unless we can come up with a very obvious icon for "center box on screen," maybe a zooming box icon or something). Then again, the same could be said for a two-finger gesture, but once we're dealing with that barrier anyway it makes sense to make it as easy to invoke as possible for people who do discover it.

Are you talking about a drag handle which is not attached to the box, but is more like a mouse, i.e. move the handle over here and the box moves over there? That wouldn't be a good fit for the direct interaction model of iOS.

No, I'm talking about one that's permanently positioned (say) 20 pixels above the top left corner of the box - you drag it around and the box moves with it.