KR101174536B1 - Processing techniques for visual capture data from a rendered document - Google Patents

Abstract

A facility for navigating an electronic document is described. This facility receives user input to select a portion of the content of the rendered document that makes up the sentence fragment. In response to receiving this user input, the facility identifies an electronic document included in the body of the electronic document, and the identified electronic document includes a selected document portion. In response to receiving the user input, the facility identifies a position within the identified electronic document in which the selected document portion occurred.

Facsimile, Sentence Fragment, Rendered Document, Content, User Input, Electronic Document, Position

Description

PROCESSING TECHNIQUES FOR VISUAL CAPTURE DATA FROM A RENDERED DOCUMENT}

The described technique relates to document processing.

There is a continuing demand for paper documents, as can be seen from the growing number of paper documents in the computer age. Today it is easier than ever to print and publish paper documents. Although electronic documents are simpler to copy, transcribe, search, and edit, paper documents are still widely used.

Given the popularity of paper documents and the advantages of electronic documents, it would be useful to combine these two advantages.

1 is a data flow diagram illustrating the flow of information in one embodiment of a core system;

2 illustrates the components involved in a typical implementation of a system in the context of a typical operating environment.

3 is a block diagram showing an embodiment of a scanner;

4 illustrates an exemplary environment in which embodiments of the system operate;

5 is a flow chart showing the steps normally executed by a system to implement a bookmark;

6 shows a scanning device having an integrated highlighter and pen;

7 is a flow diagram illustrating the steps normally executed by a system to process a text capture action;

8A-8D illustrate how often two characters of the alphabet have approximately the same * related * shapes in multiple fonts,

9 illustrates a method used by the facsimile in any embodiment to learn a completely new set of symbols using self recognition;

FIG. 10 shows a subset of the symbols shown in FIG. 9 self-limiting vertical and horizontal expansion; and

FIG. 11 shows a relationship between some of the symbols shown in FIG. 9; FIG.

summary

A system for translating and interacting with rendered documents (eg, printed or displayed documents), and associated digital "source", "copy", or "reference" versions ("system") of such documents Described. In some embodiments, the system may, in some cases, perform the following functions, but is not directly related to recognizing and translating the characters themselves; It is not itself related to recognizing and understanding printed characters. Rather, the system makes the version of the document known or machine readable (eg, ASCII or some other machine readable text) source or reference version document to be useful, machine accessible, or future available. Assume that we will lose. The system uses various features (including text) in the rendered document for navigation (determining location in the document). The locations are used in turn to allow interaction with the many sets of user functions, some of which are described below.

The system is based in part on the process of translating and decoding patterns of marks in the document (eg, text and any rendered supplemental information marks) to determine positional information. In various embodiments, such location information is associated with the document itself, eg, location within the document, often a single paragraph, sentence, word, and even a single character. However, if the physical layout of a particular rendering of a document is also known, the location information can be converted to a location on the display screen, printed pages, and the like.

In describing various embodiments of the system, the term “printed text” is used. "Printed" is generally used for rendering any form of document readable to humans (eg, on a pacer, on a display screen, in a Braille format, etc.). In many cases, it should be understood that the features and applications of the system apply very well to non-alphabetic rendered contacts such as composition, graphics and images, specific marks, and the like. Embodiments of the system include this additional use.

Part I- Introduction

One. Characteristics of the system

For each of all paper documents having a corresponding electronic counterpart, there is a discrete amount of information in the paper document from which the electronic copy can be identified. In some embodiments, the system identifies and locates an electronic copy of a document using, for example, a sample of text captured in a paper document via a handheld scanner. In most cases, the amount of text required by the device is very small in that some words of the document text can function as an identifier for the paper document and as a link to its electronic copy. In addition, the system can use these few words to identify the document as well as its location within the document.

Thus, the paper document and its digital copy can be associated in many useful ways using the present system described below.

1.1. Brief overview of the outlook

Once the system is set up to associate a portion of the text of a paper document with a particular digital entity, the system can configure a large amount of functionality for that association.

Increasingly, it is true that most paper documents have electronic copies that can be accessed on the World Wide Web or from some other online database or document collection, or by payment or subscription. Then, at the simplest level, when a user scans a few words in a paper document, the system retrieves, displays, emails, purchases, prints, or webpages an electronic document or a portion thereof. Post to As an additional example, by scanning some words in a book that a person is reading during breakfast, the audiobook version in that person's car can start reading from the moment he starts driving to go to work, or print Scanning the cartridge's serial number may initiate a process to instruct replacement.

The system implements many of these and other examples of "paper / digital integration" without requiring changes to the current document reading, printing, and publishing process, thus creating a whole new chapter in digital functionality in such conventionally rendered documents. To provide.

1.2. Terms

Typical use of the system begins with the use of an optical scanner that scans text from a paper document. However, it should be noted that other capture methods from other types of documents may be equally applicable. Thus, the system is sometimes described as scanning or capturing text from a rendered document, which terms are defined as follows:

A rendered document is a printed document or a document shown on a display or monitor. It is also a document recognizable to a person, whether in permanent form or on a temporary display.

Scanning or capturing is a systematic investigation process for obtaining information from a rendered document. This process may include optical capture using a scanner or camera (eg, a camera mounted on a mobile phone), and may include reading the document out loud and moving it to an audio capture device or typing on a keypad or keyboard. See section 15 for more examples.

2. Introduction of this system

This section describes some of the devices, processes, and systems that make up a system for paper / digital integration. In various embodiments, the system implements a wide variety of services and applications on this basic core that provide basic functionality.

2.1. process

1 is a data flow diagram illustrating the flow of information in one embodiment of a core system. Other embodiments may not use all of the steps or components shown herein, and some embodiments may use more.

Typically in optical format by an optical scanner or audio format by a voice recorder, text is captured from the rendered document (100), and this image or sound data can be used to remove artificial elements of the capture process or to improve the signal-to-noise ratio. Is handled 102. Recognition process 104, such as OCR, velocity recognition, autocorrelation, or the like, then transforms the data into a signature consisting of text, text offsets, or other symbols in some embodiments. Alternatively, the system performs an alternative form of extracting document symbols from the rendered document. In some embodiments the symbol represents a set of possible text trnascriptions. This process can be influenced by feedback from other steps, for example, if the search process and context interpretation 100 have identified some candidate documents from which the capture originated, possible interpretation of the original capture. ) Is narrowed.

Post-processing 106 takes the output of the recognition process and filters it or otherwise performs other tasks to make it useful. Depending on the embodiment implemented, at this stage some direct action 107 is immediately taken out without reference to a subsequent step, e.g. a phrase or symbol is captured that contains enough information per se to convey the user's intent. do. In this case no digital copy documents need to be referenced and even need not be known to the system.

Typically, however, the next step will be to construct a query or set of queries for use in the search (108). Some aspects of the construction of the query may depend on the search process used, and thus will not be performed until the next step. Typically, however, there may be some actions that can be performed in advance, such as the removal of obviously misrecognized or irrelevant characters.

The query or queries are then passed to a search and context analysis step 110. Here, the system optionally attempts to identify the document in which the original data was captured. To do so, the system typically uses the search index and search engine 112, knowledge of the user 114, and knowledge of the user's context or the context in which the capture occurred. The search engine 112 may use and / or index information about rendered documents, their digital copy documents, and about documents that exist on the web (internet). In addition to reading from many of these sources, it is also possible to record them and, as noted, by providing the recognition system 104 with information about the next possible word based on language, font, rendering, and knowledge of the candidate document, for example. The information can then be fed to the next step in the process.

In some circumstances, the next step is to retrieve 120 the document or copy of documents that have been identified. The source of the document 124 may be accessed directly, for example, from a local filing system or database or web server, or may be connected via some access service 122 that may execute authentication, security or payment, or in a predetermined format. Other services such as document conversion can be provided.

Applications in the system can take advantage of the extra functionality or association of data with all or part of the document. For example, as discussed in section 10.4, an advertising application may utilize the association of a particular advertising message or subject with a portion of the document. This extra associated functionality or data can be considered as one or more overlays on the document, referred to herein as " markup. &Quot; Then, the next step in process 130 is to identify any markup related to the captured data. Such markup may be provided by the user, the creator or publisher of the document, or someone else, and may be accessed directly from some source 132 or generated by some service 134. In various embodiments, markup may be applied to or associated with a rendered document and / or a digital copy of a rendered document, or a group of either or both of these documents.

Finally, as a result of the initial steps, some action is taken 140. These may be default actions, such as simply recording the found information, or may be dependent on data or documents, or may be derived from markup analysis. Sometimes this action is simply passing data to another system. In some cases, various possible actions suitable for capturing a specific point in a rendered document may be displayed on a menu on an associated display, such as a docker versus local display 332, a computer display 212, or a mobile phone or PDA display 216. Will be provided to the user. If the user does not respond to this menu, a default action can be taken.

2.2. Component

2 illustrates the components involved in a typical implementation of a system in the context of a typical operating environment. As shown, the operating environment includes one or more optical scanning capture device 202 or voice capture device 204. In some embodiments, the same device performs both functions. Each capture device may communicate with other parts of the system, such as computer 212 and mobile station 216 (eg, mobile phone or PDA), using either a direct wired or wireless connection or via network 220. Communication using wired and wireless connections typically includes a wireless base station 214. In some embodiments, the capture device is integrated with the mobile station and optionally shares with some of the audio and / or optical components used in the voice communications and photography device.

Computer 212 may have a memory containing computer executable instructions for processing orders from scanning devices 202 and 204. As an example, the order may include an identifier (such as the serial number of the scanning device 202/204 or an identifier that partially or completely distinguishes the scanner user), scanning context information (eg, scan time, scan location, etc.), and And / or include scanning information (such as a text string) used to distinguish the document being scanned from another. In alternative embodiments, the operating environment may include smaller or more components.

In addition, search engine 232, document source 234, user account service 236, markup service 238, and other network services 239 are available on network 220. The network 220 may be an integrated intranet, public internet, mobile phone network or some other network, or any interconnect thereof.

Regardless of how the devices are connected to each other, they can operate according to known commercial transaction and communication protocols (eg, Internet Protocol (IP)). In various embodiments, the functionality and capacity of the scanning device 202, the computer 212, and the mobile station 216 may be fully or partially integrated into one device. Thus, the terms scanning device, computer, and mobile station may refer to the same device, depending on whether the device combines the capabilities and capacities of scanning device 202, computer 212, and mobile station 216. have. Moreover, some or all of the functionality of search engine 232, document source 234, user account service 236, markup service 238, and other network services 239 may be of any device and / or other. It may be implemented on a device not shown.

2.3. capture device

As noted above, the capture device may capture text using an optical scanner capable of capturing image data from the rendered document, or using an audio recording device that captures the user reading aloud the text, or using other methods. Can be. Some embodiments of the capture device may capture images, graphic symbols, icons, and the like, as well as machine readable code such as barcodes. The device may be extremely simple to configure as a transducer, some storage, and a data interface, depending on other functionalities located somewhere in the system, or may be a more complete device. For illustration purposes, this section describes devices based on optical scanners and the like with an appropriate number of features.

Scanners are well known devices for capturing and digitizing images. As a class in the copier industry, the first scanner was a relatively large device for capturing an entire document at one time, and recently, portable optical scanners of a convenient type, such as pen-type handheld devices, have been introduced.

In some embodiments, a portable scanner is used to scan text, graphics, or symbols from a rendered document. The portable scanner includes a scanning element that captures text, symbols, graphics, and the like from the rendered document. In addition to documents printed with paper, in some embodiments, the rendered document includes a document displayed on a screen, such as a CRT monitor or LCD display.

3 is a block diagram of one embodiment of a scanner 302. Scanner 302 is an optical scanning head 308 that scans information from a rendered document and converts it into machine-compatible data, and typically delivers an image of a lens, aperture, or rendered document to the scanning head. Optical path 306, which is an image conduit for the device. The scanning head 308 may combine a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) image device, or other type of optical sensor.

The microphone 310 and associated circuitry converts the sounds of the environment (including the words read aloud) into mechanical signals, and other input functions may also be in the form of other tactile sensors such as buttons, scroll-wheels or touchpads 314. Exists as.

Feedback to the user is made possible via a visual display or indicator 332, or through a speaker or other audio transducer 334, and via the vibration module 336.

Scanner 302 includes logic 326 that processes signals received in other formats and / or interpretations and interacts with other various components. Logic 326 may be operable to read and write data and program instructions stored in associated storage 330 such as RAM, ROM, flash, or other suitable memory, and may read time signals from clock unit 328. . Scanner 302 also includes an interface 316 that communicates the scanned information and other signals with the network and / or associated computing device. In some embodiments, scanner 302 may have an on-board power source 332. In another embodiment, the scanner 302 may be powered from a wired connection with another device, such as a universal serial bus (USB) connection.

As an example of the use of the scanner 302, a reader may use the scanner 302 to scan some text in a newspaper article. The text is scanned into the bitmap image through the scanning head 308. Logic 326 causes the bitmap image to be stored in memory 330, with an associated time-stamp read from clock unit 328. Logic 326 may also perform optical character recognition (OCR), or may perform a post-scan process for converting a bitmap image into text. Logic 326 also performs a convolution-like process, e.g., to find repeated occurrences of a character, symbol, or object and to determine the number or distance of another character, symbol, or object between these repeated elements, You can selectively extract symbols from images. The reader can then upload the bitmap image (text or other symbol if the post-scan process is done by logic 326) to the associated computer via interface 316.

As another example of the use of the scanner 302, the microphone 310 can be used as a voice capture port to allow a reader to capture some text of a newspaper article as an audio file. Logic 326 causes the audio file to be stored in memory 328. Logic 326 may also perform speech recognition or other post-scan process to convert the audio file into text. The reader can then upload the audio file (or text generated by the post-scan process performed by logic 326) to the associated computer via the interface 316 as described above.

Part 2-Overview of the scope of the core system

As paper-to-digital integration becomes more common, there are many aspects of existing technologies that can be changed or better implemented to better utilize this integration. This section covers some of these issues.

3. Search

Searching through collections of documents, even large collections such as the World Wide Web, is now commonplace for ordinary users who use the keyboard to create search expressions entered into search engines. This section and the next section discuss the construction of queries derived from documents rendered by capture and both aspects of the search engine that handle these queries.

3.1. Scan / Speak / Type as Search Inquiry

The use of the system described above typically begins with several words being captured from a document rendered using any of several methods, including those mentioned in section 1.2 above. End-to-end in the system so that document collection is used to extend the recognition process when input needs some interpretation for conversion to text, such as for OCR or voice input. There may be feedback. End-to-end feedback is derived from possible matches in the candidate document by performing an approximation of recognition or interpretation, by identifying one or more sets of candidate matching documents, and then further refine and limit recognition and interpretation. By using the information, it can be applied. Candidate documents may be weighted according to expected relevance (eg, based on the number of people scanned in these documents, or popularity on the Internet), and these weights may be applied in this iterative recognition process.

3.2. Short search

Since the ability to select search queries based on some words becomes very large when the relative positions of the words are known, only a small amount of text may be captured in order for the system to identify the position of the text in the collection. Very generally, adjacent sequences, such as short phrases, can be input text.

3.2.1. Find documents and locations within documents from short text captures

In addition to finding the document from which the phrase originated, the system can identify a location within the document and also take action based on this knowledge.

3.2.2. Alternative way to find location

The system can also use other methods of finding the document and location, such as by using a watermark or other special mark on the rendered document.

3.3. Combine other elements into search queries

In addition to the captured text, other elements (ie, information about the user's identification, profile, and context) may be part of the search query, such as capture time, the user's geographic location and identification, knowledge of the user's habits, and recent activity. Can be formed.

In particular, if these are very recent, other information about document identification and previous capture may form part of the search query.

The identification of the user may be determined from a unique identifier associated with the capturing device and / or biometric or other supplemental information (voice speed, fingerprint, etc.).

3.4. Knowledge of the unreliable nature of search queries (OCR errors, etc.)

The search query can be constructed taking into account the type of error that is likely to occur in the particular capture method used. An example of this is to indicate a suspected error in the recognition of certain characters, in which case the search engine may treat these characters as wildcards or give them lower priority.

3.5. Local cache of indexes for run / offline use

Sometimes the capturing device may not be communicating with search engines or collections when capturing data. For this reason, information that helps offline use of the device may be downloaded to this device in advance or to some entity with which the device communicates. In some cases, much or all of the index associated with the collection may be downloaded. This will be discussed further in Section 15.3.

3.6. Inquiries in any form can be recorded or executed later

If there may be a cost or delay associated with communicating the query or receiving the results, this pre-loaded information can improve the performance of the local device, lower the communication cost, and also be useful and timely. User feedback can be provided.

If no communication is possible (the local device is "offline"), the query can be saved and sent to the rest of the system when the communication is restored.

In this case, it is important to send a time stamp with each query. Capture time can be a very important factor in interpreting queries. For example, section 13.1 discusses the importance of capture time with respect to previous captures. Note that the capture time is not always the same as the time the query is executed.

3.7. Parallel search

For performance reasons, multiple queries may be initiated in either serial or parallel in response to a single capture. Multiple queries may be sent in response to a single capture, for example, to add new words to the capture or to query multiple search engines in parallel.

For example, in some embodiments, the system sends queries of a particular index for the current document to a search engine on a local machine, to a search engine on a unified network, and to a remote search engine on the Internet.

The result of one particular search may be given a higher priority than the others.

The response to a given query may indicate that a query currently in progress is unnecessary, in which case the query in progress may be canceled before completion.

4. Paper and Search Engines

It is often desirable for search engines that deal with traditional online queries to deal with those that come from the rendered document. Existing search engines can be extended or modified in many ways to make them more suitable for use with the present system described above.

The search engine and / or other components of the system may create and maintain indexes with extra or different characteristics. The system can change incoming paper-derived inquiries, or change the way inquiries are handled in the search for results, so that these paper-derived inquiries are typed into web browsers and other sources. Can be distinguished from In addition, when the results are returned by a search derived from a paper as compared to those from other sources, the system may take different actions or provide different options. Each of these approaches is described below.

4.1. Indexing

Often, the same index can be retrieved using either paper-derived or traditional queries, but the index can be enhanced for use in the current system in a variety of ways.

4.1.1. Knowledge of the paper format

Extra fields can be added to this index that can be helpful in the case of paper-based searches.

Index entry indicating document availability in paper format

The first example is a field indicating that the document is known to be present or distributed in paper form. If the query comes from paper, the system may give higher priority to such documents.

Popular paper format knowledge

In this example, statistical data related to the popularity of the paper documents (and, optionally, to specific areas within these documents), such as the amount of scanning operation, the number of publications provided by the publisher or other sources, etc. are used. It gives higher priority to such documents, and raises the priority of digital copies (eg, for browser-based queries or web searches).

Rendered  Knowledge of the format

Another important example may be to record information about the layout of a particular rendering of a document.

For example, with respect to any particular edition of a book, the index may include information about where the line breaks and the page breaks, what fonts are used, whether capitalization differs from normal, and so on.

The index may also include information about proximity to other items on the page, such as images, text boxes, tables, and advertisements.

Use of Semantic Information about the Original

Finally, semantic information can be inferred from the source markup, but it is not clear in the paper document, for example, that a particular piece of text refers to an item provided for sale, or any particular sentence refers to program code. The fact that it is included can also be recorded in the index.

4.1.2. Index with knowledge of capture methods

The second factor that can change the characteristics of the index is the knowledge of the capture types that may be used. If the index considers characters that cannot be easily distinguished in the OCR process, or contains some knowledge of the fonts used in the document, it would be beneficial to make a search initiated by optical scan. Similarly, if the query comes from speech recognition, an index based on the phoneme of similar sounds will be searched more effectively. Another factor that may affect the use of indexes in the model described above is the importance of iterative feedback during the recognition process. If the search engine can provide feedback from the index as the text is captured, it can greatly improve the accuracy of the capture.

Indexing with Offsets

If the index is likely to be searched using the offset-based / autocorrelation OCR method described above in section 9, in some embodiments, the system stores symbol information of the appropriate offset or index.

4.1.3. Multiple indexes

Finally, in the system described above, it may typically be possible to perform a search over multiple indices. Indexes can be maintained on multiple devices or on converged networks. Some indexes may be downloaded to the capture device, or may be downloaded to a device close to the capture device. Separate indexes may be created for users or groups of users with particular interests, habits, or permissions. An index can exist for each file system, each directory, and even for each file on a user's hard disk. Indexes can be published and subscribed by users and systems. It would then be important to construct indexes that can be distributed, updated, merged, and effectively separated.

4.2. Handling of Inquiries

4.2.1. 것을 Capture is from paper

When the search engine recognizes that the search query comes from a paper document, the search engine can take another action. Search engines can handle queries, for example, in a way that gives them more tolerance for the types of errors that may appear in a particular capture method.

The search engine can infer it from some of the indicators included in the query (such as a flag indicating the nature of the capture) or from the query itself (for example, it can recognize errors or uncertainties typical of the OCR process). .

Alternatively, the query from the capture device may reach the engine by a different connection channel or port or type than from another source, and may be distinguished in that way. For example, some embodiments of the system will send a query to a search engine through a dedicated gateway. Thus, the search engine knows that all queries going through the dedicated gateway are from paper documents.

4.2.2. The use of context

Section 13 below describes various other elements that are outside of the captured text itself but can be of significant help in identifying the document. These include the history of recent scans, the long-term reading habits of a particular user, the geographic location of the user, the user's recent use of a particular electronic document, and the like. These elements are referred to herein as "contexts."

Part of the context can be handled by the search engine itself and can be translated into search results. For example, a search engine may track a user's scan history and may cross-reference this scan history with existing keyboard-based queries. In such cases, the search engine maintains and uses more state information for each individual user than most existing search engines do, and each interaction with the search engine is longer than typical and multiple search It can be thought of as extending across the fields.

Part of the context can be sent to the search engine in a search query (section 3.3), stored in the engine to serve as part of a future query, and thus a filter or secondary search applied to the results from the search engine. .

Data-to search Stream  input

An important input to the retrieval process is the wider context of how the user community interacts with the rendered version of the document-for example, which documents are most widely read and to whom. This is similar to a web search that returns the most frequently linked page or the most frequently selected page from the last search result. See sections 13.4 and 14.2 for a more detailed discussion of this.

4.2.3. Document sub-area

The system described above issues and uses not only information about the document as a whole, but also information about sub-regions of the document, even individual words. Many existing search engines focus on finding files or documents related to a particular query. Search engines that work on finer grains and can even identify locations within a document will provide a significant benefit to the system described above.

4.3. Provided results

The search engine may use some of the additional information it currently maintains to influence the results provided.

The system may also provide the particular document that the user is accessing as a result of just belonging to a paper copy (see section 7.4).

Search engines can also go beyond simple search of text and provide new actions or options suitable for the systems described above.

5. Markup, Annotation, and Metadata

In addition to performing a capture-search-retrieve process, the system described above associates extra functionality with a document, and more specifically with a particular location or segment of text in the document. This extra functionality is often associated with the rendered document due to, but not entirely, its electronic copy. As one example, a hyperlink of a web page may have the same functionality when the print output of that web page is scanned. In some cases, this functionality is not limited to electronic documents, but stored or created elsewhere.

This layer of additional functionality is referred to herein as "markup."

5.1. Overlays , static and dynamic

One way to think about markup is as an "overlay" on a document, which can provide additional information about the document or part of the document, or specify an action associated with the document or part of the document. Markups may include human readable content, but are often intended to be invisible to the user and / or intended for mechanical use only. As such an example, when the user captures text from a specific area of the rendered document or captures an audio sample describing the pronunciation of a particular phrase, there is an option to display a pop-up menu on the nearby display.

5.1.1. Multiple layers available from different sources

Any document can have multiple overlays simultaneously, and they can be derived from various locations. Markup data may be generated or supplied by the writer, user, or another side of the document.

Markup data may be attached to or embedded in an electronic document. Markup data can be found in conventional locations (eg, where the same document but with a different file name suffix). Markup data may be included in the search results of the query that locates the original document, or may be found by separate queries in the same or different search engines. Markup data can be found using the original captured text and other capture information or contextual information, or can be found using the location of the capture and existing inferred information about the document. Even if the markup itself is not included in the document, the markup data can be found at a location specified in the document.

Similar to the way in which links to conventional html web pages are often embedded as static data in html documents, markup can be mostly static and specific to the document. However, markup may be dynamically generated and / or applied to a large number of documents. One example of dynamic markup is information to which the document is attached, including the latest stock prices of the companies mentioned in the document. One example of markup that is widely applied is translation information that is automatically available in a particular language for multiple documents or sections of the document.

5.1.2. Personal "plug-in" floor

The user can also install markup data or sign its specific source, thus personalizing the system response to a particular capture.

5.2. Keywords and phrases, trademarks and logos

Some elements of a document may have a specific "markup" or functionality associated with them based on their own characteristics rather than location within a particular document. By way of example, there are logos and trademarks that can link the user to additional information about an organization, as well as specific marks printed on the document for purely scanning purposes. The same principle applies to "keywords" or "key phrases" in text. Organizations can register specific phrases they are related to or want to be associated with and attach specific markup that becomes available wherever the phrase is scanned.

Any word, phrase, etc. can have an associated markup. For example, the system may add a particular item to a pop-up menu (eg, a link to an online bookstore) whenever the user captures the word "book" or book title or topic related to the book. In some embodiments of the present system, a digital copy document or an index may be referenced to determine if a capture occurred near the word "book", book title, or topic associated with the book, and the movement of the system may be such a proximity of the keyword element. Will change accordingly. In the above example, note that due to markup, data captured from non-commercial text or documents can cause commercial transactions.

5.3. User -supplied content

5.3.1. User comments and annotations , including multimedia

Annotation is another type of electronic information that can be associated with a document. As an example, a user may attach an audio file of his or her thoughts on a particular document for later retrieval as a voice annotation. As another example of a multimedia annotation, a user may attach a photo to what is called a document. The user supplies annotations for the document but the system can associate annotations from other sources (eg, share other users in the workgroup).

5.3.2. Proof -notes from reading

An important example of user-sourced markup is a paper document annotation as part of proof-reading, editing or reviewing purposes.

5.4. Third -Party Content

As mentioned above, the markup data may be supplied by a third party, such as by another reading agent of the document. Good examples are community-managed information about online discussions and reviewers, specific tasks, translations and explanations by volunteers.

Another example of third-party markup is provided by an advertiser.

5.5. Dynamic markup based on the user 's data stream

By analyzing data obtained from documents by several or all users of the system, markup may be generated based on the activity and interests of the community. An example would be an online bookstore that generates annotations or markups that tell a user that "the person who enjoys this book also enjoys ...". The markup can be less anonymous and can tell the user which person on his contract list has recently read this document. Another example of datastream analysis is included in section 14.

5.6. Markup based on external events and data sources

Markup is often based on data sources and external events, such as input from an integrated database, information from the public Internet, or statistics collected by the local operating system.

The data source may be more local and may provide information about the user's context, such as the user's identity, region, and activity in particular. By way of example, the system can provide a markup layer that can communicate with a user's mobile phone and give the user the option to send a document or call someone recently by phone.

6. Authentication, Personalization and Security

In most situations, the user's identity is known. Sometimes this becomes an "anonymous identity", where the user is identified only by the serial number of the capture device, for example. Typically, a system can be used to personalize the system and it is expected to have a more detailed knowledge of A in order to allow activities and transactions to be performed in the name of the user.

6.1. User history and "lifetime library"

One of the most concise and useful functions that the system can perform is the user's record of the text captured by the user, details of any document found, its location within this document, and any action taken as a result. To retain any additional information related to the capture.

This stored history is beneficial for both the user and the system.

6.1.1 for the user

The user can be given a "life library", which is a record of everything the user has read and captured. This may simply be a personal concern, but may be used in a library by a scholar, for example, collecting data for the section of his next paper.

In some circumstances, a user may wish to publish a library by publishing it on the Web in a similar way to a weblog, so that others can know and read about it.

Finally, if the user acquires some text and the system cannot immediately act on this capture (eg, because an electronic version of the document is still available), the capture can be stored in a library and meet the needs of the user. Accordingly or automatically, afterwards. The user can subscribe to new markup services and apply them to previously captured scans.

6.1.2. For the system

Records of past captures of users are also useful for the system. Many aspects of system operation can be improved by knowing the user's reading habits and history. The most straightforward example is that any scan done by the user is likely to come from the document the user last scanned, especially if the previous scan was done a few minutes before the last one. Similarly, documents are more likely to be read in start-off order, so, for example, in the case of English documents, later scans are more likely to occur farther down in the document. Such factors can help the system locate the capture in the case of ambiguity and can also reduce the amount of text to be captured.

6.2 Scanners such as Payment , Identity and Authentication Defense

Since the capture process begins with some kind of device, such as an optical scanner or voice recorder, the device can be used as a key to identify the user and authenticate certain actions.

6.2.1. Association of scanner with phone or other account

The device may be embedded in the mobile phone or otherwise associated with the mobile phone account. By way of example, the scanner may be associated with a mobile phone account by inserting a SIM card associated with the mobile phone account into the scanner. Similarly, the device may have functionality for a card embedded in or connected to a credit card or other payment card. The device can be used as a payment token and the payment transaction can be initiated by capture from the rendered document.

6.2.2. Use scanner input for authentication

The scanner may also be associated with a particular user or account through the process of scanning for some tokens, symbols or text associated with the user or account. The scanner may also be used for biometric information, for example by scanning a user's fingerprint. In the case of an audio-based capture device, the system may identify the user by matching the user's voice pattern or by requiring the user to speak a certain password or phrase.

As an example, if the user is provided with the option to scan certain quotes from the book or purchase the book from an online merchant, the user may select this option and then prompt to scan the user's fingerprint to confirm the transaction.

Section 15.5. See also 15.6.

6.2.3. Security scanning device

When a capture device is used to authenticate and identify a user and initiate a transaction on behalf of the user, it is very important that the communication between the device and other parts of the system is secured. It is also very important to protect against situations where other devices band the scanner, or so-called "intermediate" attacks where communication between other components and devices is intercepted.

Techniques for providing such security are well understood in the art; Various embodiments, hardware and software in the device, and elsewhere in the system, are configured to implement such techniques.

7. Model and Element Publishing

The advantage of such a system is that there is no need to change the conventional process of creating, printing and publishing documents in order to take advantage of the system's many advantages. The creator or publisher of the document, hereafter simply referred to as the "publisher", has a reason for wanting to create a function that supports the above system.

This section is mainly related to published documents. See section 10 titled "P-Commerce" for information about other commercial transactions, such as advertisements.

7.1. Electronic companion to the printed document

The system allows the document to have an associated electronic presence. Often, publishers place books on CD-ROMs containing additional digital information, teaching movies, and other multimedia data, sample code or documents, or additional reference materials. In addition, some publishers may have information that may be updated after the time of publication, and certain publications that provide such material, such as typos, additional comments, refined reference materials, indexes, and additional sources of related data, and translations into other languages. Maintain an associated website. Online forums allow readers to comment on publications.

The above system allows such materials to be more tightly coupled to the rendered document than before, and their discovery and interaction with them can be made easier for the user. By capturing a portion of the text from the document, the system can automatically connect the user to the digital material associated with the document, more specifically associated with a particular portion of the document. Similarly, a user can be connected to comments and interpretation by an online community that discusses that section of text, and by other readers. In the past, such information would typically need to be filled by searching for a particular page or chapter.

This exemplary application is in the field of academic texts (section 17.5).

7.2. "Sign Up" for Printed Documents

Some publishers may have a mailing list that readers can subscribe to if they want to be notified of new relevant material or if a new edition of the book is published. With the described system, a user can register an interest in a particular document or portion of a document and register even if the publisher is before considering providing any such functionality. Readers' interests may be provided to publishers and may influence their decisions as to when and when to provide entirely new publications on updates, additional information, new editions, or topics that have been identified as interest in existing books. Can be.

7.3. Printed marks with specific meaning or containing specific data

Many aspects of the system are simply enabled through the use of text already present in the document. If the document is produced under the knowledge that it can be used in conjunction with the system, the extra function can be added by printing the extra information in the form of a specific mark, which can be used to identify the text or the required action, Or otherwise improve the system's interaction with the document. An obvious and most important example is to instruct the reader that the document is clearly accessible through the system. A particular term icon can be used to indicate that the document has an online forum associated with it.

Such symbols may be purely intended for the reader or may be recognized by the system when used and scanned to initiate certain actions. Full data can be encoded into symbols to identify more than symbols; It can also store information about the position of the symbol, the version, the document, which can be recognized and read by the poem.

7.4. Authenticate by owning a paper document

There are some situations in which ownership of or access to a printed document grants a user certain privileges, such as, for example, access to an electronic copy or additional material of the document. With such a system, such privileges can be granted simply as a result of the user capturing a portion of the text from the document, or scanning a particular printed symbol. If the system needs to ensure that the user has owned the entire document, it can prompt the user to scan a particular item or phrase, such as "second line of page 46," from a particular page. .

7.5. Expired documents

If the printed document is a gateway to extra material and functionality, access to such features may be time limited. After the expiration date, the user may be required to obtain a new version of the document to pay a fee or to access such features again. Paper documents are still available, of course, but they lose some of their enhanced electronics. This is advantageous if the publisher receives a fee for access to the electronic material, or if the user sometimes needs to obtain a new version, or is associated with an older version of the printed document that remains during distribution. Because there is, it is hoped.

7.6. Popularity Analysis and Publishing Decisions

Section 10.5 discusses the use of the statistics of the system to influence advertising prices and author compensation.

In some embodiments, the system infers the popularity of the publication from the activity of the paper document as well as the electronic communication book associated with it. These factors assist in making decisions about what to publish in the future of the publisher. If the chapter turns out to be too popular in an existing book, it may be worth expanding to a separate publication.

8. Document Access Service

An important aspect of the described system is to give a user with access to a rendered copy of a document the ability to access an electronic version of that document. In some cases, the document is freely available on a private or public network that the user has access to. The system uses the captured text to identify, detect, and retrieve the document, and in some cases displays it on the user's screen or places it in the user's email box.

In some cases, the document is available in electronic form but may be inaccessible to the user for various reasons. There may not be enough connections to retrieve the document, the user may not have permission to search it, there may be a cost associated with the authority to access it, or the document may be new, just to name several simplicity. It is replaceable with the version and can be withdrawn. The system typically provides feedback to the user about these situations.

As described in section 7.4, the degree or nature of access granted to a particular user may be different if the user already has access to a printed copy of the document.

8.1. Authenticated document access

Access to the document may be limited to a particular user or a user who meets certain criteria, or may only be available in certain circumstances, such as when the user is connected to a secure network. Section 6 describes the user's trust and some ways in which the scanner can be established.

8.2. Buying Documents -Clone-Ownership Reward

Electronic documents, which are freely available in the public, may be accessible as a payment or as a reward for the publisher or copyright holder. The user may implement a payment facility or use a payment method associated with the user, including those described in section 6.2.

8.3 Documentation Escrow and Proactive Search

Electronic documents are often temporary; Digital source versions of the rendered document are currently available but may not be accessible in the future. The system retrieves and stores existing versions on behalf of the user, even if the user is not required, thus ensuring that availability will be available if you request it in the future. It is also available for use of the system, such as searching as part of the process of identifying future captures.

In the case where payment is required for access to a document, a trusted "document escrow" service is such as in the case of payment of the most appropriate fee, with the guarantee that the copyright holder will be fully compensated in the future if the user must request the document from the service. As such, the document can be retrieved on behalf of the user.

Variations on this subject can be implemented if the document is not available in electronic form at the time of capture. The user may authorize the service to make a payment or submit a request for the document on behalf of the user if the electronic document should be available at a later date.

8.4. Associate with other subscriptions and accounts

Sometimes payments may be waived, reduced or met based on an existing association with another account or subscription. Subscribers to the printed version of the newspaper may be automatically entitled to retrieve the electronic version, for example.

In other cases, the association may be less direct; Users can be granted access based on accounts established by their employers, or based on scanning of printed copies sourced by friends who are subscribers.

8.5 Replacing photocopying with scanning and printing

The process of capturing text from a paper document, identifying an electronic original, and printing a portion or original of the original associated with the capture form an alternative to conventional photocopying with various advantages.

The paper document does not have to be in the same position as the final printed copy, and in any case does not have to be there at the same time.

Wear and damage caused by paper photocopying processes, especially old, weak and valuable documents, can be prevented.

The quality of the copy is usually higher.

A record may be kept of which document or part of the document is most frequently copied.

Payment can be made to the proprietor as part of the process.

Unauthorized copying is prohibited.

8.6 positioning valuable originals from photocopy

In particularly valuable cases, such as when documents are legal documents or documents of historical or other specific importance, people often use copies of these documents for four years, while keeping the originals in a safe place.

The described system can be connected to a database that records the location of the original document, such as in a storage warehouse, which facilitates access to the copy to find the original paper document stored by someone.

9. Text Recognition Technology

Optical character recognition (OCR) technology has traditionally focused on images containing large amounts of text, from flat-bed scanners that capture entire pages. OCR techniques often require significant training and correction by the user to produce useful text. While OCR technology often requires significant processing power for machines that perform OCR, many systems use dictionaries, which are expected to operate effectively in infinite terms.

All of the above features can be improved with the system described above.

While this section is focused on OCR, many of the issues discussed map directly to other recognition technologies, especially voice inlets. As described in section 3.1, the process of clinging from the paper is accomplished by the user by reading text into the device capturing the audio. Those skilled in the art will appreciate that the principles described herein for images, fonts, and text fragments often apply to audio samples, user voice models, and phonemes.

9.1 Optimization for the right device

Scanning devices for use in such systems are compact, portable and low power. The scanning device can capture only a few words at a time, and in some implementations may not be able to capture one complete character at a time, while a single horizontal slice can be captured through text, many such horizontal slices being The text together forms a recognizable signal that can be inferred. The scanning device also has very limited processing power or storage and thus may perform all OCR processes in some embodiments, while many embodiments further, if possible, later to convert the captured signal into text. It depends on the connection to a powerful device. Finally, it has very limited functionality for user interaction, and therefore needs to delay any request for subsequent user input, or operate in "best-guessing" mode to a greater extent than now.

9.2 " Uneven " OCR

A major new feature of OCR in such systems is that in general, it can be retrieved in digital form and inspect an image of text present anywhere. Correct transcription of text is not necessarily required from the OCR engine. The OCR system outputs a matrix or set of possible matches including probability weights in some cases, which can be used to search for digital sources.

9.3 Iterative OCR-guessing, disambiguation, speculation ...

If the device performing the recognition can contact the document index during processing, the OCR process can be notified by the content of the document body as it proceeds, potentially providing significantly greater recognition accuracy.

Such a connection allows the device to notify the user if enough text has been captured to identify the digital source.

9.4. Use similar rendering knowledge

This may also assist in the recognition process, such as the font typeface used in the layout or printing of several aspects of a similarly printed rendering of a document, or the font typeface used in printing (section 4.1). One).

9.5. Font caching- font determination on host , download to client

As candidate source text in the document compass is identified, its font, or rendering, can be downloaded to the device to aid in recognition.

9.6. Auto Correct and Character Offset

Whereas the component characters of a text fragment may be the best recognized way of representing a fragment of text that can be used as a document signature, other representations of text may be found when attempting to place the text fragment in a digital document and / or database. In the case of clarifying the representation of the text fragment in a readable form, it can be well represented that the actual text of the text fragment does not need to be used. Other representations of text fragments can provide the advantage of a lack of actual text representation. By way of example, optical character recognition of text fragments is often an error, unlike other representations of captured text fragments that can be used to navigate to regenerate text fragments without relying on optical character recognition for the entire fragment. It tends to be easy. It may be more appropriate for some devices currently used in the system.

Those skilled in the art will appreciate that there are many ways to describe the appearance of a text fragment. Such characterization of text fragments includes, but is not limited to, the following: word length, relative word length, character height, character width, character type, character frequency, token frequency, and the like. In some embodiments, the offset between the matching text tokens (ie, the number of interference tokens plus one) is used to characterize the fragment of the text.

Conventional OCR uses knowledge of fonts, character structures and forms in an attempt to determine the characters of the scanned text. Embodiments of the invention are different; They employ various ways of using the rendered text itself to assist in the recognition process. These embodiments use characters (or tokens) to “recognize each other”. One example of grading self-awareness is "template matching", which is similar to "convolution". To perform such self-awareness, the system slices a copy of the text horizontally relative to itself and notes the matching area of the text image. Conventional template matching and convolution techniques include a variety of related techniques. Techniques for tokenizing and / or recognizing characters / tokens are used to correlate the characters / tokens with their own components if they match.

In the case of autocalibration, the fully connected region that matches is of interest. This occurs when a character (or group of characters) overlays another instance of the same character (or group). Fully matched regions that match automatically provide tokenization of text within component tokens. As two copies of the text slide past each other, an area is noted where complete matching occurs (ie all pixels of the vertical slice are matched). If a character / token matches itself, it also matches the horizontal range of the match (eg, the linked match of text).

At this stage there is no need to determine the actual identity of each token, i.e. a specific letter, number or symbol, or group thereof, corresponding to the token image, but only an offset for the next occurrence of the same token in the scanned text. Only need to decide. The offset number is the distance (number of tokens) to the next occurrence of the same token. If the token is unique within the text string, the offset is zero. The sequence of token offsets generated in this way is a signature that can be used to identify the scanned text.

In some embodiments, the token offset determined for the string of scanned tokens is compared to an index that indexes the body of the electronic document based on the token offset of their content (section 4.1.2). In another embodiment, the token offset determined for the string of scanned tokens is converted to text and compared to a more conventional index that indexes the body of an electronic document based on their content.

As noted above, a similar token-correlation process may be applied to speech fragments when the capture process consists of audio samples of speech words.

9.7. Font / character "self-recognition"

Conventional template-matching OCR compares the scanned image to a library of character images. In essence, the alphabet is stored for each font and the newly scanned image is compared with the stored image to find matching characters. This process continues with an initial delay until the correct font is identified. After that, the OCR process is relatively fast because most documents use the same font throughout. Subsequent images can thus be converted into text by comparison with the recently identified font library.

The most commonly used font's character form is relevant. As an example in most fonts, the letter "C" and the letter "e" are visually related-"t" "f" and so on. The OCR process is enhanced by using the relationship to construct a template for letters that have not yet been scanned. For example, if the reader scans a short string of text from a paper document of a font that has not been previously seen and the system does not have a set of image templates to compare with the scanned image, the system will not see all the letters of the alphabet yet. Possible relationships between certain characters can be leveraged to construct a font template library. The system can then use the configured font template library to recognize subsequent scanned text and to further refine the configured font library.

9.8. Anything not recognized (including graphics) is sent to the server

If the image cannot be machine transcribed into a suitable form for use in the search process, the image may be available for future use by the user on its own, for possible manual transcription, or with different resources available to the system. When possible, it can be archived for later processing.

10. P- Commerce

Many of the actions performed by the system are the result of several commercial transactions occurring. The phrase P-commerce is used herein to describe commercial activities initiated from paper through the system.

10.1. Sale of documents by physically printed copies

When the user captures text from the document, the user is provided with the document for purchase in paper or electronic form. The user is provided with documents that are cited or mentioned in the paper document, or documents by the same author or similar subject matter.

10.2. Sale of those initiated or assisted by paper

Capture of text can be linked to the activity of use in a variety of ways. The captured text can be a catalog designed to sell the item, in which case the text can be very directly related to the purchase of the item (section 18.2). The text may also be part of an advertisement, in which case the sale of the advertised item may continue.

In other cases, users capture other text whose potential interests may be inferred in commercial transactions. Readers of novel sets in a particular country may be interested in the holidays there. The user may capture a particular fragment of text that knows that some commercial opportunity may be presented to them as a result, or it may be ancillary to their capture activity.

10.3. Being sold Item top Capture of labels , icons, serial numbers, barcodes

Sometimes text or symbols are actually printed on an item or on its packaging. For example, the serial number or product id can often be found on a label on the bottom or back of the piece of electronic defense. The system provides the user with a convenient way to purchase one or more identical items by capturing the text.

10.4. Contextual Advertising

In addition to the direct capture of text from an advertisement, the system allows a new kind of advertisement that does not necessarily have to be explicit in the rendered document, but it is nevertheless based on what people read.

10.4.1. Ads based on scan context and history

In conventional paper publishing, advertisements consume a large amount of space relative to the text of a newspaper article, and a limited number of these can be placed around a particular article. In the system described above, advertisements may be associated with individual words or phrases and may be selected according to the particular interests shown by the user by capturing the text and considering their history for past scans.

With such a system, it is possible for advertisers to get a great deal of feedback on the validity of their advertisements for purchases that are closely tied to a particular printed document and for the effectiveness of their advertisement in a particular printed publication.

10.5. Model of compensation

The system enables several new models of rewards for advertisers and marketers. Publishers of printed documents containing advertisements may earn some revenue from purchases generated from their documents. This is true regardless of whether the advertisement is in the original printed form; It can be added electronically by the publisher or by a hard-liner or third party, and the source of such advertisements can be subscribed by the user.

10.5.1. Popularity-based reward

Analysis of the statistics generated by the system may indicate the popularity of the marginal portion of the publication (section 14.2). In a newspaper, it may indicate the amount of time a reader spent looking at a particular page or article, or the popularity of a particular document. In some circumstances, a writer or publisher may be eligible to be rewarded based on the activity of the reader rather than more traditional metrics, such as the number of copies distributed or the words recorded. An author whose work is frequently read for authority on the subject may be sold as a copy of his apparel, but may be considered different in future contracts with a rarely viewed writer.

10.5.2. Popularity-based advertising

The decision about advertising in the document may be based on statistics about the readership. Advertisements around the most popular columnists can be sold at a premium rate. Advertisers can be billed or compensated at certain times after a document is published based on knowledge of how it is accepted.

10.6. Marketing based on lifetime library

The scan history or "lifetime library" described in sections 6.1 and 16.1 can be an extremely valuable source of information about a user's habits or interests. Depending on the appropriate consent and privacy issues, such data notifies the user of the provision of goods or services. Even in the anonymous form, the statistics collected can be very useful.

10.7. Sales / information on later date (when available)

Advertisements and other opportunities for commercial transactions may not be immediately available to the user at the time of text capture. By way of example, the opportunity to purchase a qualifier for a novel may not be available at the time a user reads the novel, but the system may offer them an opportunity if the qualifier is published.

The user may capture data relating to purchases or other commercial transactions, but may not choose to initiate and / or complete the transaction at the time the capture was made. In some embodiments, data relating to capture is stored in a lifespan library, and these lifespan library entries may be in an "active" state (ie, capable of subsequent interaction similar to those available at the time the capture was done). Thus, the user can review the capture at some later point in time and can optionally complete a transaction based on the capture. Because the system can track when and where the original capture occurred, all parties involved in the transaction can be properly compensated. For example, a writer who writes a story-a publisher who published a story-that appears immediately after an advertisement that captures data by the user, selects a particular capture from history, if the user visited their life library six months later, And by purchasing "purchase this item from Amazon" from the pop-up menu (which may be the same as or similar to the menu optionally presented at the time of capture).

11. Operating system and application integration

Modern operating systems (OSs) and other software packages have a number of features that can be beneficially used with the system, and can be modified in various ways to provide a better platform for its use.

11.1. Integration of scan and print-related information in metadata and indexing

New and upcoming file systems and their associated databases often have the ability to store various metadata associated with each file. Typically, this metadata includes such things as the ID of the user who created the file, creation date, last modification, and last use. Newer file systems allow extra information such as keywords, image features, document sources and user comments to be stored and in some systems this metadata can be extended to arbitrary. The file system can therefore be used to store information useful for implementing the current system. By way of example, data may be stored by the file system when a given document was last printed, as the data may be used to describe in detail which text and from whom was captured from the paper using the system.

The operating system also begins incorporating search engine functionality that allows the user to more easily find local files. These functions can be beneficially used by the system. That means that many of the search-related concepts discussed in sections 3 and 4 apply to all personal computers as well as to today's Internet-based and similar search engines.

In some cases, a particular software application will include a set of capabilities provided by the OS and support for the system beyond it.

11.2. capture OS support for devices

As the use of capture devices, such as pen scanners, is becoming more common, it is desirable to build a support system into the operating system in much the same way as the support provided for mice and printers, where the applicability of the capture device may be limited to a single software application. Because it extends beyond. The same is true for other aspects of the operation of the system. Some samples are described below. In some embodiments, the entire described system, or key portion thereof, is provided by the OS. In some embodiments, support for the system is provided by application programming interfaces (APIs) that can be used by other software packages, including directly implementing aspects of the system.

11.2.1. Support for OCR and Other Recognition Technologies

Most methods for capturing rendered documents require some recognition software to interpret the source data, such as scanned images or some spoken words, such as text suitable for use with the system. Some OSs include support for voice or handwriting recognition, although it is less common for the OS to include support for OCR, since the use of OCR in the past has been limited to a small range of applications.

As the recognition component becomes part of the OS, they can take advantage of the other facilities provided by the OS. Most systems include spelling dictionaries, grammar analysis tools, phraseization and localization facilities, all of which can be beneficially employed by the system for the recognition process of the system, as they are specific users. Can be customized for a particular user to include words and phrases that are often encountered.

If the operating system includes a full-text indexing facility, these may also be used to inform the recognition process, as described in section 9.3.

11.2.2. Actions to be Taken for Scanning

If an optical scan or other capture occurs and is given to the OS, it has a default action to be taken under circumstances where no other system claims ownership of the capture. Examples of default actions are to present an alternative option to the user or to send captured text to the OS's built-in search facility.

11.2.3. OS has default actions for specific documents or document types

If a digital source of a rendered document is found, the OS may have a standard action to be taken when that particular document or class of documents is scanned. Applications and other subsystems can register ODs as potential handlers for certain capture types, in a manner similar to broadcast by applications for their ability to handle certain file types.

The markup data associated with the rendered document, or capture from the document, may include instructions to launch a particular application to the operating system and pass application arguments, parameters, data, and the like.

11.2.4. Into a standard action Mapping and Interpreting Gestures

The use of "gestures" is described in section 12.1.3, and especially in the case of optical scanning, the place where a particular move is made with the handheld scanner represents a standard action, such as marking the beginning and end of an area of text.

This is similar to an action, such as using the cursor keys to select an area of text, or pressing the shift key on the keyboard while using the whee on the mouse to scroll the document. Such actions by the user are sufficiently standard that they are hacked in a system-wide manner by the OS, thus ensuring a constant action. The same action is desirable for scanner gestures and other scanner-related actions.

11.2.5. Set response to standard (and non-standard) icons / text printed menu items

In the same way, items of text or other symbols of text cause standard actions to occur when scanned, and the OS can provide a choice for them. By way of example, scanning the text "[print]" in any document may allow the OS to retrieve and print a copy of that document. The OS provides a way to register such actions and associate them with a particular scan.

11.3. Support in system GUI components for typical scan-initiated activities

Most software applications are based on standard graphical user interfaces provided by the OS.

The use of these components by the developer is, for example, as pressing the left-cursor in any text-editing context moves the cursor to the left, without all programmers independently implementing the same functionality. It helps to ensure consistent behavior across multiple packages.

Similar consistency in these components is desirable when the activity is initiated by a text-capture or other aspect of the system described above. Some examples are given below.

11.3.1. Interface for finding specific text content

Typical use of this system may be scanning a portion of a paper document for a user, opening an electronic counterpart in a software package that can be displayed or manipulated for the system, scrolling and scanning the scanned text. May be highlighted (section 12.2.1.). The first part of this process of discovering and opening electronic documents is typically provided by the OS and is standard throughout the package. However, the second part, which places a particular part of the text in the document and causes the package to scroll and highlight it, has not yet been standardized and is often implemented differently by each package. The availability of standard APIs for this function significantly improves the motivation of this aspect of the system.

11.3.2. Text interaction

If a piece of text is located in a document, the system will want to perform various actions on that text. By way of example, the system may require surrounding text, so the user's capture of several words results in accessing the entire sentence or paragraph containing them in the system. Again, this functionality can be usefully provided by the OS rather than being implemented in every piece of software that handles text.

11.3.3. Contextual ( popup ) menu

Some of the operations enabled by the system require user feedback, which may be optimally desired within the context of the data handler. In some embodiments, the system typically uses an application popup menu associated with clicking the right mouse button over some text. The system inserts extra options into such menus and allows them to be displayed as a result of the activity as scanning a paper document.

11.4. Web / network interface

In today's growing networked world, most of the functionality available on individual machines can be accessed through the network, and the functionality associated with such systems is no exception. As an example, multiple paper documents received by a user in an office environment may be printed by other users' machines on the same joint network. In response to the capture, a system on one computer may, depending on the appropriate permission controls, query the other machine for a document that may correspond to the capture.

11.5. Printing of Documents That Cause Archiving

One important factor in the integration of paper and documents is to keep as much information as possible about the conversion between the two. In some embodiments, the OS maintains a single record of when the document was printed by Luke. In some embodiments, the OS takes one or more actions that make it more suitable for use of the system. Examples include the following:

Keep a digitally rendered version of every printed document with information about the source from which the document was printed

Keep a subset of useful information about the printed version, as well as the fonts used and where line breaks occurred as examples, which can help in future scan interpretation.

Keep a version of the source document associated with any printed copy

? Automatically index documents on print and save results for future navigation

11.6. My (Printed / Scanned ) Documents

The OS often maintains a certain category of files or folders of certain importance. The user's document can be found by way of design or by way of example in the "My Documents" folder. The standard file-open dialog may automatically include a list of recently opened documents.

On an OS optimized for use with such a system, such categories can be enhanced or augmented in a manner that takes into account the user's interaction with the paper version of the stored file. Categories such as "My Documents" or "My Recently-Read Documents" can be usefully identified and incorporated into their operation.

11.7. OS-level markup hierarchy

Since the critical aspects of the system are provided using the "markup" concept discussed in Section 5, it is beneficial to have a support framework for such markup provided by the OS in a manner that was accessible to the OS itself and multiple applications. Further, the layers of markup may be provided by the OS, based on the functionality that the OS can provide and its knowledge of the document under its control.

11.8. Use of OS DRM Features

The growing number of operating systems includes "digital rights management"; It supports some form of software entity or machine control over the use of specific data, depending on the privileges granted to specific users. As an example, unauthorized copying or distribution of certain documents may be prohibited.

12. User Interface

The user interface of the system may be entirely on the PC if the capture device is relatively poorly functioning or connected by cable, or it may be entirely on the device if it is sophisticated or itself has significant processing power. Some or all of the functionality of the system may be implemented on other devices such as mobile phones or PDAs.

The description in the next section is about what implementation may be desirable, but it is not necessarily appropriate in all cases and can be modified in many ways.

12.1. capture On the device

With all capture devices, but especially in the case of optical scanners, the user's attention is generally placed on the device and on the paper during scanning. The feedback and any input required as part of the scanning process is preferably not required to distract the user beyond what is necessary, such as on a screen of a computer, for example.

12.1.1. Feedback by the scanner

Handheld scanners have a variety of ways of providing feedback to the user about specific conditions. The most obvious type is the direct visual type when the scanner has an indicator light or full display, and an audible type when the scanner has an alarm, click or other sound. An important alternative is tactile feedback, in which case the scanner simulates vibrations, beeps, or otherwise the user's touch sense, and projection feedback indicating the state by stripping anything from the colored light spot to the sophisticated display. It includes.

Important direct feedback that may be provided to the device includes the following.

Feedback by the scanning process-the user's scanning is overspeed, too large an angle, or drifting too high or too low on a particular line

Sufficient content-important for disconnect behavior-if a match exists, the discovery of one match has been scanned sufficiently to ensure a certain degree

Known Context-Source of Text Found

Known Unique Context-One Unique Source of Text Found

Availability of content-an indication of whether the content is available to the user for free or for a fee

For example, if there is sufficient performance to display some or all of the document, much of the user interaction normally associated with later stages of the system may occur at the capture device.

12.1.2. Scanner control

The device may provide a variety of methods for the user to input in addition to basic text capture. Even when the device is closely associated with a host device having input options such as a keyboard and a mouse, it may be disruptive for the user to switch back and forth between, for example, operating the scanner and using the mouse.

Handheld scanners may include buttons, scroll / jog wheels, touch sensitive surfaces, and / or accelerometers for sensing device movement. Some of these enable a richer set of interactions while still maintaining the scanner.

For example, in response to scanning any text, the system provides the user with some possible set of matching documents. The user selects one of the lists using the scroll wheel on the side of the scanner and confirms the selection by clicking the button.

12.1.3. gesture

The main reason for moving the scanner in the middle of the paper is to capture text, but some movements can be detected by the device and used to indicate other intentions of the user. This movement is referred to herein as a gesture.

For example, a user may indicate a large area of text by scanning the first few words in conventional left-right order and the last few words in reverse, ie right-left order. The user may also indicate the vertical range of the target text by moving the scanner down the page a few lines up. The scan may then indicate the cancellation of the previously scanned operation.

12.1.4. Online / offline behavior

Many aspects of the system may rely on network connectivity between the system components, such as scanners and host laptops, or with the outside world in the form of connections to company databases and Internet searches. However, it will be the case that such connectivity may not always exist and thus may be considered to be "offline" of some or all of the system. It is desirable to keep the system useful in this situation.

The device can be used to capture text when not in contact with another part. Very simple devices can simply store image or audio data associated with a timestamp that ideally indicates when it was captured. When the device is in contact with the rest of the system, various captures can be uploaded here and then processed. The device may also upload other data associated with the capture, such as voice annotations or location information associated with the optical scan.

More complex devices can perform some or all of the system's operations themselves, even when not connected. Various techniques for improving the performance of this are described in Section 15.3. Sometimes some but not all of the desired actions can be performed while offline. For example, text may be recognized, but source identification may be dependent on a connection with an internet based search engine. Thus, in some embodiments, the apparatus stores enough information about how much each operation has progressed so that the rest of the system proceeds effectively when the connection is restored.

The operation of the system is generally useful by means of ready-to-use connections, but there are some situations where it may be beneficial to take some captures and process them in batches. For example, as described in section 13 below, the identification of a particular capture source can be greatly enhanced by examining other captures performed by the user at about the same time. In a fully connected system where live feedback is provided to the user, past captures can only be used when the system processes the current capture. However, if one of the batches saved by the device when the capture is offline, the system may consider any data available from previous and subsequent captures when performing this analysis.

12.2. As host device

Scanners often communicate with some other device such as a PC, PDA, telephone or digital camera to perform many of the functions of the system, including more detailed interactions with the user.

12.2.1. Action taken in response to capture

When the host device receives the capture, it can initiate various operations. Following the locating, there is an incomplete list of possible actions the system has performed, along with a copy of the electronic document and the location within the document associated with this capture.

● Detailed information of the capture can be stored in the user history (Section 6.1).

● Documents can be retrieved from local storage or remote (section 8).

Update the metadata of the operating system and other records associated with the document (section 11.1);

• You can examine the markup associated with the document to determine the appropriate action to follow (section 5).

● Launch a software application to perform document editing, viewing, or other operations. Application selection depends on the content of the source document or scan, or on some other aspect of capture (Sections 11.2.2, 11.2.3).

● You can scroll through the application to highlight, move the insertion point, or indicate the capture location (section 11.3).

You can modify the exact boundaries of the captured text, for example to select whole words, sentences or phrases around the captured text (section 11.3.2).

● Give the user the option to copy the capture text to the clipboard or to perform other standard operating systems or application specific actions on it.

● You can associate comments with documents or captured text. This may be done directly through user input or may have been captured first in the case of a voice annotation, eg associated with an optical scan (section 19.4).

Markup can be examined to determine another possible set of actions for the user to select.

12.2.2. Context popup menu

Sometimes the proper action the system takes is clear, but in some cases it's up to the user. One good way to do this is to use a "pop-up menu" or, if the content is displayed on the screen, a so-called "context menu" that appears close to the content (see section 11.3.3). In some embodiments, the scanner device displays a pop-up menu on the paper document. The user may interact with the computer display using a scanner, either from a menu using traditional methods such as a keyboard and mouse, or by using controls on the capture device (section 12.1.2), by using gestures (section 12.1.3), or by using a scanner. It can be chosen by action (Section 12.2.4). In some embodiments, the popup menu that may appear as a result of the capture includes a default item that appears if the user does not respond, e.g., an action that occurs when the user ignores the menu and performs another capture.

12.2.3. Feedback on clarification

When the user initiates text capture, there will be some document or other text location that can be matched first. As more text is captured and other factors are taken into account (Section 13), the number of candidate locations is reduced until the actual location is identified or no further clarification is possible without user input. In some embodiments, the system displays in real time the locations found for the number of elements in the display in the form of a document or for example in the form of a list, thumbnail image or text segment and to reduce the number as capture continues. In some embodiments, the system displays thumbnails of all candidate documents, the size and position of which thumbnails depend on the probability of an exact match.

Once the catcher is clearly identified, it can be emphasized to the user, for example using audio feedback.

Sometimes the captured text will appear in many documents and will be recognized as a quotation. The system may indicate this on the screen, for example, by grouping documents containing cited references around the original source document.

12.2.4. Scanning from the screen

Some optical scanners can capture text displayed on the screen as well as paper. Thus, the term 'rendered document' is used herein to indicate that paper prints are not the only form of rendering and that the capture of text or symbols for use in the system can also be equally valuable when the text is displayed on an electronic display. do.

The user of such a system may need to interact with the computer screen for a variety of different reasons, such as selecting a list of options. It may be inconvenient for the user to place the scanner and initiate it using a mouse or keyboard. While other sections have described the physical controls of a scanner (described in section 12.1.2) or a gesture (described in section 12.1.3) as a way to enter these tools without having to change them, some text can be obtained using the scanner on the screen itself. B Scanning symbols is an important alternative provided by this system.

In some embodiments, the optics of the scanner can be used in the same manner as the light pen, so that with the help of the special hardware or software of the computer, the position on the screen can be detected directly without the need to actually scan the text.

13. Context Description

An important aspect of the system is that it uses other factors beyond simple capture of text strings to help identify the documents used. Capturing an appropriate amount of text will often uniquely identify the document, but in many cases will identify some candidate documents. One solution is to allow the user to confirm the document being scanned, but the preferred alternative is to use other factors that automatically narrow down the possibilities. Such supplemental information can significantly reduce the amount of documents requiring capture and / or significantly increase the reliability and speed with which the location of the electronic copy can be identified. This extra material is called "context" and is briefly described in Section 4.2.2. We will look deeper later.

13.1. System and capture context

Perhaps the most important example of this information is the user's capture history.

It is likely that a given capture comes from the same document as the previous one, or especially from the associated document if the previous capture occurred in the last few minutes (section 6.1.2). Conversely, if the system detects that the font has changed between two scans, it is likely that it is from another document.

The user's long-term capture history and reading habits are also useful. It can also be used to develop a model of interest and association of users.

13.2. User's Real Context

Another example of a useful context is the user's geographic location. For example, users in Paris are much more likely to read Le Monde than the Seattle Times. Thus, the timing, size, and geographical distribution of the printed version of the document can be important and can be speculated to some extent in the operation of the system.

For example, the time of day may also be relevant for a user who always reads one type of publication while going to work and who reads something else on the train during lunchtime or leaving home.

13.3. Related Digital Context

User's recent use of electronic documents, including those searched and retrieved by more conventional means, may also be useful indicators.

In some cases, such as a corporate network, other factors can be considered useful:

● Has the document been recently printed?

● Has the document been recently modified on a corporate file server?

● Has the document been recently emailed?

All of these examples may imply that the user is likely to read more paper version documents. Conversely, if a canteen with a document can be sent to any place where it may have been printed or can be confirmed that the document has never been printed, it can be safely removed from any search originating from the paper.

13.4. Miscellaneous Statistics-Global Context

Section 14 covers the analysis of data streams from paper-based retrieval, but here all of the statistics on the reputation of documents with other readers, the timing of those reputations, and some of the documents that are most frequently scanned may be useful for the retrieval process. It should be recognized that this is an example of more factors. The system enables Google to rank pages in the paper world.

See also Section 4.2.2 for some other meanings of the context for search engines.

14. Data string analysis

As a side effect of the use of this system, it generates excessively valuable data streams. This data string is a record of what the user reads when and in many cases a record of what the user finds especially valuable in what they are reading. Such data has never been truly available in paper documents before.

Some of the ways in which this data can be useful for the system and its use are described in Section 6.1. This section focuses on other uses. Of course there are important privacy issues that can be considered as any data distribution to what people are reading, but issues such as preserving the anonymity of the data are well known to those skilled in the art.

14.1. Document tracking

If the system knows which document a given user is reading, the system can also infer who is reading a given document. This enables tracking of documents throughout the organization, for example, who reads the document, how widely distributed it takes, how long it takes to distribute, and who has seen the current version while others are working on it from an outdated copy. Enable analysis.

For published documents with a wider distribution, tracking individual copies is more difficult, but analysis of the distribution of readership is still possible.

14.2. Reading ranking-popularity of documents and subregions

In situations where a user captures text or data of particular interest to them, the system can infer the popularity of a particular document and the popularity of a subregion of that document. This forms valuable input to the system itself (Section 4.2.2) and important sources of information for writers, publishers and advertisers (Section 7.6, 10.5). Such data can also be integrated in search engines and search indexes, for example, to help rank search results for queries from rendered documents, and / or to rank prior queries typed within a web browser. Useful when

14.3. User Analysis-Profile Creation

Knowing what the user is reading allows the system to generate a fairly detailed model of the user's interests and activities. This may be useful for a general statistical basis—for example, it may enable other interactions with individual users as described below, such as “35% of users who purchase such newspapers also read the author's recent books”. .

14.3.1. Social networking

One example is to connect one user with another person with a related interest. This may be a person already known to the user. The system can ask a university professor, "Did you know your colleague at XYZ University is reading this paper?" The system may ask the user, "Do you also want to connect with your neighbor who is reading Jane Air?" This connection can be the basis for the automatic formation of intimate social structures with book clubs, both in the physical world and online.

14.3.2. Marketing

In Section 10.6, we have already mentioned the idea of providing products and services to individual users based on their interaction with the system. For example, current online book sellers often recommend users based on their previous interactions with the book seller. This recommendation becomes even more useful when based on the actual interaction with the book.

14.4. Marketing based on other aspects of the data stream

Some of the ways in which the system can affect the publishing of documents, the advertising of the documents, and other sales initiated by paper have been described (v. 10). Some commercial activities eventually do not have direct interaction with paper documents, but may be affected. For example, it may be of interest to a person trying to do a health club to know that people in one community spend more time reading the sports sector than the financial sector of the newspaper.

14.5. Data types that can be captured

In addition to the above statistics, such as who reads what extent and when of a document, it may be of interest to examine the actual content of the captured document regardless of whether the document is located.

In many cases, the user will not only capture some documents but also cause some activity to occur as a result. This could be, for example, emailing a reference to a document to an acquaintance. It is very useful to know that someone considered this document worth sending an email even when no user or email recipients were identified.

In addition to the various methods described for inferring the value of a particular document or portion of text, in some cases the user will externally indicate the value by assigning a rating to it.

Finally, if a particular set of users is known to form a group, such as if the users are known to an employee of a particular company, then the collective statistics of that group can be used to infer the importance of a particular document for that group.

15. Device features and functions

A capture device for use with the system only needs a way of capturing text from the rendered version of the document. As noted above (section 1.2), this capture can be accomplished through a variety of methods, including taking a picture of a portion of the document or typing some characters on a mobile phone keyboard. Such capture may be performed using a small handheld optical scanner capable of recording one line or two text at a time, or using a voice capture device such as a voice record in which the user reads text from a document. The device used can be a combination of these, such as an optical scanner, which can also record voice annotations, and the capturing function can be embedded in some other device such as a mobile phone, PDA, digital camera or portable music player.

15.1. I / O

Many useful additional I / O devices for this device are described in Section 12.1. It includes buttons, scroll wheels and touch pads for input and a display, indicator light, voice and tactile transducers for output. Sometimes the device may add most or little of them. Sometimes the capture device will be able to communicate with other devices already equipped with it, for example using a wireless link (section 15.6), and sometimes a capture function will be added to this other device (section 15.7).

15.2. Connectivity

In some embodiments, the apparatus implements most of the system itself. However, in some embodiments, the device often communicates with a PC or other computing device and uses a communications facility to communicate with the wider world.

Typically, these communication facilities are in the form of general-purpose data networks such as Ethernet, 802.11 or UWB, or in the form of standard peripheral access networks such as USB, IEEE-1394 (Firewire), Bluetooth ^™ or infrared. When a wired connection such as Firewire or USB is used, the device can be supplied with electricity through the connection. In some cases, the capture device may appear on a connected machine and become a conventional peripheral such as a USB storage device.

Finally, the device may in some cases be used in conjunction with this device or "dock" with another device for convenient storage.

15.3. Caching and Other Online / Offline Features

Sections 3.5 and 12.1.4 dealt mainly with separate operations. If the capture device does not communicate with other parts of the system with a limited subset of the overall system functionality, the device may still be useful even if the available functionality is sometimes reduced. In the simplest step, the device can record the raw image or audio data to be captured, which can then be processed. However, for the user, feedback where possible about whether the captured data is likely to be sufficient for handwork, whether it can be recognized or likely to be recognized, and whether the data source can be identified or later identified. It may be important to provide The user will then know whether the capturing activity is valuable. Even if all of the above is unknown, the raw data can still be stored and thus at least the user can refer to it later. The user may be provided with the scanned image, for example, when the scan cannot be recognized by the OCR process.

To illustrate some of the range of options available, rather the following describes a minimal optical scanning device and then a much more full-featured scanning device. Many devices occupy a midpoint between these two.

15.3.1. Simple Scanner-Affordable Offline Example

Simple scanners have a scanning head that can read pixels on the page as they travel along the length of the text line. It can detect motion along the page and record pixels with some information about this motion. It also has a clock that allows each scan to be time stamped. The clock is synchronized with the host device when a simple scanner is connected. The clock cannot represent the actual time of the day, but relative time can be determined so that the host can infer the actual time of the scan or, in the worst case, the elapsed time between scans.

Simple scanners do not have enough processing power to perform OCR itself, but have some basic knowledge about their relationship to typical word length, word spacing, and font size. This tells the user when the scan may be readable, whether the head is moving too quickly, too slowly or too inappropriately across the paper, and when enough words of a certain size may have been scanned for the document to be identified. It contains some basic indicator lights.

Simple Scanner has a USB connection and can be plugged into a USB port on your computer, where it will be charged. On a computer, a simple scanner appears to make the time stamped data file a Grogodin USB storage device, and the rest of the system software takes over from this point.

15.3.2. Super Scanner-Advanced Offline Example

Superscanners also depend on connectivity for their overall operation, but this has a significant amount of onboard storage and processing that can help make better judgments about the data captured while offline.

As the superscanner moves along a line of text, the captured pixels are stitched together and passed to an OCR engine that attempts to recognize the text. Many fonts, including those from the user's most read publications, are downloaded and synchronized with the user spelling checker dictionary on the PC, thus helping to do this, as they have a dictionary containing many words that the user frequently encounters. Gives. In addition, a list of words and expressions with frequency of use can be stored on the scanner, which can be combined with a dictionary. The scanner can use frequency statistics to aid in the recognition process and to inform the decision as to when a sufficient amount of text was captured. More frequently used expressions are likely to be less useful as the basis for search queries.

In addition, a full index of the most commonly read newspapers and articles of the latest issue of the periodical is stored on the device as if it were an index of the books you recently purchased from an online book seller or if you scanned something in the last few months. Save it. Finally, the titles of thousands of the most popular publications with data available to the system are stored so that when there is no other information, the user can scan the title and capture from a specific task can then be retrieved electronically. You can have a good idea about the same thing.

During the scanning process, the system informs the user that the captured data is of sufficient quality and that the electronic copy has sufficient characteristics to be retrieved when the connection is restored. Often the system is informed to the user that the scan was successful and the context was recognized in one of the onboard indexes, and the publication of interest also informs that the data is known to be available in the system and that a subsequent, search must be successful.

Superscanners dock to cradles connected to a PC's FireWire or USB port, and in addition to uploading captured data, various onboard indexes and other databases are updated based on recent user activity and new publications. Also provided is a device for connecting to a wireless public network or for communicating with a mobile phone via Bluetooth, which device can communicate with the public network when available.

15.4. Structure for Optical Scanning

Some structures that may be particularly desirable for optical scanner devices are discussed below.

15.4.1. Flexible positioning and convenient optics

One reason why paper continues to be popular is that it is convenient to use in a variety of situations, for example, where computers are not practical or inconvenient. Thus, a device for capturing a significant portion of the user's interaction with the paper must likewise be easy to use. In the past, this was not the case with scanners, and even the smallest handheld device was somewhat unwieldy. Designed to come into contact with the page had to be kept at a precise angle with the paper and moved very accurately along the length of the text to be scanned. This may be acceptable when scanning a business report on an office desk, but may be impractical when scanning a novel's expression while waiting for a train. Scanners based on camera type optics operating at a distance from paper may likewise be useful in some situations.

Some embodiments of the system use a scanner that uses an image conduit, which is an optical fiber bundle, to scan in contact with paper and transfer an image from a page to an optical sensor device instead of a lens. Such a device may be configured to be held in its natural location. For example, in some embodiments, the portion that is in contact with the page is wedge shaped, which may allow a user's hand to move more naturally on the page in the same motion as using a highlighter pen. The conduit may be in direct contact with or in close contact with the paper and have a replaceable transparent tip that may protect the image conduit from possible damage. As described in Section 12.2.4, the scanner can be used to scan from the screen as well as paper and the tip material can be selected to reduce the likelihood of damage to these displays.

Finally, some embodiments of the device instruct the user by using light, sound or tactile feedback when the user scans too quickly, too slowly, too non-uniformly or drifts too high or too low on the scanning line during the scanning process. something to do.

15.5. Security, Identification, Authentication, Personalization, and Billing

As described in section 6, the capture device may form an important part of the identification and authentication for secure transactions, purchases, and various other operations. Thus, in addition to the circuitry and software required for this role, a variety of hardware architectures can be added to enhance security, such as smart card readers, RFID, or keypads for typing PINs.

It may also include various biometric sensors to aid in user identification. For example, in the case of an optical scanner, the scanning head may read a fingerprint. The voice pattern of the user can be used for the voice recorder.

15.6. Device association

In some embodiments, the device may form an association with other nearby devices to increase itself or their functionality. For example, in some embodiments, the display of a nearby PC or phone is used or a network connection is used to provide more detailed feedback on this operation. On the other hand, the device may serve as a security and identification device for acknowledging an operation performed by another device. It can also simply form an association to function as a peripheral of the device.

An interesting aspect of this association is that it can be initiated and authenticated using the capture device of the device. For example, a user who wants to securely identify himself or herself to a public computer terminal may use the scanning device of the apparatus to scan a code or symbol displayed on a specific area of the screen of the terminal to validate key transmission. A similar process can be performed using the speech signal extracted by the speech recording device.

15.7. Integration with other devices

In some embodiments, the functionality of the capture device is integrated with some other device already in use. The integrated device can share power supply, data capture and storage capacity, and network interface. Such integration may be performed simply for convenience, to reduce costs, or to enable functionality that would not be available without such integration.

Some examples of devices in which the capture function may be integrated are as follows.

● Traditional peripherals such as a mouse, stylus, USB "webcam" camera, Bluetooth ^TM headset or remote control

Other processing / storage devices such as PDAs, MP3 players, voice recorders, digital cameras or mobile phones

● Watches, jewelry, pens, car key ornaments, etc.

15.7.1. Mobile phone integration

As an example of the benefits of integration, we look at the use of a modified mobile phone such as a capture device.

In some embodiments, the phone hardware may be processed by the phone itself, or processed by the system at the other end of the phone call or the memory of the phone for future processing, if text capture can be properly performed through speech recognition. It is not modified to support the system as it can be stored within. Many modern phones have the ability to download software to implement part of the system. However, such speech capture may be the next best option, for example in many situations with significant background noise, and even when accurate speech recognition is optimal. Audio devices can be optimally used to capture voice annotations.

In some embodiments, cameras embedded in many mobile phones are used to capture images of text. The phone display, which normally serves as the camera's viewfinder, can superimpose information on the quality of the image and suitability for the OCR in which the segment of text is captured and a copy of the text onto the live camera image if the OCR can be carried out.

In some embodiments, the phone is modified to provide this functionality with a separate Bluetooth connection peripheral or clip-on adapter that adds or communicates with a dedicated capture device. Whatever the nature of the capture mechanism, integration with modern mobile phones offers many other advantages. The phone has a wider world connection, which means that queries can be provided to remote search engines or other parts of the system and documents can be retrieved for immediate storage or viewing. Telephones generally have sufficient processing power to allow many of the system's functions to be performed locally and sufficient storage to capture a reasonable amount of data. The amount of storage can also often be extended by the user. The telephone has a reasonably good display and voice device to provide a vibration function for user feedback and tactile feedback. It also has a good power supply.

Most importantly, the phone is a device that most users already carry.

Section III-Application Examples

This chapter describes the use of the system and a list of possible applications. This list is purely illustrative and not exclusive.

16. Personal Application

16.1. Life library

The Life Library (see Section 6.1.1) is a digital archive of any important document that the subscriber wishes to store and is a set of embodiments of the services of the system. Important books, magazine articles and newspaper clippings can all be stored digitally in the Life Library. In addition, it is possible to store the subscriber's comments, comments, and notes with the document. The Life Library can be accessed via the Internet and the World Wide Web.

The system creates and manages a life library document archive for subscribers. The subscriber indicates what documents he or she wishes to store in his or her life library by scanning information from the document or by instructing the system to add a particular document to the subscriber's life library. The scanned information is typically text from a document, but can also be a barcode or other code that identifies the document. The system accepts this code and uses it to identify the source document. After the document is identified, the system can store a copy of the document in the user's life library or a link to a source from which the document can be obtained.

One embodiment of the life library system may verify that a subscriber is authorized to obtain an electronic copy. For example, if a reader scans text or identifiers from a copy of an article in the New York Times (NYT) and the article is added to the reader's life library, the life library system will check with the NYT to see if the reader is subscribed to the online version of the NYT. If so, the reader gets a copy of the article stored in his or her life library account, and otherwise the information identifying the document and how to order it is stored in his or her life library account.

In some embodiments, the system maintains a subscriber profile for each subscriber that includes access rights information. Document access information can be compiled in several ways, two of which are: 1) the subscriber providing document access information to the life library system, along with his / her account name and password, and 2) the life library service provider's information. A query is made to a publisher with a response and the publisher responds by providing access to an electronic copy when the life library subscriber is authorized to access the article. If the Life Library subscriber is not authorized to have an electronic copy of the document, the publisher provides a price to the Life Library service provider and then gives the customer the option to purchase the electronic document. If so, the Life Library Service Provider will pay the publisher directly and later charge the Life Library Customer or the Life Library Service Provider will immediately charge the customer's credit card for purchases. The life library service provider gets a small fixed cost or a few percent of the purchase price to facilitate the transaction.

The system may store documents in a subscriber's individual library and / or any other library in which the subscriber has recordkeeping authority. For example, when a user scans text in a printed document, the life library system can identify the rendered document and its electronic copy. Once the original document is identified, the life library system can record information about the original document in the user's individual library and in a group library in which the subscriber has recordkeeping rights. Group libraries are joint archives such as document repositories for groups working together on projects, groups of academic researchers, group weblogs, and so on.

Life libraries can be organized in chronological order, by topic, by subscriber level, by publication type (newspaper, book, magazine, technical paper, etc.), by reading place or time, or by ISBN or decimal classification. In one alternative, the system may know the taxonomy based on how other subscribers have classified the document. The system can suggest a taxonomy to a user or automatically categorize documents for the user.

In various embodiments, annotations can be inserted directly into the document or maintained as separate files. For example, when a subscriber scans text in a newspaper article, the article is highlighted and stored in its life library along with the scanned text. Alternatively, keep the article in the Life Library along with its associated annotation file (thus keeping the archived document unmodified). Embodiments of the system may maintain a link to an original document copy in each subscriber's library, a copy in a master library accessible to many subscribers, or a copy maintained by a publisher.

In some embodiments, the life library only stores document revisions (eg, highlights, etc.) and links to online versions of documents (stored elsewhere). The system or subscriber merges the changes and documents when the subscriber subsequently retrieves the document.

If the comments are kept in separate files, the original document and the comments file are provided to the subscriber and the subscriber combines them to create a modified document. Alternatively, the system combines before providing the two files to the subscriber. As another alternative, the annotation file is an overlay to the document file and can be overlaid on the document by software in the subscriber computer.

Subscribers to Life Library Services pay a monthly fee to ensure that the system maintains their archives. Alternatively, the subscriber pays a small amount (eg, a small payment) for each document stored in the archive. Alternatively, the subscriber pays to access the subscriber's archive at a fee per access. Alternatively, the subscriber can compile the library so that others can access articles / comments on the revenue sharing model, such as life library service providers and copyright holders. Alternatively, Life Library Service Providers receive payment from publishers when Life Library subscribers order documents (revenue sharing model with publishers, where Life Library Service Providers get some share of publisher revenue) .

In some embodiments, the life library service provider acts as an intermediary between the subscriber and the copyright holder (or copyright clearing center, abbreviated CCC) to facilitate billing and payment of copyrighted articles. The Life Library Service Provider provides this intermediary service by using the subscriber's billing information and other user's account information. Essentially, the life library service provider leverages the existing relationship with the subscriber to enable the purchase of copyrighted articles on behalf of the subscriber.

In some embodiments, the life library system can store excerpts from documents. For example, when a subscriber scans text from a paper document, the area around the scanned text is extracted and placed in the life library, rather than the entire document stored in the life library. This is particularly useful if the document is long because it preserves the state of the original scan, preventing the subscriber from rereading the document to find the portion of interest. Of course, a hyperlink to the full electronic copy of the paper document can be included with the excerpt.

In some embodiments, the system provides information about documents in the Life Library, such as the author, publication title, publication date, publisher, copyright holder (or the copyright holder's licensing agent), ISBN, link to the document's public annotation, leadrank, and so on. It also saves. Some of this additional information about the document is in the form of paper document metadata. A third party can create public annotations for access by anyone other than themselves, such as the general public. Links to comments by third parties on the document are useful because they enhance the subscriber's understanding of the document by reading other users' comment files.

In some embodiments, the system stores articles by class. This feature allows Life Library subscribers to quickly store electronic copies of the entire class of paper documents without access to each paper document. For example, when a subscriber scans some text from a copy of a National Geographic magazine, the system offers the subscriber the option to archive all back issues of National Geographic. If the subscriber decides to archive all back issues, the Life Library Service Provider checks with the National Geographic Society to see if the subscriber is authorized to do so. If not approved, Life Library Service Provider may broker the purchase of the right to keep National Geographic magazine collections.

16.2. Life saver

A modification or improvement of the life library concept is the "life saver", which uses text captured by the user to reason more about other activities. By scanning menus at specific restaurants, programs at certain theater performances, timetables at specific train stations, or articles from local newspapers, the system enables inferences about the user's location and social activities, for example, as a website. Diary can be configured. The user will be able to edit and edit this diary, add additional articles such as photos and view the scanned item again.

17. Academic Applications

The portable scanners supported by these systems have many compelling uses in academic settings. These scanners can enhance student / teacher interaction and enhance the learning experience. Among these uses, students can annotate learning articles to meet their unique needs, teachers can monitor academic performance and automatically identify source articles cited in their work.

17.1. Children's books

Children's interaction with paper documents, such as books, is monitored by a literacy learning system employing a particular set of embodiments of the present system. Children use a handheld scanner that communicates with other elements of the literacy acquisition system. In addition to a handheld scanner, the literacy acquisition system includes a computer with a display and a speaker and a database accessible by the computer. The scanner is connected to a computer (by wire, short-range RF, etc.). If your child sees a word you don't know in a book, scan it with a scanner. In one embodiment, the literacy acquisition system compares the scanned text with resources in the database to identify words. The database may include dictionaries, thesaurus, and / or multimedia files (eg, sounds, graphics, etc.). After the words have been identified, the system uses computer speakers to pronounce the words and their definitions to the children. In another embodiment, these words and definitions are displayed on a computer monitor by a literacy learning system. The multimedia file for the scanned word may be played through the computer monitor and the speaker. For example, if a child reading "Blonde Beauty and Three Bears" scans the word "bear", the system may pronounce the word "bear" and play a short video of the bear on a computer monitor. In this way, children learn to pronounce the written word and visually learn what the word means through a multimedia introduction.

A literacy acquisition system provides instant audio and / or visual information to enhance the learning process. Children use this supplemental information to quickly gain a deeper understanding of the written document. The system can be used to teach beginner readers to read and to help children gain broader vocabulary. This system provides words that are not familiar to children or information that children want more information about.

17.2. Literacy

In some embodiments, the system compiles a personal dictionary. When a reader sees a new, funny, especially useful or annoying word, save it as a computer file (with definitions). These computer files become reader's personalized dictionaries. Such dictionaries are generally smaller in size than ordinary dictionaries and thus can be downloaded to a mobile terminal or associated device and thus become available even if the system cannot be accessed immediately. In some embodiments, the personal dictionary entry includes a voice file to help pronunciation of the appropriate word and information identifying the paper document on which the word was scanned.

In some embodiments, the system generates customized spelling vocabulary tests for students. For example, when a student reads an assignment, the student can scan a strange word with a portable scanner. The system stores a list of all words scanned by the student. The system then presents the customized spelling / lexical test to the student on the associated monitor (or prints this test on the associated printer).

17.3. Music lessons

The placement of sheet music is similar to placing text on a line of text. The system can capture sheet music using the same scanning device as described above to capture text, and a similar process of constructing a search in a database of known music allows a piece of music to be captured to be identified and subsequently retrieved, played back, or Can be the basis for some other activities.

17.4. Plagiarism detection

Teachers can use this system to scan text from students' work and provide scanned text to the system to detect plagiarism or identify originals. For example, a teacher who wants to confirm that an excerpt in a student's work is from a source cited by the student can scan a portion of the excerpt and compare the title of the document cited by the student with the title of the document identified by the system. . Similarly, the system may use a scan of text from an assignment submitted to the student's original work to show if the text has been copied.

17.5 improved textbox

In some embodiments, capturing text from an academic text can provide a more detailed description of the student or staff, additional practice, discussion of the student and staff about the material, relevant exemplary past exam questions, additional papers on the subject, and lectures on the topic. To records, etc. (see section 7.1).

17.6 Language Learning

In some embodiments, the system is used to teach foreign languages. For example, scanning a Spanish word can cause the word to be read aloud in Spanish with an English definition of the word.

The system provides instant auditory and / or visual information to improve the new language acquisition process. The reader uses this supplemental information to quickly gain a deeper understanding of the data. The system can be used to teach a beginner student who wants to read a foreign language and to help students acquire more vocabulary, etc. The system provides the reader with information about foreign language words that are unfamiliar or that the reader wants more information about. do.

Readers' interactions with paper documents, such as newspapers or books, are monitored by the language skill system. The reader has a portable scanner that communicates with the language skill system. In some embodiments, the language skill system includes a computer with a display and speakers, and a database accessible by the computer. The scanner communicates with a computer (connected by wiring, short range RF, etc.). When a reader sees an unknown word in an article, the reader scans it with a scanner. The database may include a foreign language dictionary, a dictionary of synonyms, and / or multimedia files (voice, graphics, etc.). In one embodiment, the system compares the scanned text with resources in its database to identify the scanned words. After the words are identified, the system uses computer speakers to convey the words and their definitions to the reader. In some embodiments, both words and their definitions are displayed on a computer monitor. Multimedia files on grammar tips associated with the scanned words can also be played through computer monitors and speakers. For example, when the word "to speak" is scanned, the system pronounces the word "hablar", plays a short audio clip indicating the proper Spanish pronunciation, and displays a complete list of the various verb variations of "hablar". do. In this way, the student learns to pronounce words written in letters, visually learns how to spell words by multimedia presentation, and learns how to change verbs. The system also describes grammar tips on the proper use of "hablar" with common phrases.

In some embodiments, a user scans a word or short phrase from a rendered document in a language other than the user's native language (or language that the user reasonably understands). In some embodiments, the system maintains a priority list of the user's "preferred" languages. The system identifies the electronic copy of the rendered document and determines the scan location within the document. The system also identifies a second electronic copy of the translated document in any of the user's preferred languages and determines a location in the translated document that corresponds to a scan location in the original document. When the corresponding location is not known exactly, the system identifies a small area (eg paragraph) containing the corresponding location of the scanned location. The corresponding translated location is then presented to the user. This provides the user with an accurate translation of a particular phrase at the scanned location, often including a slang or other idiom that is difficult to translate accurately on a word-per-word basis.

17.7 Collection of Research Data

Users who study a particular topic can find all sorts of printed and on-screen materials that they want to record as relevant to the topic in a series of personal archives. The system can be automated as a result of scanning short phrases in any one material, and can also generate references suitable for insertion into a publication on a subject.

18. Commercial application

Obviously, commercial activities can be created from almost all the processes discussed in this article, but here we focus on a few clear income streams.

18.1 Price-Based Search and Indexing

Conventional Internet search engines generally provide free retrieval of electronic documents and also do not charge the content provider for including the content provider's content in the index. In some embodiments, the system bills the content provider in connection with billing the user and / or paying the search engine and / or operating and using the system.

In some embodiments, a subscriber to a system service pays a fee for a search originating in the scanning of a paper document. For example, a stockbroker could read an article in the Wall Street Journal about a new product offered by Company X. By scanning the company X's name from the paper document and agreeing to pay the necessary fees, securities brokers can use the system to search a special or proprietary database to obtain premium information about the company, such as analyst reports. I use it. The system also arranges the documents that are read most in paper form to prioritize indexing, for example by ensuring that all newspapers published on a particular day are indexed and available until the article is published.

The content provider pays a fee associated with a particular term in the search query presented from the paper document. For example, in one embodiment, the system selects the most preferred content provider based on an additional context about the provider, in which case the context is paid by the content provider to raise the list of results. In essence, the search provider adjusts the paper document search results based on a prior financial agreement with the content provider. See details of keywords and key phrases in section 5.2.

If access to certain content is limited to a specific group of people (such as clients or employees), such content may be protected by a firewall, and as a result, generally not indexable by third parties. The content provider may nevertheless want to supply an index for protected content. In such case, the content provider may pay a fee to the service provider to provide indexes of the content provider to system subscribers. For example, a law firm can index all of a client's documents. The documents are stored behind the law firm's firewall. However, law firms access their documents through their portable scanners for their employees and clients to provide indexes (or pointers to the indexes) to service providers, and law firms' employees or clients use their portable scanners to deliver them. When presenting an e-scanned search term, the service provider wants to search the law firm's index in turn. The law firm may provide a list of employees and / or clients to the service provider's system to enable the function, or verify the access rights by contacting the law firm before the system retrieves the law firm's index. Can be. In the above example, note that the index provided by the law firm is limited to that client's document, not the index of all the law firm's documents. Thus, a service provider can authorize a law firm's client access only to documents that the law firm has indexed for that client.

There are at least two separate revenue streams that may result from a search originating in the paper document: one is the revenue stream from the search function, and the other is the import flow from the content delivery function. The revenue of the search function is generated from the paid subscription of the scanner user, but may also be generated according to the fee per search. Content delivery revenue may be shared with the content provider or copyright owner (the service provider may take a fixed fee, such as a certain percentage or minimum payment for each delivery), but also whether the service provider brokers the transaction. Regardless of whether or not it may be generated by a “consignment” model in which the subscriber places an order from an online catalog and the system takes a fee or a percentage for every item that is delivered or provided to the system. In some embodiments, the system service provider receives revenue for all purchases made by the subscriber from the content provider for a given time or any subsequent time a purchase of the identified product has been made.

18.2 Catalog

A consumer can use a portable scanner to make a purchase from a paper catalog. The subscriber scans from the catalog information that identifies the catalog. This information is text from a catalog, a barcode, or another identifier of the catalog. The subscriber scans the information identifying the product she / he wants to purchase. The catalog mailing label may include a consumer identification number that identifies the consumer to the catalog vendor. If so, the subscriber can also scan this consumer identification number. The system acts as an intermediary between the subscriber and the vendor to assist in catalog purchase by providing the consumer's choice and consumer identification number to the vendor.

18.3 Coupon

The consumer scans the paper coupon and stores an electronic copy of the coupon on a remote device such as a scanner or a computer for later recovery and use. The advantage of electronic storage is that the consumer is free from the hassle of carrying paper coupons. An additional advantage is that the electronic coupon can be taken from any location. In some embodiments, the system may track the expiration date of the coupon, warn the consumer about a coupon that is about to expire, and / or delete the expired coupon from storage. The advantage to the coupon issuer is that it is likely to receive more feedback about who is using the coupon and when and where it is captured and used.

19. Typical Applications

19.1 Form

The system can be used to auto-populate an electronic document corresponding to a paper form. The user scans a series of text or bar codes that uniquely identify the paper form. The scanner communicates the identity of the information with the form identifying the user to an adjacent computer. Adjacent computers have an Internet connection. The adjacent computer can access a first database of forms and a second database having information about the user of the scanner (such as a service provider's subscriber information database). The adjacent computer accesses the electronic version of the paper form from the first database and automatically fills in the fields of the form from the user information obtained from the second database. The adjacent computer then e-mails the completed form to the intended recipient. Alternatively, the computer can print the completed form on an adjacent printer.

In some embodiments, rather than accessing an external database, the system includes a portable scanner containing user information, such as in an identity module, SIM, or security card. The scanner provides information identifying the form to an adjacent PC. The adjacent PC accesses the electronic form and asks the scanner for the information needed to fill the form.

19.2 Business Cards

The system can be used to automatically populate an electronic address book or other contact list from a paper document. For example, when receiving a business card of a newly met person, the user can capture an image of the card with his / her mobile phone. The system locates an electronic copy of the business card, which can be used to update the onboard address book of the mobile phone with the contact information of the newly met person. The electronic copy may contain more information about the newly met person, which may be compressed into a business card. The onboard address book can also store a link to the electronic copy so that any changes made to the electronic copy are automatically updated in the phone's address book. In this example, the business card optionally includes a symbol or text indicating that there is an electronic copy. Without an electronic copy, cell phones can use knowledge of OCR and standard business card formats to fill in entries in the address book for new contacts. The symbol may also help the process of extracting information directly from the image. For example, a telephone icon next to a telephone number on a business card may be recognized to determine the location of the telephone number.

19.3 Proofreading / Editing

The system can improve the calibration and editing process. One way the system can improve the editing process is by linking the editor's interaction with the paper document to its electronic copy. When an editor reads a paper document and scans various parts of the document, the system will make appropriate comments and edits to the electronic copy of the paper document. For example, when an editor scans a piece of text and makes a gesture control of a "new paragraph" with the scanner, the computer communicating with the scanner inserts a break of "new paragraph" at the position of the scanned text in the electronic copy of the document. something to do.

19.4 Voice Annotation

A user can annotate a document by scanning a text portion from the document and making a voice recording associated with the scanned text. In some embodiments, the scanner has a microphone for recording the user's verbal annotations. After the oral comment is recorded, the system identifies the document from which the text was scanned, locates the scanned text within the document, and attaches a voice comment at that point. In some embodiments, the system converts speech to text and annotates as comments in the text.

In some embodiments, the system keeps the annotation separate from the document only as a reference to the annotation included in the document. The annotation then becomes the annotation markup layer for the document for the specified subscriber or user group.

In some embodiments, for each capture and associated annotation, the system identifies the document, opens it using a software package, scrolls to the scan location, and plays back the voice annotation. The user then interacts with the document, referring to voice annotations, changes made by themselves or others, or other recorded comments.

19.5 Textual Help

The system described above can be used to enhance the paper document with an electronic help menu. In some embodiments, the markup layer associated with the paper document includes help menu information for the document. For example, when a user scans text from a particular portion of a document, the system checks the markup associated with the document and provides a help menu to the user. The help menu is displayed on the display of the scanner or on an associated adjacent display.

19.6 Using the Display

In some situations, it is advantageous to be able to scan information from a television, computer monitor, or other similar display. In some embodiments, a portable scanner is used to scan information from computer monitors and televisions. In some embodiments, the portable optical scanner has an illumination sensor optimized for working with typical cathode ray tube (CRT) display technologies such as rasterizing, screen blanking, and the like.

Voice capture devices that operate by capturing audio of a user reading text from a document generally operate whether the document is on paper, on a display, or on some other medium.

19.6.1. Public Kiosk and Dynamic Session IDs

One use of direct scanning of the display is associated with a device as described in section 15.6. For example, in some embodiments, a public kiosk displays a dynamic session ID on its monitor. The kiosk is connected to a communication network such as the Internet or a corporate intranet. The session ID changes periodically, but at all times when the kiosk is used to ensure that at least the new session ID is displayed to all users. To use the kiosk, the subscriber scans the session ID displayed on the kiosk; By scanning the session ID, the user expresses to the system that he wishes to temporarily connect the kiosk to his scanner for the scanning of the printed document or the delivery of content from the kiosk screen itself. The scanner may communicate the session ID and other information (serial number, account number, or other identifying information) that authenticates the scanner directly to the system. For example, the scanner sends a session initiation message through the user's mobile phone (paired with the user's scanner via Bluetooth ™), directly with the system (where "direct" means not passing the message through the kiosk). Can communicate. Alternatively, the scanner may use the kiosk's communication link by establishing a wireless link with the kiosk and sending session initialization information to the kiosk (perhaps via a short range of RF such as Bluetooth ™, etc.); In response, the kiosk sends session initialization information to the system via its internet connection.

The system can prevent others from using a device already connected to the scanner during the period (or session) with which the device is connected with the scanner. This feature is useful to prevent others from using a public kiosk before another person's session is over. As an example of this concept related to using a computer in an internet cafe, a user scans a barcode on the monitor of a PC she / he wants to use; In response, the system sends the session ID to the monitor on which it is displayed; The user initiates the session by scanning the session ID from the monitor (or entering it through a keypad or touch screen or microphone on a portable scanner); The system associates the session ID with its serial number (or other identifier that uniquely identifies the user scanner) in its database so that other scanners can scan the session ID and the monitor is unavailable during his / her session. Let's do it. The scanner communicates with the PC connected to the monitor (via a wireless link, such as Bluetooth ™, or via a wired connection, such as a docking station), or via other means, such as a mobile phone, directly (ie, without a PC).

Fourth part - system detail

4 is a diagram illustrating an exemplary environment in which embodiments of the system operate. The system functions in a distributed computing environment 400 that includes a plurality of devices interconnected by a wireless network 401, the Internet 402, or another network (not shown). All these communications and connections are interconnected through appropriate network connections using appropriate network communication protocols. In various embodiments, the server and other devices communicate with each other according to respective APIs, forming further embodiments of the system. In alternative embodiments, the device and server may communicate in accordance with open / standard protocols.

The server and other devices may include an OCR device 411 or other text capture device used to capture text from the rendered document 412; A wireless device 421 and / or user device 422 in which the text capture device uploads various representations of the captured text and other user inputs, and the system can provide various types of feedback to the user; A user account server 431 and a user account database 432 associated therewith, wherein the system manages user account information for the user; A search engine server 441 and associated search database 442 that the system uses to execute a query that includes captured text from the rendered document to identify a location within the electronic document where the captured text occurs; And a document server 451 and a document database 452 associated therewith that the system retrieves a copy of the document determined to be included in the captured text. Although such a server has been described as a single device, each server may in fact contain more than one device in an actual system embodying an embodiment of the system. It will also be understood that the server includes a file server, a database server, or a mixture comprising a file server and a database server. Although various servers may be described as separate devices, those skilled in the art will also appreciate that in other embodiments of the system, the servers may reside on a single device.

If the scanner includes a magnetic sensor, the data can be coded into the document magnetically, as well as optically, acoustically, contactly, and the like.

While the process of converting electronic documents to printed forms has been largely from computing offsets, there has been a need for effective ways to rereference the original digital sources of printed documents. In some embodiments, this is accomplished by scanning the required position in the document to identify the distinct text "sign", which sign provides information that can be used to locate the corresponding position in the original digital source document. . The system is directed to a server having access to a database of electronic documents, preferably including an electronic version of the paper document of interest (although useful results may be obtained even when that is not the case, as described below). Send a digital sign. The server then identifies the corresponding location and associates it with the original scan of the paper document. Achieving this relationship enables a number of useful innovations related to the use of printed documents in various contexts. Various embodiments of the system are described below.

In one aspect, the system uses the assistance or documenting information to transform document recognition into document navigation (e.g., locating and traversing portions of the document and generating information about traversal). Can be seen. Some of the many "hints" that the system uses / discovers include:

How fast the user reads

Which direction the user reads

-How long do users join?

-Daily and weekly actions of users (such as reading Sunday newspapers on Sunday mornings)

-Recent marks made by users and documents

-Type of material / topic that has historically been of interest to the user

Clear user profile

The current user location (may be given by the wireless environment, the environment near the user's PC and / or activity on the user's PC)

The nature of the text

- etc

In many cases, the first mark the user makes in the document will be used to capture the typeface or font. The meaning of this character object can then be determined by the template-matching plus disambiguation approach mentioned elsewhere (offset based). Once the current typeface or font is known, the device can capture and send the actual text (eg ASCII) or use the (offset based) template-matching representation mentioned elsewhere.

Because uppercase letters do not appear often, in some embodiments, the system handles them in a special way. Because the system has a source or reference copy of useful documentation, the system may or may not expect capitalization (oral).

In many cases, there is no guarantee that the special case of the document will be rendered as it appears in the source or reference copy. Nevertheless, the system can often estimate how the rendered copy will handle this mark (capital).

A good example is capital letters that usually start in English sentences. As this is rare, it will generally not be easy to use template matching and disambiguation to translate this initial capital. One alternative solution is to essentially ignore the capital by ignoring each new paragraph, the first character of the character.

In disambiguation, uppercase letters and other frequent marks are automatically handled correctly in that the character that appears only once (not repeated) is given a special default offset (eg, a code of zero).

When a special index is constructed from an offset based representation (or other obscure representation), uncertainty about the leading character can be expected, i.e. the system knows from the source document that the capital will occur at a particular location and that it will not match. It should be noted that there is.

This issue of capital is a good example of how the system is distinguished from conventional OCR systems. Because the system assumes useful source documents (now or in the future) (and in some cases relies on the source documents), various inactivations and problems are easily handled. Since our focus is primarily on document navigation rather than translation, the issue of tripping up a conventional OCR system (such as needing to have special information about the shape of all capital letters in all fonts) causes problems for the system. Don't let that happen.

As an example, a user may want to indicate a sentence that occurs in a rendered document of "Take as an example this sentence." Conventional OCR systems attempt to understand and translate the letter "T" to ensure that the first word was "Take" or "Make" or "Fake" or "Rake." However, the system only finds distinct reference features for navigation purposes. The system can simply drop "T" and search the source document for "ake as an example", which can be expressed in letters, offsets, or other forms. Translation of the initial character is not essential, as long as the remainder of the idiom constitutes a sign that is identified.

Another consideration of this issue and distinction comes from understanding that conventional OCR was used to recognize (ie, translate) text characters. For example, the user of the OCR pen moves the pen on a line of text to capture and translate the text, and the user of the system has a different intent. They move their bars or scanning devices on a line of text to * indicate * or "indicate" such a location in the document, thus enabling many of the features and functions associated with the location in the document.

Moreover, when a user is interested in a particular underlying text, their action is generally to act on that text rather than capturing and translating the text. Thus, the user can underline this text, convert it to italics, cite it, place a bookmark, and so on.

One useful feature of the system is to place bookmarks that each point to a position in the rendered document. In general, these can be used later to find locations in the document. One simple but interesting application is to mark where the user last read in documents that are much related to the traditional meaning of "bookmark". It may be possible for the user to easily access this information that was left off in the book or other document. Such data may appear on a user's PC or PDA or mobile phone. In some embodiments, this device itself indicates the last read position by using a small LCD display or the like within the device itself. In some embodiments, the display is an LED that is binary, eg, on or off. Such LEDs may be on if the user scans text they have already read and off if they scan new text. In this way, the user can "hunt" where the user left off.

5 is a flow diagram illustrating the steps normally executed by a system to implement a bookmark. In step 501, the system receives the text scanned by the user. In step 502, the system preprocesses the text scanned in step 501. In step 503, the system compares this scanned text to document history maintained for the user. In step 504, if this scanned text was previously scanned by the user, the system proceeds to step 505 to indicate a previous scan; otherwise, the system proceeds to locate the scan in the document ( Proceed to 506). In step 507, if the scan is located before the last bookmark, the system proceeds to step 508 to indicate that it has previously read; otherwise, the capability 509 to indicate that it has not read before Proceed to).

In some embodiments, prominent examples of new capabilities provided within the system can be seen in the usage history of library books, school books, and the like. Writing in a book in margin notes, underlaying and highlighting, or in other forms, has always been a reader's wish. However, in the cases mentioned (and even in your own books), there is a huge barrier to making these marks, i.e. they do not interfere with others (sometimes on your own) to enjoy the work in the future. do. The system allows the user to mark and annotate a book or document, while at the same time choosing not to touch the original.

One way to see the user's actions in a particular document is by overlay or transparency. Even if the user is interacting with the physical rendering of the document, their marks are virtual, i.e. they are captured and stored electronically, so no physical marks need to appear in the rendered version. However, in some embodiments, the system employs a text capture device with an integrated highlighter or pen. 6 illustrates a scanning device with an integrated highlighter or pen. It can be seen that the scan device 600 includes both the highlighter 601 and the pen 602 that are selectively retracted.

Thus, such an overlay can be thought of as an abstract, virtual layer. These layers can be integrated or "nested" with the source or reference version of the document. In one example, this occurs when a user views a reference document on their computer screen, where data superimposed from the user's action appears or merges on top of the reference document. Note that the reference document does not need to be modified to be displayed in this way. In some embodiments, overlay information is combined with a reference or source document when the user prints the document. In some embodiments, the system applies such overlay to or combines the overlay with the source document when the source document is delivered electronically to the user. For example, the source document and overlay may be combined into a PDF document and emailed to the user.

In either of these examples, the source document does not need to be changed because the user's overlayed information can be stored as a separate layer. Thus, many users can all mark and interact with a single copy of the document. The user's marks and notes are stored separately, so the original does not need to be changed.

Each user's data is usually compared small to the underlying document. Consider the case of highlighting. All that needs to be saved is the start position and end position of the highlighted text in the document, and the highlight color. One way of storing this data is to store it as a character offset from the start of the document. Another way is to store it as an address such as Document: Page: Line. Alternatively, the system stores the actual x-y coordinates of the user's action in the rendered document.

The document used with the system has a special mark (eg, a barcode, etc.) that the user can scan to indicate which issue or copy of the document is being scanned. This additional identifying information allows the system to determine which document the user has and so how it is rendered.

In some cases, the system may require that the user scan the identification code or mark so that the user can use the system to interact with the document. This may be required before the user uses the device on some part of the document. Alternatively, the system may allow the user to interact with the document, but the system may require the user to perform an identification scan at some future point. Or, alternatively, this identification scan may be optional, and if it is not there, the system may have additional ambiguities, i.e. less certainty about the particular document being used. With other incidental scans, the system learns more about a particular document.

In some embodiments, the capture device may display an error indicator or signal (e.g., LED or audio) to inform the user that such documents are neither recognized nor enabled (i.e., an identification scan is required). Tons). This identification scan can be used to specifically indicate which document the user has, for example, a local morning newspaper, so that the system can cache the document or the associated dictionary to determine the disassembly and location of the scan. Notice the copied copy.

The particular mark to be scanned may be one or two dimensional barcodes, special areas of human readable text, or coded data. In some embodiments, an area of text in a rendered document is used to indicate to the user that such area should be scanned for document recognition (eg, with margin marks, or highlighting, or underlining, or with specially colored ink). By special marking).

All of the above descriptions may also apply to a plurality of marks in a document. For example, different articles in a magazine or newspaper, individual advertisements, individual pages, etc. in a magazine may have a mark, or a user may be required to clearly scan one or more items in a sub-area of a document. Thus, individual parts of a document can be clearly identified by the user's distinct actions.

In some cases, this scan can be used to help the system know the user's context. In other cases, such a scan may enable or unlock features of the system that would not be useful. For example, a purchase from a printed catalog may not be allowed unless the user scans the address label with the user's identification code thereon.

In some embodiments, a user may specifically scan an area of text to achieve their context, the context of which location within which document. For this purpose, the terminal device may have a special switch or input to indicate this required function to set the context. Alternatively, the user may execute a special gesture as a device capable of instructing the context-setting reference scan function, such as by reverse scanning text. Alternatively, the motion or gesture by such a device may indicate "erase" or "undoing" of the previous action.

In general, the motion and action of the device can be used to indicate user intent. The long list of possible motions includes:

-Scanning in the reading direction = Document signature is generated.

Scanning in reverse direction = context setting.

Dragging in the vertical and vertical directions of the page (the system can count the horizontal lines traversed and also capture the data fragments from the digits traversed by the system) = local setting.

Before and after motion, or up-down motion = undoing the previous action.

-Show area circle of text = Select area.

-Tapping or clicking-through a switch or sensor in the end of the device in contact with the rendered document, or through a separate switch that can be controlled by the user = context related menu request.

Note that this is only a partial list. It should also be noted that the rich possibilities obtained by combining two or more of these motions and by changing the order in which these motions are performed.

One use of the system is in document signing. It should be noted that the device may record at what time, location, etc. the particular document was scanned, including which part was scanned. In addition, the optics in the device can capture and store an image of the signature. In a system, a document may be provided with a special mark or code at one or more locations (this includes a unique identifier code for the entire document). Such marks may include specially marked or indicated human readable text (eg, sections of legal documents that are bold or underlined, etc.). The user can then scan various parts of the document to indicate that they have read. In addition, they may selectively scan the documents and may scan and read their signatures by the terminal device. The device itself can incorporate the recording means as shown in FIG. 6, in which case the user can scan and sign with one device.

There has long been a need to code documents and subparts of documents with special codes, for example bar codes. Historically, however, efforts to resolve this have not been completely successful. One reason for this may be that the simple barcode scanner itself was not useful enough for the end user. This causes a chicken-eg situation (a situation in which it is difficult to guess first): the user does not carry the scanner, so the publisher does not print the code. The publisher does not print the code, so the user does not need or use a code-scanning device.

However, the unique combination of document navigation tools and plus code scanners (plus, optionally, OCR-enabled tools) can overcome this barrier. Because of the value and usefulness of text scanning capabilities and / or OCR capabilities, a user needs, carries or uses a terminal device that may include hardware and / or software capable of reading coded (eg, bar code) information. Will be motivated to It should be noted that all additional components for handling barcodes may be located at a server or other location within the system. The terminal device may simply capture an image of the barcode to be read and send it for translation.

In some embodiments, the device recognizes when the image it is scanning is one-dimensional as in a barcode. For example, the device can check whether there is an axis with no information (in software, electronically in hardware, or a combination thereof). One-dimensional barcodes have this property, which consists of parallel lines that are assumed to be parallel to the y-axis. In this case, only the changes along the x-axis across the line, perpendicular to them on the x-axis, contain the information. In the case where the device sees data having such one-dimensional characteristics, the device may have local intelligence (hardware and / or software) to reduce this scanned data by overlooking / ignoring the y-axis. That is, the device can read or translate the code partially or completely (eg, before communicating with the server).

This description of barcodes raises interest in the technical components of most OCRs or systems-"deskewing". Deskewing is the process of removing any artificial angular components from the scanned or imaged data. A situation that often occurs with hand scanners is that the user keeps them at an angle rotated about an axis perpendicular to the page, so that the captured data includes an artificial angle or tilt. It should also be noted that this angle may also change over time, for example, as the user moves their hands across the page. It is beneficial if such artificial skew is eliminated in one of the steps of processing data or images.

When the system employs a template-matching or convolution based method (described elsewhere), one advantage is that first, artificial skew or angle is not important. That is, letters or symbols that are each skewed at the same angle will match each other without removing these skew components.

Many types of fonts have a number of strong vertical elements. These are often straight lines perpendicular to the baseline. In some embodiments, the system deskews the text by performing a mathematical transformation on the data for which the skew angle can be readily determined. This transformation can be applied locally so that variations in skew (eg, across a single line of text) can be detected and measured locally.

The template-matching (offset based), convolution method that can be used in the present system has the ability to discover objects using these objects that were generated early, such as templates. The result of this capability of interest is that any repeating object may have easily readable information, and tokens representing such information need not be predefined or known to the system.

As an example, a document contains a string of 1s and 0s, such as "100101001" (eg, representation of binary data).

In the template matching method, the system does not need to recognize and understand the meaning of "1" or "0". Rather, the system can translate the "string of the object of the first type, two objects of the second type, the object of the first type ..." and the sample string. This information in the sample string can of course be represented by "abbababba" or "011010110".

The data can be encoded using any number of distinct objects or symbols, and the space can be treated as one of these objects (if space is used, the measured distance can be used to count adjacent spaces or each space object Can be limited to always have a count of 1, such that "1 11 111 1" is translated equal to "1 11 111 1" From this point of view, 26 (assuming lower than all) Languages written in the character Roman alphabet are just a special example of this encoding, in which the number of symbols is 26.

In some embodiments, the system represents data in a sequence, such as the example "011010110" above, as an offset, that is, the number of character position surviving reoccurrence examples of each symbol. In this representation, "011010110" is represented as 3,1,2,2,2,3,1,?,?, Where each digit corresponds to a character in the original string, and the value of the digit is the same character. Is the offset or distance for the next occurrence of.

In this representation, certain overlooked elements and / or prohibited codes exist. For example, after the first digit, 3, the digit "2" can never occur in nature because this means that this second character is the same as the first character, in which case "3" is "1". "It would have been back.

Also, the last two entries (marked with "?") Are redundant, and the characters in these positions are known to them by early offset / reference, so these two trailing positions have little or no information. not. That is, if a character has an offset of m (assuming we move to the right and measure the offset to the right), then the next character cannot have offset m-1, and subsequent characters are offset m Cannot have -2, etc. This is because these "forbidden" offsets collide and contradict earlier offsets.

In some embodiments, the system examines the forbidden code for decoding and / or representing data. For example, in some embodiments, the facility examines the forbidden code by using the forbidden code to store additional data, such as an exception code. Thus, whenever the sequence of offsets includes the code "m, m-1", the system may enter a particular mode or routine, and may process the following code specially.

This effect can be cumulative, ie each offset must meet the constraints of all the offsets shown previously. As an example, consider the data string "xyzyyxzzzyx" input with offset "5,2,4,1,5,5,1,1,?,?,?". Each of these entries has the following constraints:

5-may be any, not any previous blow.

Cannot be 2-4 (other previous entries would have been 1, not 5)

Cannot be 4-3 or 1 (because they contradict previous 5 and 2)

It can be 1-2 (to collide with 5) or can be 3 (to collide with 4)

Can't be 5-2 (collises 4)

Cannot be 5-4 or 1 (collides with 4 and 5)

Cannot be 1-4 or 3 (collides with 5 and 5)

Cannot be 1 to 3 or 2 (collides with 5 and 5)

? Cannot be 2 or 1 (collides with 5 and 5)

? -Cannot be 1 (collides with 5)

? -Anything beyond this position may be arbitrary because no reference is made.

In other words, any offset that extends beyond one character position puts a logical constraint on all interfering positions.

The purpose of this observation is to perform error detection. For example, if a forbidden code is received, the system can translate it to an error and report it or act on it.

Another use is to encode additional data. In some embodiments, the system translates the forbidden code as an "escape sequence" that invokes a special action or process or reads embedded data continuously from the stream, and then the system (the escape sequence is Information may be retained, or it may be known or conveyed by the system) and then resynchronized to the input stream and processing continues.

Another use of this forbidden code is to reduce the amount of information in encoding the data. Here, one algorithm says: "If this is a first forbidden (minimum) forbidden code, treat it as a first allowed / valid code; if this is a second (second minimum) forbidden code, then it is the second allowed / Process as valid code. " In this way, a smaller (forbidden) number may result in a larger (allowed) code, so that the number of bits used to store or transmit data is reduced.

The use of such a plurality of illustrated forbidden codes can be combined, for example, the first (minimum) forbidden code may be translated as an escape sequence, and the higher code is then mapped to a useful valid code. Other uses of this code can also be used.

In general, almost no additional information is carried by repeated sequences. Thus, for example, the character sequence "abcabcabcabcabc" is repeated more concisely as "5 (abc)". The same is true for the offset representation of the repeating sequence. "abcabcabcabcabc" is expressed as an offset as "333333333333 ???", which may be expressed as "12 (3) ???".

In another example, "abcbcbcabcbcbcabcbcbc" is first coded as an offset, "72222337222233? 2222?", Then "74 (2) 2 (3) 74 (2) 2 (3)? 4 (2) 2 (?)". And then further reduced to " 2 (74 (2) 2 (3))-4 (2) 2 (?) &Quot;). (The convention shown here is "count", where the parentheses do not restrict the object, and in a data system this may be expressed in many ways.)

In some embodiments, the system is designed such that together with the iteration count, only the object itself is stored or transmitted. Applied to the repeated column of the offset of the previous example, "abcabcabcabcabc" is first offset, i.e., count (object): 12 (3) ??? -It can simply be stored or transmitted as "+3 ???", where "+" is an indicator that the object repeats-and can be coded as "333333333333 ???" which can be represented in turn as. Alternatively, the system stores or sends only "3" and drops any reference to the iteration count.

The issue of this repeating sequence is important because the system may not know how far the user will scan. As an example, a user may scan a dashed line "----------------". If the system assumes that these dashed lines will go to arbitrary lengths, the user may not want to scan as they wish, so they do not know how many dashed lines appear. In this case, the system simply stores or transmits, in some embodiments, a “length 1 repeating sequence”.

This also works for more complex sequences. The section of the story the user is reading may include boundary markers.

"-***-***-***-***-***-***-***-***-***"

In some embodiments, the system does not require the whole to be scanned for this to be recognized. The offset from this sequence may be represented as "31641153164115 ..." (counting spaces as objects). This may be stored or transmitted as a count plus object, or as "plural indicators" ("+3164115"-see the use of the plus sign above), or without a plurality of any indications (only "3164115").

These two latter examples are regular expressions that are close to the matching structure. The example "+3164115" corresponds to "match greater than one" and the example "3164115" corresponds to "one or more matches". In this case, when searching for a sequence in an index or database, the agreed convention is that any sequence matches itself by one or more consecutive occurrences.

Some of the utilities of this structure occur when there is a source or reference copy of a document in the system, and the system will attempt to position the user's position. If both terminal and / or local system components and back-end components (eg, arches, indexes, etc., perhaps server-based) understand and agree on how repeated sequences are handled, redundant data is dropped for storage and communication. Can be set. The server indexing the data in the "one or more" example of the previous paragraph can store only a single first example of the repeating sequence (either raw data or derived offset), and the scanning terminal device provides one example of the repeating sequence. You can also save or send it.

Another way to describe this process is that any iterative sequence is represented by counts, or in a simpler alternative model, in such a way that the whole is completely ignored. Thus, the phrase "*** buy cheap cheap tools here !!! ***" may be indexed or expressed as "* buy cheap cheap tools here! *", Or the offset is "*** buy cheap cheap tools here!" !!! *** may be similarly compressed.

"11 * 4 ??? 6666666? ** ?? 6? 1 ??? 8? 2 ?? 11 ?? 11?" (Here, an offset greater than 9 is shown by " * ") is compressed as follows.

"2 (1) * 43 (?) 7 (6)? ** 2 (?) 6? 13 (?) 8? 22 (?) 2 (1) 2 (?) 2 (1)?". Or remove all the iterations (but leave a "*" for offsets greater than 9)

"1 * 4? 6? **? 6? 1? 8? 2? 1? 1?"

For example, a separate system operating at a remote server may then search for a sequence that matches this compressed iteration, or reference an index to find this sequence. To do so, a separate system processes each object in the sequence as it happens "more than once" and finds this match by implementing code and algorithms as regular representations.

Similar efficiencies in storage and / or transmission can be obtained by noting that when character offsets are employed, all scans of text end with an unknown offset (shown as "???" above). This assumes that the user is scanning from left to right, and the offset is for the next matching character to the right at the point where the scan should end, so that some of the last character does not know the offset, where The next occurrence of the character will not be included in the scan. In one data encoding technique, this unknown offset may be expressed as zero, but another embodiment is to drop this trailing unknown offset from the transmitted or stored data.

Template matching and / or autocorrelation will recognize a series of occurrences of this same object using an example of a token or object or character or symbol as a template. A simplified overview is provided here.

Suppose a user is scanning a single horizontal line of text with a terminal device.

When a user scans a single horizontal line of text with a capture device, in some embodiments, the system captures an image of the text and stores it in memory and / or transmits it. In some embodiments, the system executes template matching on-the-fly by immediately calculating the offsets of the matching objects and storing only individual templates, which can be discarded when the offsets are known.

Initially, the system does not need to know much (if any) about the type of object it is scanning. In the process of template matching, this form emerges when various templates are found.

This also applies to the horizontal extents of the character. Special knowledge of whitespace and character width can be helpful (for example, the ratio of width to height of most characters is for x, or that the average word length is for y, etc.) no. Indeed, in some embodiments, the capture device ignores whitespace together.

In some embodiments, when a user scans (or after) a portion of a horizontal line of text, the system effectively "convolves" that line, ie, slides a copy of the line horizontally past itself, Find an area that matches well. At the start of this process, the step of determining the baseline of the text and the step of deskewing it can be helpful, all of which are well known techniques in the field of document imaging. However, it should be noted that there is a way to search the matching area without one of these steps.

As areas that match or nearly match, their horizontal size is noted. In this example, we assume that the vertical line of the match is the overall height of the character (we will consider some matching technique later, which we call "self-aware").

This process can optionally be chosen to use linked region analysis, assuming that tokens / objects / characters / symbols are composed of "connected" (ie, continuous) pixels or inks. Thus, an example of searching for a matching connected area is given. It should be noted that this is related to the knowledge of whitespace because, by definition, the area outside the connected area of ink is whitespace.

In some embodiments, the system uses a simple horizontal size (as one of plural different methods). That is, the system may introduce components of connected regions or white spaces to assist the process, but little or no attention is paid to the connected regions or white spaces, and to the position and horizontal width of the matching regions of pixels or ink. do.

7 is a flow diagram illustrating the steps normally executed by a system to process a text capture action. In step 701, the system receives text that is captured by the user. In step 702, the system preprocesses the text scanned in step 701. In step 703, the system identifies words and line boundaries in the scanned text. In step 706, the system convolves the text, as described above. In step 705, the system uses the delimiter to determine the boundary of the unknown area in the text. In step 706, the system processes the scan to generate a representation of the captured text. In step 707, the system retrieves a matching textual representation within the body of the electronic document. In step 708, if the search of step 707 is successful, the system proceeds to step 709 to return a successful search notification, otherwise the system proceeds to step 710. In step 710, if the search can be refined, the system proceeds to step 711, otherwise the system proceeds to step 712 to return a failed search notification. In step 711, the system informs the user that purification is required. After step 711, the system proceeds to step 701 to receive additional text captured by the user in response to the notification of step 711.

In matching two areas of text taken from the rendered document, the system encounters the issue of "goodness of the match". Since all physical measurements contain errors, the matching process of the system will inherently never be accurate. Thus, in some embodiments, the system makes a decision about how well one region matches another region. Multiple tools can be used to accomplish this, many of which are already well known in the field of OCR, document imaging, and machine vision. In some embodiments, one way of estimating the fit or match used by the system is to first find the best alignment of the objects being compared and then calculate the difference on that object. For example, in the case of a simple black and white pixel (no gray scale), the system simply finds a pixel that is on in one image / object and off in another image / object. This count of "errors" is a rough estimate of the pit.

This count estimate may be improved by "normalizing the count estimate" by the total number of pixels included (by dividing the count estimate by the total number of pixels included). Thus, in various embodiments, the system uses one of the following.

fit_error = #_bad_pixels / #_pixels_in_x_y_region_compared

or:

fir_error = #_bad_pixels / #_pixels_in_object

The former takes into account the number of error pixels compared to the area being compared. The latter takes into account the number of error pixels or matching, non-error pixels compared to this number. In various embodiments, the system adds various purifications to this technique, or uses other matching techniques from OCR and the like.

In the described matching process, the error and the physical distribution of the matching pixels (eg the x-y axis) are of potential importance. In the derived and estimated language, there may be better fit when the matching pixels are " focused " (ie, continuous, occurring close together). Error pixels are " distributed " (ie, discontinuously occur far away).

As an example of this, consider two situations. In one case, the system compares the image of "r" with the image of "n". Only a small portion of the image contains error pixels, where the right side of "n" is inclined to the baseline, while "r" does not. Note that there may not be a large number of error pixels, but they occur close together and in succession. Next, consider comparing two images of the letter "n". Here, the quality of this image is not good, for example if it is blurred or damaged. In such a case, there may be very many error pixels, but it should be noted that there is a high probability that it will be distributed in many places around this character. This demonstrates that widely distributed error pixels have fewer errors than closely packed error pixels.

This description of matching is linked to other related innovations called "self matching". Historically, OCR has employed clear knowledge of absolute character shapes and fonts to recognize various characters. In some cases, the system uses indirect knowledge of * related * character shapes to recognize the character.

8A-8D illustrate how often two characters of an alphabet approximate the same * related * shapes in multiple fonts. FIG. 8A shows the letters "D", "C", and "L" in Arial lower-case font, and FIG. 8C shows these letters in Times New Roman lower-case font. While these fonts are very different and distinct, the relationship between these characters in a particular font "in" is almost the same.

8B and 8D show that in either font, "d" can be configured with some degree of accuracy by adding "c" and "l" together. Or, in pseudo-algebraic terms, "d = c + l". There are many other similar relationships that are true for many fonts such as "e = c +-", "P = B-b + l", and "8 = 6 + 9". This relationship is not meant to be correct, but it does mean that the relevant shapes of the various characters in different fonts are * approximately * identical. This relationship is approximate, but this relationship allows one group of characters in a font to be used to recognize (or actually construct) additional characters.

One application of this technique is in OCR. Once the system determines some of the characters in the font (we may have no knowledge of one of them), the system can predict and / or recognize the rest of the unknown characters.

Sometimes it is possible to identify some characters using simple methods of cryptography, such as character frequency and n-gram analysis. In conjunction with other characters, observation of how often the character appears at any position in the word can provide initial information about the character, even if the character appears in a font we do not know. The simplest example may be the single characters "a", "l", when we see a single character word, we know immediately that they will be one of these two letters. Similarly, repeated characters (e.g., ee, oo) are likely not to be "hh" or "qq".

Suppose the system has learned the characters "d" and "o" and met the character "c" but do not yet know what it is. By comparing the shape of the learned character with the character "C" shape, the system matches the "d" except for the vertical stem and matches the "o" except for the right edge of the character. Decide if The system has now identified the new character "c" because it knows that in most fonts it is an associated character of "d", "o", and "c". In doing so, the system adds this to the repertoire of known symbols and uses it to decode the additional characters. Thus, the system's knowledge of a font for which the system has no specific knowledge can be gradually increased and expanded based on its * general * knowledge of * related * character shapes. One way to implement this self-recognition system is by the m × m matrix of general relations, where m is the number of alphabetic characters. Each entry in this table describes how character i is related to character j, which is likely to contain general relevant geometry information and rules ("character i extends below the baseline, character j is not)). , It is likely to include a reference to an additional character in the alphabet (eg, the entry for row "d" and column "c" may be "-l", which is "c" to "c"). Subtracting "l" to make a, and the entry for row "c" and column "d" may be "+ l", which means adding "l" to make "d" from "c" ).

A special case of self-recognition occurs when a set of symbols or tokens (ie, a new font) are constructed with the distinct intention that they are self-recognizable as described above. That is, given a subset of these symbols, others can be derived or inferred. These symbols can be used for error checking with each other because their graphical components / designs are correlated. This redundancy also provides sophistication for noise.

In order to employ this method, there is no requirement that an * arbitrary * of symbols be known in advance in the system. By using one of the aforementioned methods of template-matching or correlation, the system can determine the entire set of symbols used based on matching and mismatching symbols. The system then uses known relationships between symbols to identify each of the symbols, in some cases to generate any missed / unused symbols, even though the system has never seen such a set of symbols before. .

9 is a diagram illustrating a method used by facsimile in some embodiments to learn a whole new set of symbols using self recognition. This diagram shows a set of symbols 900 consisting of a 2x2 array of small "boxes". Using knowledge of whitespace and baseline to achieve vertical and horizontal spacing, there are 16 possible symbols, that is, 4 single boxes at 4 corners, 6 permutations including 2 boxes, 3 boxes 4 permutations comprising, and one symbol with all four boxes, and a pipespace symbol consisting of no boxes.

FIG. 10 is a diagram illustrating a subset of the symbols shown in FIG. 9 that self-define vertical and horizontal extents. Each symbol in subset 100 is two boxes wide and two boxes high.

FIG. 11 is a diagram illustrating a relationship between some of the symbols shown in FIG. 9. For example, each of the relationships 1101 and 1102 illustrates a relationship in which one symbol of the set may consist of two other symbols. It should be noted that we can select as a subset these symbols with the best redundancy and / or error correction characteristics. It should be noted that some symbols may be constructed in at least two different ways from combinations of symbols in the set.

*************

Multiple dedicated barcode scanning devices that were targeted in the consumer market have failed commercially, perhaps because barcode scanning alone does not provide sufficient value and functionality for widespread consumer adoption. On the other hand, the combination of document marking / scanning and bar code scanning will create a combined function that will be used by a wide range of consumer interests. Such a combination of text and barcode imaging can be accomplished with many of the same components, hardware and software. However, one real new component of interest is a scanning device that uses a reference or source copy of the rendered document being scanned and the user's actions in the rendered document are interpreted in relation to and mapped to the source or reference document.

Another means for locating the source document is when the marked version has a machine readable code that provides a direction for retrieving such source document and / or identifies the source document (eg, a URL). Such code may be a barcode, a machine readable font, or any machine readable means for carrying this information.

An interesting extension of the machine readable document ID and document locator is that this data includes access information. In other words, the machine-readable code contains information that allows the document to access the system when the system is password protected, for example, behind an integrated firewall or the like. It should be noted that additional input may be requested by the user or other individual confirming the request to access this document.

In some embodiments, the system maintains a relationship between a user's notes and marks and document content and features associated with those notes and marks. This may be important, for example, if the source document is re-rendered in a different style or format, and the system wants to redisplay the user's mark at the correct location. As one example, when a user drags a line through a word of text, the system may want to show the line through the same text in continuous rendering.

One means of accomplishing this is to "anchoring" each user mark or group of marks with some recognizable feature (eg, verbal mark, image, etc.) in the document. In some embodiments, the system anchors by finding recent network source-document features (eg, by geometric distance) and associating the mark with such a mark.

In some embodiments, the system associates a user's mark with a high-rated feature to find nearby features and weight or rank them. As an example, if the system encounters a margin note by the user, the system examines all nearby words and matches the most relevant words (eg, keywords rather than stop-words, which may be related to the logic of the source document text). You can also associate this mark. This aspect of the system may employ some of the many known techniques for identifying important elements in a document.

Thus, notes and marks with associated anchor points can be associated with the digital document so that the relevant notes or marks can be accessed when viewing or editing the document (eg, with a word processor). For example, all of these notes may appear as special symbols embedded in the document (and may be stored as options within this document). The user may then hover or click on this symbol to reveal an embedded or associated note or mark. Likewise, a user's notes may be turned on or off via menu commands.

conclusion

Those skilled in the art will appreciate that the system described above may be employed or extended in various ways. Although the foregoing description refers to specific embodiments, the scope of the present invention is limited only by the claims that follow and the elements described therein.

Claims (25)

A method of processing data captured from a rendered document, The method is performed by a computer, The method comprises: Receiving user input to select a portion of the rendered document, wherein the selected document portion includes text; Determining a convolution-encoded representation of text included in the selected document portion; Retrieving the convolutional-encoded representation of text in a corpus of electronic documents to identify (1) an electronic document contained within a corpus of electronic documents, wherein the identified electronic document is the captured document portion. Comprising: (2) identifying a position within the identified electronic document in which a document portion is selected; Providing, on a display device, a menu of selectable user actions associated with a position identified in the identified electronic document, wherein at least one of the selectable user actions is associated with the identified position prior to receipt of the received user input. Was associated-; Accessing a markup layer associated with the identified electronic document; And Identifying the actions defined by the markup layer for the identified position in the identified electronic document Including, The identified actions are provided in a menu of selectable user inputs. How data is processed. delete The method of claim 1, wherein the identified electronic document and the identified position are identified without determining a standard character set representation of the selected document portion. delete 4. The method of claim 3, wherein determining the convolutional-encoded representation is performed using an image of the selected document portion. The method of claim 1, further comprising encoding each of the two selected document portions into a convolution-encoded representation, wherein the two selected document portions comprise text of two different natural languages. How to process data. 2. The method of claim 1, comprising encoding a data independent of the text contained in the selected document portion into the convolutionally encoded representation by including a forbidden value in the convolutionally encoded representation. Data processing method further comprising. 4. The method of claim 3, wherein the selected document portion is unique among the corpus of electronic documents, the version of the selected document portion from which a mark made by a user is deleted is not unique among the corpus of electronic documents, and the mark is a single word. , Data, punctuation, and images. delete delete delete delete delete delete delete delete delete delete A storage device operable to store content readable by a logic device, the storage device storing instructions that may cause a computing system to perform a method of processing data captured from a rendered document; The method comprises: Electronically capturing a portion of the rendered document, wherein the captured document portion comprises text and electronically capturing a portion of the rendered document comprises using the camera of the mobile phone to render the rendered document. Optically capturing an image of a portion of the document or capturing a spoken version of a text fragment in the rendered document using a microphone of the mobile phone; Determining a convolutional-encoded representation of text included in the captured document portion; Retrieving the convolutional-encoded representation of text in the corpus of electronic documents to identify (1) the electronic document included in the corpus of electronic documents, the identified electronic document comprising a captured document portion; (2) identifying a position in the identified electronic document from which a document portion is selected; Accessing a markup layer for the identified electronic document that associates actions to be performed with respect to particular positions within the identified electronic document; Identifying actions using the accessed markup layer associated with the identified position in the identified electronic document, wherein at least one of the identified actions is selected before the electronic capture of the portion of the rendered document. Associated with the identified position in the document; And Providing, via a display associated with the mobile phone, one or more user selectable options to execute the identified actions associated with the identified position in the identified electronic document. Storage device comprising a. The method of claim 1, wherein receiving a user input to select a portion of the rendered document comprises capturing electronically a portion of the rendered document, wherein capturing the portion of the document electronically. Optically capturing an image of a portion of the rendered document. The method of claim 1, wherein receiving a user input to select a portion of the rendered document comprises capturing electronically a portion of the rendered document, wherein capturing the portion of the document electronically. Capturing text fragments spoken by a user in a portion of the rendered document. 20. The storage device of claim 19, wherein accessing the markup layer comprises accessing a data structure that maps actions to be executed on text in identified positions in the identified rendered document. 20. The storage device of claim 19, wherein the identified actions include standard actions associated with every particular portion of the rendered document. A system for processing data captured from a rendered document, Means for optically capturing a textual base portion of the rendered document; Means for recognizing text within an optically captured text base portion of the rendered document; Means for generating a convolutional-encoded representation of text included in the captured document portion; Retrieving the convolutional-encoded representation of text in a corpus of electronic documents to identify (1) an electronic document included in a corpus of electronic documents, the identified electronic document comprising the text base portion; 2) means for identifying a position in the identified electronic document that is optically captured; Means for accessing a markup layer associated with the identified electronic document; Means for identifying two or more actions defined by a markup layer for an identified position in the identified electronic document, wherein the identified two or more actions are determined prior to optical capturing of the text based portion of the rendered document. Includes an action associated with the identified position in the identified electronic document; And Means for displaying user selectable options associated with the identified two or more actions. Data processing system comprising a. 25. The data processing system of claim 24, wherein the two or more identified actions comprise an action dynamically generated after optical capturing of the textual base portion of the rendered document.

Images