Exhibit 46 - DA's Infringement Contentions - Detailed

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 1 of 55
Exhibit 46
Claim 1
Ex. 46 - 1
Infringement
Siri implements methods for processing a natural language input provided by a user. It takes the user's natural
language input, entered in writing or speech, and processes the input to, for example, send messages, schedule
meetings, and place phone calls.
See, e.g.,
http://www.apple.com/ios/siri/siri-faq/ (Exhibit A)
- What is Siri? Siri is the intelligent personal assistant that helps you get things done just by asking. It allows you to use your
voice to send messages, schedule meetings, place phone calls, and more. But Siri isnt like traditional voice recognition
software that requires you to remember keywords and speak specific commands. Siri understands your natural speech, and
it asks you questions if it needs more information to complete a task.
1. A method for
processing a natural
language input
provided by a user,
the method
comprising:
U.S. Pat. Publ'n No. 2012/0016678 (Exhibit B)

- [0008] - [0010] According to various embodiments of the present invention, an intelligent automated assistant is
implemented on an electronic device, to facilitate user interaction with a device, and to help the user more effectively engage
with local and/or remote services. In various embodiments, the intelligent automated assistant engages with the user in an
integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain
information or perform various actions. . . . In some embodiments, the intelligent automated assistant systems of the present
invention can perform any or all of: actively eliciting input from a user, interpreting user intent, disambiguating among
competing interpretations, requesting and receiving clarifying information as needed, and performing (or initiating) actions
based on the discerned intent.
- [0042] - [0043] FIG. 28 is a flow diagram depicting an example of a method for natural language processing according to
one embodiment. FIG. 29 is a screen shot illustrating natural language processing according to one embodiment.
- [0315] In various embodiments, algorithms or procedures used by assistant 1002 for interpretation of text inputs, including
any embodiment of the natural language processing procedure shown in FIG. 28, can be used to rank and score candidate
text interpretations 124 generated by speech-to-text service 122.
See also, U.S. Pat. Publn No. 2007/0100790 20 (Exhibit M) (As described in further detail with respect to FIG. 3, an
active ontology such as the active ontology 100 may be built to process facts in order to achieve any one of a number of
goals, including activity recognition (e.g., natural language processing, human activity recognition, etc.)); Guzzoni et al.,
Active: A unified platform for building intelligent web interaction assistants, Proceedings of the 2006 IEEE/WIC/ACM
International Conference on Web Intelligence and Intelligent Agent Technology (2006) (Exhibit J) (In our example, we have
used Active to model and deploy a natural language driven assistant able to retrieve information about restaurants and
movies. . . . To create a language processing application using Active Ontologies, the first step consists of using the Active
Editor to graphically specify the application domain, sketching key concepts and relationships (figure 1).); Guzzoni et al.,
Modeling Human-Agent Interaction with Active Ontologies, Association for the Advancement of Artificial Intelligence (2007)
(Exhibit L) (The core of the application consists of three Active Ontologies implementing the overall behavior of the
intelligent assistant: language processing.).
Claim 1
Ex. 46 - 2
Infringement
Siri provides a natural language input query by the user. For example, Siri allows the user to ask Siri a question or
ask it to do something in natural speech, as the user would speak to a person. Siri processes this natural language
query by providing it to a natural language processing procedure.
See, e.g.,
- How do I ask Siri something? To talk to Siri, hold down the Home button and youll hear two quick beeps and see What
can I help you with? on the screen. Just begin speaking.
- Do I have to say things a certain way to get Siri to respond? No. You can speak to Siri as you would to a person in a
natural voice with a conversational tone. If you want to know what the weather will be like tomorrow, simply say What will
the weather be like tomorrow? Or Does it look like rain tomorrow? Or even Will I need an umbrella tomorrow? No matter
how you ask, Siri will tell you the forecast.
providing a natural
language query input
by the user;

- [0009] - [0011] According to various embodiments of the present invention, the intelligent automated assistant integrates a
variety of capabilities provided by different software components (e.g., for supporting natural language recognition and
dialog, multimodal input, personal information management, task flow management, orchestrating distributed services, and
the like). . . . In some embodiments, the intelligent automated assistant systems of the present invention can perform any or
all of: actively eliciting input from a user, interpreting user intent, disambiguating among competing interpretations,
requesting and receiving clarifying information as needed, and performing (or initiating) actions based on the discerned
intent. . . . In addition, in various embodiments, the assistant of the present invention provides a conversational interface that
the user may find more intuitive and less burdensome than conventional graphical user interfaces. The user can engage in a
form of conversational dialog with the assistant using any of a number of available input and output mechanisms, such as for
example speech, graphical user interfaces (buttons and links), text entry, and the like.
- [0411] - [0412] Referring now also to FIG. 28, there is shown a flow diagram depicting an example of a method for natural
language processing according to one embodiment.. . . .The method begins 200. Language input 202 is received, such as
the string "who is playing this weekend at the filmore" in the example of FIG. 29.
See also, Guzzoni et al., Active: A unified platform for building intelligent web interaction assistants, Proceedings of the 2006
IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (2006) (Exhibit J) (The
ontology-like domain definition is a dynamic processing environment which reacts to incoming words (captured by the user
interface) to produce a command to be performed by the system.); Guzzoni et al., Modeling Human-Agent Interaction with
Active Ontologies, Association for the Advancement of Artificial Intelligence (2007) (Exhibit L) (The goal of a language
processing component is to gather input utterances, understand their meaning, and to finally generate a command to be
delegated for execution. . . . The assistant interacts in a natural way (plain English) with the meeting organizer and
attendees.).
Claim 1
Ex. 46 - 3
Infringement
Siri processes the natural language input as it is entered by the user, without changing its structure or content.
See, e.g.,
language processing according to one embodiment.. . . .The method begins 200. Language input 202 is received, such as
the string "who is playing this weekend at the filmore" in the example of FIG. 29.
performing, based on
the input, without
augmentation,
Gruber, Siri, a Virtual Personal Assistant (Exhibit E)

As you express what you want to do in the way most comfortable to you, Siri applies a patented algorithm to sift through
multitudes of possible interpretations, applying what it knows about your location, the time, your preferences, and your task
context to determine the most probable understanding of your intent.
See also, Guzzoni et al., Modeling Human-Agent Interaction with Active Ontologies, Association for the Advancement of
Artificial Intelligence (2007) (Exhibit L) (The first stage of the application performs language processing. Incoming
utterances from the meeting organizer are gathered by Active server extensions from either the intelligent assistants email
account or its instant messenger client impersonation. Utterances are then asserted as facts to trigger an Active Ontology in
charge of language processing based on the method described in section Language processing.).
Claim 1
a search of one or
more language-based
databases including
at least one metadata
database comprising
at least one of a
group of information
types comprising:
Ex. 46 - 4
Infringement
Based on the user's input, Siri searches language-based databases that include data as well as information about
the data, including what it means or how it is used in context, to determine the users intent. This information
includes Siri's active ontology and the databases it includes or references as well as user-specific information such
as contacts, geospatial location, time, music, preferences, and past Siri usage.
See, e.g.,
- [0226] Active ontologies may include and/or make reference to any database that can be mapped to concepts or other
representations in ontology 1050. Domain entity databases 1072 and vocabulary databases 1058 are merely two examples
of how active ontology 1050 may integrate databases with each other and with other components of automated assistant
1002.
- [0370] - [0374] Domain model component(s) 1056 may provide lists of words that might match a domain concept or entity,
such as names of restaurants, which may be used for active elicitation of input 100 and natural language processing 200. . . .
Domain model component(s) 1056 may show the relationship between partial information for interpreting natural language,
for example that cuisine may be associated with business entities (e.g., "local Mexican food" may be interpreted as "find
restaurants with style=Mexican", this inference is possible because of the information in domain model 1056). . . . Domain
model component(s) 1056 may provide the information for generating natural language paraphrases and other output
formatting, for example, by providing canonical ways of describing concepts, relations, properties and instances.
- [0384] - [387] In at least one embodiment, language interpreter component(s) 1070 of assistant 1002 may be operable to
perform and/or implement various types of functions, operations, actions, and/or other features such as, for example, one or
more of the following (or combinations thereof): Analyze user input and identify a set of parse results. . . . Parse results are
associations of data in the user input with concepts, relationships, properties, instances, and/or other nodes and/or data
structures in models, databases, and/or other representations of user intent and/context.
- [0401] - [409] In at least one embodiment, a given instance of language interpreter component(s) 1070 may access and/or
utilize information from one or more associated databases. In at least one embodiment, at least a portion of such database
information may be accessed via communication with one or more local and/or remote memory devices. Examples of
different types of data which may be accessed by the Language Interpreter component(s) may include, but are not limited to,
one or more of the following (or combinations thereof): Domain models 1056; Vocabulary 1058; Domain entity databases
1072; Short term memory 1052; Long term personal memory 1054; Task flow models 1086; Dialog flow models 1087;
Service capability models 1088.
Claim 1
Ex. 46 - 5
Infringement
Apple, iOS Software License Agreement (Exhibit F)
- 4.(c) Siri and Dictation. When you use Siri or Dictation, the things you say will be recorded and sent to Apple in order to
convert what you say into text and, for Siri, to also process your requests. Your device will also send Apple other information,
such as your first name and nickname; the names, nicknames, and relationship with you (e.g., my dad) of your address
book contacts; and song names in your collection (collectively, your User Data). All of this data is used to help Siri and
Dictation understand you better and recognize what you say. It is not linked to other data that Apple may have from your use
of other Apple services. By using Siri or Dictation, you agree and consent to Apples and its subsidiaries and agents
transmission, collection, maintenance, processing, and use of this information, including your voice input and User Data, to
provide and improve Siri, Dictation, and other Apple products and services.
MIT Tech Review, TR10: Intelligent Software Assistant (March/April 2009) (Exhibit D)
- The user can ask a broad question like this because Siri has information that gives clues about what the user intends. For
example, the software might store data about the user's location, schedule, and past activities. Siri can deal with open-ended
questions within specific areas, such as entertainment or travel.
IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (2006) (Exhibit J) (Active
Ontologies are made up of a relational network of concepts connected by relationships. Concepts and relationships serve to
define both data structures in the domain (e.g. a movie has a title, actors, a genre, a rating) as well as a processing
environment using rule sets that perform actions within and among concepts.); Guzzoni et al., Active, A Platform for Building
Intelligent Software (Exhibit K) (An Active Ontology is made up of interconnected processing elements called Concepts,
graphically arranged to represent the domain objects, events, actions, and processes that make up an application. Concepts
communicate with each other through channels, passing state information, hypotheses, and requests. Concepts reason
about incoming channel messages, to determine how the information should be combined and acted upon.); Guzzoni et al.,
Modeling Human-Agent Interaction with Active Ontologies, Association for the Advancement of Artificial Intelligence (2007)
(Exhibit L) (To implement this technique, we have created a specialized Active Ontology to work as a service registry and
dynamic service broker. Service providers register their capabilities and attributes by asserting a set of facts. This data set
represents a simple service registry where providers can register their capabilities to later be discovered and invoked.).
Claim 1
Ex. 46 - 6
Infringement
Siri's search includes searching case information, including information previously provided by the user and the
user's past use of Siri.
See, e.g.,
- [0414] Short- and/or long-term memory 1052, 1054 can also be used by language interpreter component(s) 1070 in
generating candidate syntactic parses 212. Thus, input that was provided previously in the same session, and/or known
information about the user, can be used, to improve performance, reduce ambiguity, and reinforce the conversational nature
of the interaction.
Apple, iPhone User Guide, Chapter 4 (Exhibit G)
- Siri also wants to know about the important people in your life, so put those relationships on your personal info cardSiri
can help you. For example, the first time you tell Siri to call your sister, Siri asks you who your sister is (if you dont already
have that info on your card). Siri adds that relationship to your personal info card so it doesnt have to ask next time.
case information;
Nova Spivack, How Siri Works - Interview with Tom Gruber, CTO of SIRI (Jan. 26, 2010) (Exhibit C)
- The kinds of tasks that Siri is particularly good at involve semistructured data, usually on tasking involving multiple criteria
and drawing from multiple sources. For example, to help find a place to eat, user preferences for cuisine, price range,
location, or even specific food items come into play.
- Siri does learn in simple ways today, and it will get more sophisticated with time. As you said, Siri is already personalized
based on immediate context, conversational history, and personal information such as where you live. Siri doesn't forget
things from request to request, as do stateless systems like search engines. It always considers the user model along with
the domain and task models when coming up with results. The evolution in learning comes as users have a history with Siri,
which gives it a chance to make some generalizations about preferences.
- The [Siri] software offers specific suggestions based on the user's personal preferences and its ability to categorize.
- A virtual assistant gives different answers depending on individual preferences and personal context (place, time, history),
and if you give it permission, learns more about you so that it can shorten your time-to-task. Information you teach Siri in one
domain (e.g. movies) is applied automatically to opportunities rising from other domains. Any personal information you
provide Siri is stored in a highly secure, PCI-compliant co-location center, and used only with your explicit permission to
accelerate your task completion.
Claim 1
Ex. 46 - 7
Infringement
Active network of concepts holds the context of the conversation. Nodes and leaves remember their current state, allowing
the user to use partial sentences, or complete information when necessary.).
Siri's search includes searching keywords, including vocabulary databases of words commonly used to identify
concepts.
keywords;
See, e.g.,
- [0081] . . . [T]he system of the present invention may employ one or more memories or memory modules. . . . The memory
or memory modules may also be configured to store data structures, keyword taxonomy information, . . .
- [0225] Active ontologies may include and/or make reference to vocabulary databases 1058. For example, FIG. 8 depicts a
vocabulary database of cuisines 1662, such as Italian, French, and the like, and the words associated with each cuisine such
as "French", "continental", "provincial", and the like.
- [0250] Vocabulary 1058. For example, words or phrases that match the current input may be suggested. In at least one
embodiment, vocabulary may be associated with any or one or more nodes of active ontologies, domain models, task
models, dialog models, and/or service models.
-[0437] For example, in at least some embodiments, various aspects, features, and/or functionalities of vocabulary
component(s) 1058 may be implemented as data structures that associate strings with the names of concepts, relations,
properties, and instances. These data structures may be stored in memory, files, or databases.
- [0773 - 0780] For example, the following is a candidate precedence sorting paradigm for the restaurant domain . . . 7. food
item (not sortable unless a constraint value, e.g., a keyword, is given 8. keywords (not sortable unless a constraint value,
e.g., keyword, is given).
IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (2006) (Exhibit J) (Words
are tested using regular expression pattern matching or compared against a known vocabulary set.)
Claim 1
information models;
and
Ex. 46 - 8
Infringement
Siri's search includes searching information models, including Siri's active-ontology models, Siri's language
pattern recognizers, and Siri's language models.
See, e.g.,
- [0195] Ontologies are used, for example, to build models of data and knowledge. In some embodiments of the intelligent
automated system 1002, ontologies are part of the modeling framework in which to build models such as domain models.
- [0197] - [204] In at least one embodiment, active ontologies 1050 may be operable to perform and/or implement various
types of functions, operations, actions, and/or other features such as, for example, one or more of the following (or
combinations thereof): Act as a modeling and development environment, integrating models and data from various model
and data components, including but not limited to Domain models 1056 Vocabulary 1058 Domain entity databases 1072
Task flow models 1086 Dialog flow models 1087 Service capability models 1088.
- [0251] Domain models 1056, which may constrain the inputs that may instantiate or otherwise be consistent with the
domain model. For example, in at least one embodiment, domain models 1056 may be used to suggest concepts, relations,
properties, and/or instances that would be consistent with the current input.
- [0312] - [0313] In addition, in one embodiment, the statistical language models are tuned to look for words, names, and
phrases that occur in the various models of assistant 1002 shown in FIG. 8. . . . In one embodiment, the statistical language
models are also tuned to look for words, names, and phrases from long-term personal memory 1054.
- [0381] For example, data from domain model component(s) 1056 may be associated with other model modeling
components including vocabulary 1058, language pattern recognizers 1060, dialog flow models 1087, task flow models
1086, service capability models 1088, domain entity databases 1072, and the like. For example, businesses in domain entity
databases 1072 that are classified as restaurants might be known by type identifiers which are maintained in the dining out
domain model components.
- [0412] In word/phrase matching 210, language interpreter component(s) 1070 find associations between user input and
concepts. . . . Word/phrase matching 210 may use data from, for example, language pattern recognizers 1060, vocabulary
database 1058, active ontology 1050, short term personal memory 1052, and long term personal memory 1054.
- [0440] In at least one embodiment, language pattern recognizer component(s) 1060 may be operable to perform and/or
implement various types of functions, operations, actions, and/or other features such as, for example, looking for patterns in
language or speech input that indicate grammatical, idiomatic, and/or other composites of input tokens. . . . In at least one
embodiment, language pattern recognizer component(s) 1060 are represented declaratively, rather than as program code;
this enables them to be created and maintained by editors and other tools other than programming tools. Examples of
declarative representations may include, but are not limited to, one or more of the following (or combinations thereof): regular
expressions, pattern matching rules, natural language grammars, parsers based on state machines and/or other parsing
models.
Claim 1
Ex. 46 - 9
Infringement
- Siris knowledge is represented in a unified modeling system that combines ontologies, inference networks, pattern
matching agents, dictionaries, and dialog models. As much as possible we represent things declaratively (i.e., as data in
models, not lines of code). This is a tried and true best practice for complex AI systems. This makes the whole system more
robust and scalable, and the development process more agile. It also helps with reasoning and learning, since Siri can look
at what it knows and think about similarities and generalizations at a semantic level.
environment using rule sets that perform actions within and among concepts. . . . Once the domain has been defined, a layer
of language processing is applied by associating rule sets directly to the domain concepts.).
Claim 1
database values; and
Ex. 46 - 10
Infringement
Siri's search includes searching database values, including the database objects in or accessed by Siri's active
ontology as well as user-specific information such as contacts, geospatial location, time, and music list.
See, e.g.,
information may be accessed via communication with one or more local and/or remote memory devices. Examples of
different types of data which may be accessed by the Language Interpreter component( s) may include, but are not limited
to, one or more of the following (or combinations thereof): Domain models 1056; Vocabulary 1058; Domain entity databases
1072; Short term memory 1052; Long term personal memory 1054; Task flow models 1086; Dialog flow models 1087;
Service capability models 1088.
- [0422] - [0431] In at least one embodiment, domain entity database (s) 1072 may be operable to perform and/or implement
various types of functions, operations, actions, and/or other features such as, for example, one or more of the following (or
combinations thereof): Store data about domain entities. Domain entities are things in the world or computing environment
that may be modeled in domain models. Examples may include, but are not limited to, one or more of the following (or
combinations thereof): Businesses of any kind; Movies, videos, songs and/or other musical products, and/ or any other
named entertainment products; Products of any kind; Events; Calendar entries; Cities, states, countries, neighborhoods,
and/or geographic, geopolitical, and/or geospatial points or regions; Named places such as landmarks, airports, and the like;
Provide database services on these databases, including but not limited to simple and complex queries, transactions,
triggered events, and the like.
- Siri also uses information from your contacts, music library, calendars, and reminders to better understand what you say.
So it responds more accurately when you ask to make a phone call, play music, or create an appointment or reminder.
http://www.apple.com/ios/siri/ (Exhibit I)
- Siri makes everyday tasks less tasking. It figures out which apps to use for which requests, and it finds answers to queries
through sources like Yelp and WolframAlpha. It plays the songs you want to hear, gives you directions, wakes you up, even
tells you the score of last nights game.
Claim 1
Ex. 46 - 11
Infringement
- Siri also uses information from your contacts, music library, calendars, reminders, and so forth to understand what youre
talking about.
- Because Siri knows locations (iPhone 4S or later) like current, home, and work, it can remind you to do a certain task
when you leave a location or arrive at a location. Tell Siri Remind me to call my daughter when I leave the office, and Siri
does just that.
- Siri works with Yelp, OpenTable, and others to provide information about restaurants and help you make reservations. Ask
to find restaurants by cuisine, price, location, outdoor seating, or a combination of options. Siri can show you available
photos, Yelp stars, price range, and reviews.
- Ask Siri about what movies are playing, or where you can see a specific movie. Find out when a film premiered, who
directed it and what awards it won. Siri gives theater locations, show times, and Rotten Tomato reviews.
- Siri knows a lot about sportsincluding baseball, basketball, football, soccer, and hockey. Ask Siri for game schedules,
scores from the current seasons games, or up-to-the minute scores from live games. Tell Siri to show you player stats and
compare them against other players stats. Siri tracks team records, too.
IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (2006) (Exhibit J) (In our
example, we have used Active to model and deploy a natural language driven assistant able to retrieve information about
restaurants and movies.).
Claim 1
Ex. 46 - 12
Infringement
Siri provides the result of the search to the user through a user interface by, for example, responding to the user
through the device screen or through audio.
See, e.g.,
- What happens after I ask Siri a question or ask it to do something? When you finish speaking, Siri displays the text of what
you said and provides a response.
- When you stop speaking, Siri displays what it heard and provides a response. Siri often includes related info that might be
useful. If the info is related to an appfor example, a text message youve composed, or a location you asked forjust tap
the display to open the app for details and further action. (Figure reproduced below.)
providing, through a
user interface, a
result of the search to
the user;
See also, U.S. Pat. Publn No. 2012/0265528 Fig. 15 (Exhibit N) (Referring now to FIG. 15 . . . . In screen 1551, the user
first asks What's the time in New York; virtual assistant 1002 responds 1552 by providing the current time in New York
City.); U.S. Pat. Publn No. 2012/0016678 410, Fig. 29 (Exhibit B) (Referring now also to FIG. 29, The user has entered
(via voice or text) language input 2902 consisting of the phrase who is playing this weekend at the filmore. This phrase is
echoed back to the user on screen 2901. . . . The parse result associates that input with a request to show the local events
that are scheduled for any of the upcoming weekend days at any event venue whose name matches "filmore". A paraphrase
of the parse results is shown as 2903 on screen 2901.).
Claim 1
identifying, for the

one or more
language-based
databases, a finite
number of database
objects; and
Ex. 46 - 13
Infringement
Siri identifies a finite number of database objects by, for example, generating candidate parses based on the user
input.
See, e.g.,
structures in models, databases, and/or other representations of user intent and/context. . . . Analyze user input and identify
a set of syntactic parse results, which are parse results that associate data in the user input with structures that represent
syntactic parts of speech, clauses and phrases including multiword names, sentence structure, and/ or other grammatical
graph structures. . . . Analyze user input and identify a set of semantic parse results, which are parse results that associate
data in the user input with structures that represent concepts, relationships, properties, entities, quantities, propositions,
and/or other representations of meaning and user intent. In one embodiment, these representations of meaning and intent
are represented by sets of and/or elements of and/or instances of models or databases and/or nodes in ontologies, as
described in element 220 of natural language processing procedure described in connection with FIG. 28.
- [0413] Language interpreter component(s) 1070 generate candidate syntactic parses 212 which include the chosen parse
result but may also include other parse results.
IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (2006) (Exhibit J) (Gather
nodes (i.e. the movie node) create and rate an aggregated structured object by combining inputs coming from their
children.).
Siri determines a plurality of combinations of the database objects by, for example, combining the parse results.
determining a plurality
of combinations of the
finite number of
database objects.
See, e.g.,
- [0415] In semantic matching 220, language interpreter component( s) 1070 consider combinations of possible parse results
according to how well they fit semantic models such as domain models and databases.
- [0417] A set of candidate, or potential, semantic parse results is generated 222.
IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (2006) (Exhibit J)
(Selection nodes (i.e. the event node) choose the single best rating coming from their children. . . . Through this bottom up
execution, input tokens are incrementally assembled up the domain tree to produce a structured command at the root
node.).
Claim 2
2. The method
according to claim 1,
further comprising a
step of mapping the
natural language query
to the plurality of
combinations.
Claim 3
3. The method
wherein the step of
mapping comprises
steps of:
identifying keywords in
the natural language
query; and
relating the keywords to
the plurality of
combinations.
Ex. 46 - 14
Infringement
Siri maps user's natural language query to the combinations of database objects by, for example, evaluating the
candidate parses and selecting the top parses as representations of the user's intent.
See, e.g.,
- [0421] In 232, language interpreter component( s) 1070 sort and select 232 the top semantic parses as the
representation of user intent 290.
See also, Guzzoni et al., Active: A unified platform for building intelligent web interaction assistants, Proceedings of the
2006 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (2006) (Exhibit J)
(The ontology-like domain definition is a dynamic processing environment which reacts to incoming words (captured by
the user interface) to produce a command to be performed by the system.).
Infringement
Siri uses keywords to map the natural language query to the combinations of database objects by, for example,
evaluating the candidate parses by identifying keywords in the query and relating those keywords to the set of
candidate parse results.
See, e.g.,
- [0418] - [0419] In disambiguation step 230, language interpreter component(s) 1070 weigh the evidential strength of
candidate semantic parse results 222. In this example, the combination of the parse of "playing" as "Local Event At Venue"
and the match of "filmore" as a Venue Name is a stronger match to a domain model than alternative combinations where,
for instance, "playing" is associated with a domain model for sports but there is no association in the sports domain for
"filmore". . . . For example, in one embodiment of semantic matching 220, the parse that matches both Local Event At
Venue and Venue Name is given increased evidential support because the combined representations of these aspects of
the user intent are connected by links and/or relations in active ontology 1050: in this instance, the Local Event node is
connected to the Venue node which is connected to the Venue Name node which is connected to the entity name in the
database of venue names.
Claim 4
4. The method
further comprising a
step of determining a
reference dictionary
comprising:
Ex. 46 - 15
Infringement
Siri determines a reference dictionary by, for example, providing its candidate parses in the form of an active
ontology and databases referenced by an active ontology.
See, e.g.,
- [0415] - [0417] In semantic matching 220, language interpreter component( s) 1070 consider combinations of possible
parse results according to how well they fit semantic models such as domain models and databases. In this case, the
parse includes the associations (1) "playing" (a word in the user input) as "Local Event At Venue" (part of a domain model
1056 represented by a cluster of nodes in active ontology 1050) and (2) "filmore" (another word in the input) as a match to
an entity name in a domain entity database 1072 for Local Event Venues, which is represented by a domain model
element and active ontology node (Venue Name). Semantic matching 220 may use data from, for example, active
ontology 1050, short term personal memory 1052, and long term personal memory 1054. For example, semantic matching
220 may use data from previous references to venues or local events in the dialog (from short term personal memory
1052) or personal favorite venues (from long term personal memory 1054). A set of candidate, or potential, semantic parse
results is generated 222.
Claim 4
Ex. 46 - 16
Infringement
Siri's candidate set includes cases, including the user's preferences and past usage of Siri.
See, e.g.,
information about the user, can be used, to improve performance, reduce ambiguity, and reinforce the conversational
nature of the interaction.
case information;
things from request to request, as do stateless systems like search engines. It always considers the user model along with
the domain and task models when coming up with results. The evolution in learning comes as users have a history with
Siri, which gives it a chance to make some generalizations about preferences.
- A virtual assistant gives different answers depending on individual preferences and personal context (place, time,
history), and if you give it permission, learns more about you so that it can shorten your time-to-task. Information you teach
Siri in one domain (e.g. movies) is applied automatically to opportunities rising from other domains. Any personal
information you provide Siri is stored in a highly secure, PCI-compliant co-location center, and used only with your explicit
permission to accelerate your task completion.
Claim 4
Ex. 46 - 17
Infringement
(The Active network of concepts holds the context of the conversation. Nodes and leaves remember their current state,
allowing the user to use partial sentences, or complete information when necessary.).
Siri's candidate set includes keywords, including vocabulary databases of words commonly used to identify
concepts (e.g., cuisine types).
keywords;
See, e.g.,
- [0081] . . . [T]he system of the present invention may employ one or more memories or memory modules. . . . The
memory or memory modules may also be configured to store data structures, keyword taxonomy information, . . .
- [0225] Active ontologies may include and/or make reference to vocabulary databases 1058. For example, FIG. 8 depicts
a vocabulary database of cuisines 1662, such as Italian, French, and the like, and the words associated with each cuisine
such as "French", "continental", "provincial", and the like.
embodiment, vocabulary may be associated with any or one or more nodes of active ontologies, domain models, task
models, dialog models, and/or service models.
- [0773 - 0780] For example, the following is a candidate precedence sorting paradigm for the restaurant domain . . . 7.
food item (not sortable unless a constraint value, e.g., a keyword, is given 8. keywords (not sortable unless a constraint
value, e.g., keyword, is given).
(Words are tested using regular expression pattern matching or compared against a known vocabulary set.)
Claim 4
information models; and
Ex. 46 - 18
Infringement
Siri's candidate set includes information models, including Siri's active-ontology models, Siri's language pattern
recognizers, and Siri's language models.
See, e.g.,
combinations thereof): Act as a modeling and development environment, integrating models and data from various model
and data components, including but not limited to Domain models 1056 Vocabulary 1058 Domain entity databases 1072
Task flow models 1086 Dialog flow models 1087 Service capability models 1088.
- [0251] Domain models 1056, which may constrain the inputs that may instantiate or otherwise be consistent with the
domain model. For example, in at least one embodiment, domain models 1056 may be used to suggest concepts,
relations, properties, and/or instances that would be consistent with the current input.
phrases that occur in the various models of assistant 1002 shown in FIG. 8. . . . In one embodiment, the statistical
language models are also tuned to look for words, names, and phrases from long-term personal memory 1054.
components including vocabulary 1058, language pattern recognizers 1060, dialog flow models 1087, task flow models
1086, service capability models 1088, domain entity databases 1072, and the like. For example, businesses in domain
entity databases 1072 that are classified as restaurants might be known by type identifiers which are maintained in the
dining out domain model components.
implement various types of functions, operations, actions, and/or other features such as, for example, looking for patterns
in language or speech input that indicate grammatical, idiomatic, and/or other composites of input tokens. . . . In at least
one embodiment, language pattern recognizer component(s) 1060 are represented declaratively, rather than as program
code; this enables them to be created and maintained by editors and other tools other than programming tools. Examples
of declarative representations may include, but are not limited to, one or more of the following (or combinations thereof):
regular expressions, pattern matching rules, natural language grammars, parsers based on state machines and/or other
parsing models.
Claim 4
Ex. 46 - 19
Infringement
- Siris knowledge is represented in a unified modeling system that combines ontologies, inference networks, pattern
matching agents, dictionaries, and dialog models. As much as possible we represent things declaratively (i.e., as data in
models, not lines of code). This is a tried and true best practice for complex AI systems. This makes the whole system
more robust and scalable, and the development process more agile. It also helps with reasoning and learning, since Siri
can look at what it knows and think about similarities and generalizations at a semantic level.
(Active Ontologies are made up of a relational network of concepts connected by relationships. Concepts and
relationships serve to define both data structures in the domain (e.g. a movie has a title, actors, a genre, a rating) as well
as a processing environment using rule sets that perform actions within and among concepts. . . . Once the domain has
been defined, a layer of language processing is applied by associating rule sets directly to the domain concepts.).
Claim 4
database values.
Ex. 46 - 20
Infringement
Siri's candidate set includes database values, including the database objects in or accessed by Siri's active
See, e.g.,
- [0401] - [409] In at least one embodiment, a given instance of language interpreter component(s) 1070 may access
and/or utilize information from one or more associated databases. In at least one embodiment, at least a portion of such
database information may be accessed via communication with one or more local and/or remote memory devices.
Examples of different types of data which may be accessed by the Language Interpreter component( s) may include, but
are not limited to, one or more of the following (or combinations thereof): Domain models 1056; Vocabulary 1058; Domain
entity databases 1072; Short term memory 1052; Long term personal memory 1054; Task flow models 1086; Dialog flow
models 1087; Service capability models 1088.
- [0422] - [0431] In at least one embodiment, domain entity database (s) 1072 may be operable to perform and/or
implement various types of functions, operations, actions, and/or other features such as, for example, one or more of the
following (or combinations thereof): Store data about domain entities. Domain entities are things in the world or computing
environment that may be modeled in domain models. Examples may include, but are not limited to, one or more of the
following (or combinations thereof): Businesses of any kind; Movies, videos, songs and/or other musical products, and/ or
any other named entertainment products; Products of any kind; Events; Calendar entries; Cities, states, countries,
neighborhoods, and/or geographic, geopolitical, and/or geospatial points or regions; Named places such as landmarks,
airports, and the like; Provide database services on these databases, including but not limited to simple and complex
queries, transactions, triggered events, and the like.
- Siri also uses information from your contacts, music library, calendars, and reminders to better understand what you say.
So it responds more accurately when you ask to make a phone call, play music, or create an appointment or reminder.
- Siri makes everyday tasks less tasking. It figures out which apps to use for which requests, and it finds answers to
queries through sources like Yelp and WolframAlpha. It plays the songs you want to hear, gives you directions, wakes you
up, even tells you the score of last nights game.
Claim 4
Ex. 46 - 21
Infringement
talking about.
does just that.
- Siri works with Yelp, OpenTable, and others to provide information about restaurants and help you make reservations.
Ask to find restaurants by cuisine, price, location, outdoor seating, or a combination of options. Siri can show you available
photos, Yelp stars, price range, and reviews.
(In our example, we have used Active to model and deploy a natural language driven assistant able to retrieve information
about restaurants and movies.).
Claim 5
Ex. 46 - 22
Infringement
Siri resolves ambiguity in its disambiguation step, which involves assigning and weighing the "evidential
strength" of the candidate parse results.
5. The method
wherein the step of
mapping further
comprises resolving
ambiguity between the
keywords and the
plurality of
combinations.
Claim 6
See, e.g.,
perform and/or implement various types of functions, operations, actions, and/or other features such as, for example, one
or more of the following (or combinations thereof): . . . Disambiguate among alternative syntactic or semantic parse results
as described in element 230 of natural language processing procedure described in connection with FIG. 28.
- [0418] In disambiguation step 230, language interpreter component(s) 1070 weigh the evidential strength of candidate
semantic parse results 222.
Infringement
Siri resolves ambiguity in part by ordering potential candidate parse results according to the relative probability
that the candidate interprets the user's intent. The ordering is determined by the application of rules (such as a
priori precedence ordering) or heuristics (such as minimizing path length between candidate nodes in the active
ontology).
6. The method
wherein the step of
resolving includes
determining an optimal
interpretation of the
using at least one of a
group comprising rules
and heuristics.
See, e.g.,
candidate semantic parse results 222. . . . Disambiguation 230 may use data from, for example, the structure of active
ontology 1050. In at least one embodiment, the connections between nodes in an active ontology provide evidential
support for disambiguating among candidate semantic parse results 222. For example, in one embodiment, if three active
ontology nodes are semantically matched and are all connected in active ontology 1050, this indicates higher evidential
strength of the semantic parse than if these matching nodes were not connected or connected by longer paths of
connections in active ontology 1050.
- [0768] - [ 0790] The techniques described herein allow assistant 1002 to be extremely robust in the face of partially
specified constraints and incomplete data. In one embodiment, assistant 1002 uses these techniques to generate a user
list of items in best-first order, i.e. according to relevance. In one embodiment, such relevance sorting is based on an a
priori precedence ordering. . . . For example, the following is a candidate precedence sorting paradigm for the restaurant
domain: [rule set] . . . The following is an example of a design rationale for the above sorting paradigm: [justification] . . .
The remaining constraints except for name are one or more based on incomplete data or matching. So they are weak sort
heuristics by default, and when they are specified the match one or more-or-none.
Claim 7
7. The method of claim
1 wherein providing a
input by the user
includes providing a
input by the user
without constraining
said query
grammatically or
structurally.
Ex. 46 - 23
Infringement
The natural language query input by the Siri user is not constrained grammatically or structurally.
See, e.g.,
Do I have to say things a certain way to get Siri to respond?
No. You can speak to Siri as you would to a person in a natural voice with a conversational tone. If you want to know
what the weather will be like tomorrow, simply say What will the weather be like tomorrow? Or Does it look like rain
tomorrow? Or even Will I need an umbrella tomorrow? No matter how you ask, Siri will tell you the forecast.
Claim 8
Ex. 46 - 24
Infringement
The natural language query input by Siri's user may be a sentence or a sentence fragment.
8. The method of claim

7 wherein the natural
language query
comprises a sentence,
multiple sentences, a
non-sentence word
group or a combination
thereof.
See, e.g.,
No. You can speak to Siri as you would to a person in a natural voice with a conversational tone. If you want to know
what the weather will be like tomorrow, simply say What will the weather be like tomorrow? Or Does it look like rain
tomorrow? Or even Will I need an umbrella tomorrow? No matter how you ask, Siri will tell you the forecast.
Theres so much you can say to Sirihere are some more examples, for starters:
Call Joe
Set the timer for 30 minutes
Directions to the nearest Apple store
Is it going to rain tomorrow?
Open Passbook
Post to Facebook
Tweet
Claim 9
Ex. 46 - 25
Infringement
Siri implements methods for processing a natural language input provided by a user. It takes the user's natural
language input, entered as speech or in writing, and processes the input to, for example, send messages, schedule
meetings, and place phone calls.
See, e.g.,
- What is Siri? Siri is the intelligent personal assistant that helps you get things done just by asking. It allows you to use your
voice to send messages, schedule meetings, place phone calls, and more. But Siri isnt like traditional voice recognition
software that requires you to remember keywords and speak specific commands. Siri understands your natural speech, and it
asks you questions if it needs more information to complete a task.
9. A computerimplemented
method for
processing a natural
language input
comprising:

- [0008] - [0010] According to various embodiments of the present invention, an intelligent automated assistant is
implemented on an electronic device, to facilitate user interaction with a device, and to help the user more effectively engage
with local and/or remote services. In various embodiments, the intelligent automated assistant engages with the user in an
integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain
information or perform various actions. . . . In some embodiments, the intelligent automated assistant systems of the present
invention can perform any or all of: actively eliciting input from a user, interpreting user intent, disambiguating among
competing interpretations, requesting and receiving clarifying information as needed, and performing (or initiating) actions
based on the discerned intent.
- [0042] - [0043] FIG. 28 is a flow diagram depicting an example of a method for natural language processing according to
one embodiment. FIG. 29 is a screen shot illustrating natural language processing according to one embodiment.
- [0315] In various embodiments, algorithms or procedures used by assistant 1002 for interpretation of text inputs, including
any embodiment of the natural language processing procedure shown in FIG. 28, can be used to rank and score candidate
text interpretations 124 generated by speech-to-text service 122.
See also, U.S. Pat. Publn No. 2007/0100790 18 (Exhibit M) (As described in further detail with respect to FIG. 3, an active
ontology such as the active ontology 100 may be built to process facts in order to achieve any one of a number of goals,
including activity recognition (e.g., natural language processing, human activity recognition, etc.)); Guzzoni et al., Active: A
unified platform for building intelligent web interaction assistants, Proceedings of the 2006 IEEE/WIC/ACM International
Conference on Web Intelligence and Intelligent Agent Technology (2006) (Exhibit J) (In our example, we have used Active
to model and deploy a natural language driven assistant able to retrieve information about restaurants and movies. . . . To
create a language processing application using Active Ontologies, the first step consists of using the Active Editor to
graphically specify the application domain, sketching key concepts and relationships (figure 1).); Guzzoni et al., Modeling
Human-Agent Interaction with Active Ontologies, Association for the Advancement of Artificial Intelligence (2007) (Exhibit L)
(The core of the application consists of three Active Ontologies implementing the overall behavior of the intelligent assistant:
language processing.).
Claim 9
Ex. 46 - 26
Infringement
Siri receives a natural language input query by the user. For example, Siri allows the user to ask Siri a question or
ask it to do something in natural speech, as the user would speak to a person. Siri receives this natural language
query input and processes it to determine user intent.
See, e.g.,
- How do I ask Siri something? To talk to Siri, hold down the Home button and youll hear two quick beeps and see What can
I help you with? on the screen. Just begin speaking.
- Do I have to say things a certain way to get Siri to respond? No. You can speak to Siri as you would to a person in a
natural voice with a conversational tone. If you want to know what the weather will be like tomorrow, simply say What will the
weather be like tomorrow? Or Does it look like rain tomorrow? Or even Will I need an umbrella tomorrow? No matter how
you ask, Siri will tell you the forecast.
receiving a natural
language input;

- [0009] - [0011] According to various embodiments of the present invention, the intelligent automated assistant integrates a
variety of capabilities provided by different software components (e.g., for supporting natural language recognition and dialog,
multimodal input, personal information management, task flow management, orchestrating distributed services, and the like). .
. . In some embodiments, the intelligent automated assistant systems of the present invention can perform any or all of:
actively eliciting input from a user, interpreting user intent, disambiguating among competing interpretations, requesting and
receiving clarifying information as needed, and performing (or initiating) actions based on the discerned intent. . . . In
addition, in various embodiments, the assistant of the present invention provides a conversational interface that the user may
find more intuitive and less burdensome than conventional graphical user interfaces. The user can engage in a form of
conversational dialog with the assistant using any of a number of available input and output mechanisms, such as for
example speech, graphical user interfaces (buttons and links), text entry, and the like.
language processing according to one embodiment.. . . .The method begins 200. Language input 202 is received, such as the
string "who is playing this weekend at the filmore" in the example of FIG. 29.
interface) to produce a command to be performed by the system.); Guzzoni et al., Modeling Human-Agent Interaction with
Active Ontologies, Association for the Advancement of Artificial Intelligence (2007) (Exhibit L) (The goal of a language
processing component is to gather input utterances, understand their meaning, and to finally generate a command to be
delegated for execution. . . . The assistant interacts in a natural way (plain English) with the meeting organizer and
attendees.).
Claim 9
providing from said

natural language
input a plurality of
language-based
database objects;
Ex. 46 - 27
Infringement
Siri provides a finite number of database objects by, for example, generating candidate parses based on the user
input.
See, e.g.,
structures in models, databases, and/or other representations of user intent and/context. . . . Analyze user input and identify a
set of syntactic parse results, which are parse results that associate data in the user input with structures that represent
syntactic parts of speech, clauses and phrases including multiword names, sentence structure, and/ or other grammatical
graph structures. . . . Analyze user input and identify a set of semantic parse results, which are parse results that associate
data in the user input with structures that represent concepts, relationships, properties, entities, quantities, propositions,
and/or other representations of meaning and user intent. In one embodiment, these representations of meaning and intent are
represented by sets of and/or elements of and/or instances of models or databases and/or nodes in ontologies, as described
in element 220 of natural language processing procedure described in connection with FIG. 28.
- [0413] - [0417] Language interpreter component(s) 1070 generate candidate syntactic parses 212 which include the chosen
parse result but may also include other parse results. . . . In semantic matching 220, language interpreter component( s)
1070 consider combinations of possible parse results according to how well they fit semantic models such as domain models
and databases. In this case, the parse includes the associations (1) "playing" (a word in the user input) as "Local Event At
Venue" (part of a domain model 1056 represented by a cluster of nodes in active ontology 1050) and (2) "filmore" (another
word in the input) as a match to an entity name in a domain entity database 1072 for Local Event Venues, which is
represented by a domain model element and active ontology node (Venue Name). . . . A set of candidate, or potential,
semantic parse results is generated 222.
IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (2006) (Exhibit J) (Gather
nodes (i.e. the movie node) create and rate an aggregated structured object by combining inputs coming from their children.).
Claim 9
identifying a finite
number of
permutations of the
plurality of database
objects, the
database objects
being stored in a
metadata database
comprising at least
one of a group of
information
comprising:
Ex. 46 - 28
Infringement
Siri identifies a finite number of permutations of the selected database objects by, for example, combining the parse
results where the candidate parses are data objects stored in various databases that include data as well as
information about the data, including what the data means or how it is used in context.
See, e.g.,
- [0226] Active ontologies may include and/or make reference to any database that can be mapped to concepts or other
representations in ontology 1050. Domain entity databases 1072 and vocabulary databases 1058 are merely two examples of
how active ontology 1050 may integrate databases with each other and with other components of automated assistant 1002.
- [0370] - [0374] Domain model component(s) 1056 may provide lists of words that might match a domain concept or entity,
such as names of restaurants, which may be used for active elicitation of input 100 and natural language processing 200. . . .
Domain model component(s) 1056 may show the relationship between partial information for interpreting natural language, for
example that cuisine may be associated with business entities (e.g., "local Mexican food" may be interpreted as "find
restaurants with style=Mexican", this inference is possible because of the information in domain model 1056). . . . Domain
model component(s) 1056 may provide the information for generating natural language paraphrases and other output
formatting, for example, by providing canonical ways of describing concepts, relations, properties and instances.
structures in models, databases, and/or other representations of user intent and/context.
information may be accessed via communication with one or more local and/or remote memory devices. Examples of different
types of data which may be accessed by the Language Interpreter component( s) may include, but are not limited to, one or
more of the following (or combinations thereof): Domain models 1056; Vocabulary 1058; Domain entity databases 1072;
Short term memory 1052; Long term personal memory 1054; Task flow models 1086; Dialog flow models 1087; Service
capability models 1088.
Apple, iOS Software License Agreement (Exhibit F)
- 4.(c) Siri and Dictation. When you use Siri or Dictation, the things you say will be recorded and sent to Apple in order to
convert what you say into text and, for Siri, to also process your requests. Your device will also send Apple other information,
such as your first name and nickname; the names, nicknames, and relationship with you (e.g., my dad) of your address
book contacts; and song names in your collection (collectively, your User Data). All of this data is used to help Siri and
Dictation understand you better and recognize what you say. It is not linked to other data that Apple may have from your use
of other Apple services. By using Siri or Dictation, you agree and consent to Apples and its subsidiaries and agents
transmission, collection, maintenance, processing, and use of this information, including your voice input and User Data, to
provide and improve Siri, Dictation, and other Apple products and services.
Claim 9
Ex. 46 - 29
Infringement
- The user can ask a broad question like this because Siri has information that gives clues about what the user intends. For
example, the software might store data about the user's location, schedule, and past activities. Siri can deal with open-ended
questions within specific areas, such as entertainment or travel.
IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (2006) (Exhibit J) (Selection
nodes (i.e. the event node) choose the single best rating coming from their children. . . . Through this bottom up execution,
input tokens are incrementally assembled up the domain tree to produce a structured command at the root node.); Guzzoni
et al., Modeling Human-Agent Interaction with Active Ontologies, Association for the Advancement of Artificial Intelligence
(2007) (Exhibit L) (To implement this technique, we have created a specialized Active Ontology to work as a service registry
and dynamic service broker. Service providers register their capabilities and attributes by asserting a set of facts. This data
set represents a simple service registry where providers can register their capabilities to later be discovered and invoked.).
Claim 9
Ex. 46 - 30
Infringement
Siri uses data objects that are stored in databases that include case information, including the user's preferences
and past usage of Siri.
See, e.g.,
of the interaction.
case information;
things from request to request, as do stateless systems like search engines. It always considers the user model along with the
domain and task models when coming up with results. The evolution in learning comes as users have a history with Siri,
Claim 9
keywords;
Ex. 46 - 31
Infringement
Siri uses data objects that are stored in databases that include keywords, including vocabulary databases of words
commonly used to identify concepts (e.g., cuisine types).
See, e.g.,
embodiment, vocabulary may be associated with any or one or more nodes of active ontologies, domain models, task models,
dialog models, and/or service models.
item (not sortable unless a constraint value, e.g., a keyword, is given 8. keywords (not sortable unless a constraint value, e.g.,
keyword, is given).
Claim 9
information models;
and
Ex. 46 - 32
Infringement
Siri uses data objects that are stored in databases that include information models, including Siri's active-ontology
models, Siri's language pattern recognizers, and Siri's language models.
See, e.g.,
combinations thereof): Act as a modeling and development environment, integrating models and data from various model and
data components, including but not limited to Domain models 1056 Vocabulary 1058 Domain entity databases 1072 Task flow
models 1086 Dialog flow models 1087 Service capability models 1088.
- [0251] Domain models 1056, which may constrain the inputs that may instantiate or otherwise be consistent with the domain
model. For example, in at least one embodiment, domain models 1056 may be used to suggest concepts, relations,
components including vocabulary 1058, language pattern recognizers 1060, dialog flow models 1087, task flow models 1086,
service capability models 1088, domain entity databases 1072, and the like. For example, businesses in domain entity
models.
Claim 9
Ex. 46 - 33
Infringement
- Siris knowledge is represented in a unified modeling system that combines ontologies, inference networks, pattern matching
agents, dictionaries, and dialog models. As much as possible we represent things declaratively (i.e., as data in models, not
lines of code). This is a tried and true best practice for complex AI systems. This makes the whole system more robust and
scalable, and the development process more agile. It also helps with reasoning and learning, since Siri can look at what it
knows and think about similarities and generalizations at a semantic level.
Claim 9
database values;
and
Ex. 46 - 34
Infringement
Siri's search includes searching database values, including the database objects in or accessed by Siri's active
See, e.g.,
- Siri also uses information from your contacts, music library, calendars, and reminders to better understand what you say. So
it responds more accurately when you ask to make a phone call, play music, or create an appointment or reminder.
Claim 9
Ex. 46 - 35
Infringement
talking about.
does just that.
to find restaurants by cuisine, price, location, outdoor seating, or a combination of options. Siri can show you available photos,
Yelp stars, price range, and reviews.
Claim 9
interpreting at least
one of the
permutations to
provide
determination of a
result of the natural
language input.
Ex. 46 - 36
Infringement
Siri interprets at least one of the potential candidate parse results according to the relative probability that the
candidate interprets the user's intent.
See, e.g.,
ontology 1050. In at least one embodiment, the connections between nodes in an active ontology provide evidential support
for disambiguating among candidate semantic parse results 222. For example, in one embodiment, if three active ontology
nodes are semantically matched and are all connected in active ontology 1050, this indicates higher evidential strength of the
semantic parse than if these matching nodes were not connected or connected by longer paths of connections in active
ontology 1050. . . . In 232, language interpreter component( s) 1070 sort and select 232 the top semantic parses as the
representation of user intent 290.
- [0768] - [ 0790] The techniques described herein allow assistant 1002 to be extremely robust in the face of partially specified
constraints and incomplete data. In one embodiment, assistant 1002 uses these techniques to generate a user list of items in
best-first order, i.e. according to relevance. In one embodiment, such relevance sorting is based on an a priori precedence
ordering. . . . For example, the following is a candidate precedence sorting paradigm for the restaurant domain: [rule set] . . .
The following is an example of a design rationale for the above sorting paradigm: [justification] . . . The remaining constraints
except for name are one or more based on incomplete data or matching. So they are weak sort heuristics by default, and
when they are specified the match one or more-or-none.
interface) to produce a command to be performed by the system.).
Claim 10
10. The method

wherein the step of
providing a plurality
of database objects
includes providing at
least one of the
group comprising
data types, data
instances, and
database
computational
operators.
Ex. 46 - 37
Infringement
Siri provides a plurality of database objects including data types, data instances, or database computational
operators by, for example, providing candidate parses which include data types and instances as well as
computational operators. For example, the candidate parses may be represented as an active ontology, in which
data objects are represented as nodes and connections between nodes. A node may correspond to a data type (e.g.,
restaurant) and a data instance (e.g., the name of a particular restaurant) and may be connected to other nodes
through a computational operator (relation). Generally, Siri's candidate parses are complex mappings to data
concepts (types), relations (operators), properties, instances, nodes, or data structures.
See, e.g.,
- [0206] For example, some nodes of an active ontology may correspond to domain concepts such as restaurant and its
property restaurant name. During live execution, these active ontology nodes may be instantiated with the identity of a
particular restaurant entity and its name, and how its name corresponds to words in a natural language input utterance.
- [0384] - [0387] [Language interpreter may be operable to] [a]nalyze user input and identify a set of parse results. . . . Parse
results are associations of data in the user input with concepts, relationships, properties, instances, and/or other nodes and/or
data structures in models, databases, and/or other representations of user intent and/context. Parse result associations can
be complex mappings from sets and sequences of words, signals, and other elements of user input to one or more associated
concepts, relations, properties, instances, other nodes, and/or data structures described herein.
- [0384] - [0388] [Language interpreter may be operable to] [a]nalyze user input and identify a set of syntactic parse results,
which are parse results that associate data in the user input with structures that represent syntactic parts of speech, clauses
and phrases including multiword names, sentence structure, and/ or other grammatical graph structures.
- [0384] - [0389] [Language interpreter may be operable to] [a]nalyze user input and identify a set of semantic parse results,
which are parse results that associate data in the user input with structures that represent concepts, relationships, properties,
entities, quantities, propositions, and/or other representations of meaning and user intent. In one embodiment, these
representations of meaning and intent are represented by sets of and/or elements of and/or instances of models or databases
and/or nodes in ontologies, as described in element 220 of natural language processing procedure described in connection
with FIG. 28.
Claim 11
Ex. 46 - 38
Infringement
Siri interprets the user's natural language input by mapping at least one of the candidate parses to a database query.
For example, Siri may interpret the user's input as a query to a web-based database.
See, e.g.,
- What types of things can I ask Siri about or ask it to do? You can ask Siri to make a call, find a business and get directions,
schedule reminders and meetings, search the web, and more.
- Which apps does Siri work with? Siri works with almost all your built-in apps. And its smart enough to figure out which apps
to use to provide you with answers. It also uses Search and Location Services to help you with your requests. Heres a list of
apps and services that Siri works with worldwide:
11. The method
according to claim
10, wherein the step
of interpreting
includes mapping
the at least one of
the permutations to
a database query.

- [0120] - [0123] For example, according to different embodiments, at least some intelligent automated assistant system( s)
may be configured, designed, and/or operable to provide various different types of operations, functionalities, and/or features,
such as, for example, one or more of the following (or combinations thereof): . . . enable the operation of applications and
services via natural language dialog that are otherwise provided by dedicated applications with graphical user interfaces
including search (including location-based search); navigation (maps and directions); database lookup (such as finding
businesses or people by name or other properties); getting weather conditions and forecasts, checking the price of market
items or status of financial transactions; monitoring traffic or the status of flights;
- [0162] - [0165] In one embodiment, information screen lOlE shows information gathered and combined from a variety of
services, including for example, any or all of the following: Addresses and geolocations of businesses; Distance from user's
current location; Reviews from a plurality of sources.
- [0773] - [0790] For example, the following is a candidate precedence sorting paradigm for the restaurant domain: . . . 2.
availability . . . Availability (as determined by a search on a website such as opentable.com, for instance) is a valuable sort
criterion, and may be based on a default value for sorting when not specified.
Claim 12
12. The method

according to claim
11, wherein the
database query is
formulated in a
structured query
language (SQL)
query.
Ex. 46 - 39
Infringement
Siri structures its queries for particular databases using a structured query language (SQL). For example, Siri
queries modern commercial databases, which is typically done by generating a SQL query.
Harris, Our Quest to Chat with HAL (Exhibit T)
- The proof of proper understanding of questions is, of course, providing the right answers. In modern commercial systems,
such as EasyAsk (the company I work for), this is done by generating a SQL query that, when run against a relational
database, will yield the right answer. For example, a question such as "What customers had orders in the last year but had no
orders in the last quarter?" would result in a complex SQL query. The answer, of course, is of particular interest because
these customers may need some attention from the sales force.
- Apple's Siri product has shown that this database model can be extended to generating calls to smartphone apps that can
take appropriate action. An example of this is asking your iPhone 4s "How far away is Cleveland?" Siri understands the
meaning of your question and invokes the Maps app with specific parameters, causing it to not only the answer the question
but provide directions as well.
Apple, Jobs at Apple (Exhibit U)
- Siri Software Engineer, Data
- Job Summary . . . A Siri Data Engineer works on establishing and operating substantial Big Data pipelines using cutting
edge tools and takes those pipelines from collection to insight.
- Key Qualifications . . . Expertise with BI (SQL, OLAP) and Big Data analytics (Hadoop, Hive, and/or Pig) tools with
experience writing and tuning queries in a online or offline environment. NL and Machine Learning experience is a plus
Claim 13
13. The method
wherein one or more
permutations are
eliminated.
Infringement
Siri ranks candidate parses according to the relative probability the candidate represents user intent, and eliminates
all but the top candidates.
See, e.g.,
- [0421] In 232, language interpreter component( s) 1070 sort and select 232 the top semantic parses as the representation of
user intent 290.
Claim 14
14. The method

further comprising:
providing a
reference dictionary
comprising
Ex. 46 - 40
Infringement
Siri provides a reference dictionary by, for example, providing its candidate parses in the form of an active ontology
and databases referenced by an active ontology.
See, e.g.,
- [0415] - [0417] In semantic matching 220, language interpreter component( s) 1070 consider combinations of possible parse
results according to how well they fit semantic models such as domain models and databases. In this case, the parse
includes the associations (1) "playing" (a word in the user input) as "Local Event At Venue" (part of a domain model 1056
represented by a cluster of nodes in active ontology 1050) and (2) "filmore" (another word in the input) as a match to an entity
name in a domain entity database 1072 for Local Event Venues, which is represented by a domain model element and active
ontology node (Venue Name). Semantic matching 220 may use data from, for example, active ontology 1050, short term
personal memory 1052, and long term personal memory 1054. For example, semantic matching 220 may use data from
previous references to venues or local events in the dialog (from short term personal memory 1052) or personal favorite
venues (from long term personal memory 1054). A set of candidate, or potential, semantic parse results is generated 222.
Claim 14
Ex. 46 - 41
Infringement
Siri's candidate set includes cases, including the user's preferences and past usage of Siri.
See, e.g.,
of the interaction.
cases;
Claim 14
Ex. 46 - 42
Infringement
Claim 14
keywords;
Ex. 46 - 43
Infringement
Siri's candidate set includes keywords, including vocabulary databases of words commonly used to identify
concepts (e.g., cuisine types).
See, e.g.,
embodiment, vocabulary may be associated with any or one or more nodes of active ontologies, domain models, task models,
dialog models, and/or service models.
item (not sortable unless a constraint value, e.g., a keyword, is given 8. keywords (not sortable unless a constraint value, e.g.,
keyword, is given).
Claim 14
information models;
and
Ex. 46 - 44
Infringement
Siri's candidate set includes information models, including Siri's active-ontology models, Siri's language pattern
recognizers, and Siri's language models.
See, e.g.,
combinations thereof): Act as a modeling and development environment, integrating models and data from various model and
data components, including but not limited to Domain models 1056 Vocabulary 1058 Domain entity databases 1072 Task flow
models 1086 Dialog flow models 1087 Service capability models 1088.
- [0251] Domain models 1056, which may constrain the inputs that may instantiate or otherwise be consistent with the domain
model. For example, in at least one embodiment, domain models 1056 may be used to suggest concepts, relations,
components including vocabulary 1058, language pattern recognizers 1060, dialog flow models 1087, task flow models 1086,
service capability models 1088, domain entity databases 1072, and the like. For example, businesses in domain entity
models.
Claim 14
Ex. 46 - 45
Infringement
- Siris knowledge is represented in a unified modeling system that combines ontologies, inference networks, pattern matching
agents, dictionaries, and dialog models. As much as possible we represent things declaratively (i.e., as data in models, not
lines of code). This is a tried and true best practice for complex AI systems. This makes the whole system more robust and
scalable, and the development process more agile. It also helps with reasoning and learning, since Siri can look at what it
knows and think about similarities and generalizations at a semantic level.
Siri's candidate set includes graphics.
graphics, and
See, e.g.,
- [0031] FIG. 8 is a block diagram depicting a fragment of an active ontology according to one embodiment.
- [0420] In at least one embodiment, the connections between nodes in an active ontology that provide evidential support for
disambiguating among candidate semantic parse results 222 are directed arcs, forming an inference lattice, in which
matching nodes provide evidence for nodes to which they are connected by directed arcs.
Claim 14
database values;
and
Ex. 46 - 46
Infringement
Siri's candidate set includes database values, including the database objects in or accessed by Siri's active ontology
as well as user-specific information such as contacts, geospatial location, time, and music list.
See, e.g.,
- Siri also uses information from your contacts, music library, calendars, and reminders to better understand what you say. So
it responds more accurately when you ask to make a phone call, play music, or create an appointment or reminder.
Claim 14
Ex. 46 - 47
Infringement
talking about.
does just that.
to find restaurants by cuisine, price, location, outdoor seating, or a combination of options. Siri can show you available photos,
Yelp stars, price range, and reviews.
Claim 14
identifying, in the
natural language
input, a plurality of
elements that
belong to the
reference dictionary;
Ex. 46 - 48
Infringement
Siri identifies elements that belong to the reference dictionary by, for example, disambiguating the candidate
semantic parse results.
See, e.g.,
- [0415] - [0417] In semantic matching 220, language interpreter component( s) 1070 consider combinations of possible parse
results according to how well they fit semantic models such as domain models and databases. In this case, the parse
includes the associations (1) "playing" (a word in the user input) as "Local Event At Venue" (part of a domain model 1056
represented by a cluster of nodes in active ontology 1050) and (2) "filmore" (another word in the input) as a match to an entity
name in a domain entity database 1072 for Local Event Venues, which is represented by a domain model element and active
ontology node (Venue Name). Semantic matching 220 may use data from, for example, active ontology 1050, short term
personal memory 1052, and long term personal memory 1054. For example, semantic matching 220 may use data from
previous references to venues or local events in the dialog (from short term personal memory 1052) or personal favorite
venues (from long term personal memory 1054). A set of candidate, or potential, semantic parse results is generated 222.
ontology 1050.
Claim 14
determining
complete paths that
are implied by the
plurality of elements
that span elements
of the natural
language input and
which belong to the
graphics of the
reference dictionary.
Claim 15
15. The method

according to claim
14, including
determining a path
that includes
elements of at least
the informational
models and
database values.
Ex. 46 - 49
Infringement
Siri determines complete paths implied by the candidate semantic parse results by, for example, determining the
paths of connections in the active ontology.
See, e.g.,
- [0419] Disambiguation 230 may use data from, for example, the structure of active ontology 1050. In at least one
embodiment, the connections between nodes in an active ontology provide evidential support for disambiguating among
candidate semantic parse results 222. For example, in one embodiment, if three active ontology nodes are semantically
matched and are all connected in active ontology 1050, this indicates higher evidential strength of the semantic parse than if
these matching nodes were not connected or connected by longer paths of connections in active ontology 1050. For example,
in one embodiment of semantic matching 220, the parse that matches both Local Event At Venue and Venue Name is given
increased evidential support because the combined representations of these aspects of the user intent are connected by links
and/or relations in active ontology 1050: in this instance, the Local Event node is connected to the Venue node which is
connected to the Venue Name node which is connected to the entity name in the database of venue names.
Infringement
Siri determines paths that include elements of the information models and database values by, for example,
determining the paths of connections in the active ontology.
See, e.g.,
- [0419] Disambiguation 230 may use data from, for example, the structure of active ontology 1050. In at least one
embodiment, the connections between nodes in an active ontology provide evidential support for disambiguating among
candidate semantic parse results 222. For example, in one embodiment, if three active ontology nodes are semantically
matched and are all connected in active ontology 1050, this indicates higher evidential strength of the semantic parse than if
these matching nodes were not connected or connected by longer paths of connections in active ontology 1050. For example,
in one embodiment of semantic matching 220, the parse that matches both Local Event At Venue and Venue Name is given
increased evidential support because the combined representations of these aspects of the user intent are connected by links
and/or relations in active ontology 1050: in this instance, the Local Event node is connected to the Venue node which is
connected to the Venue Name node which is connected to the entity name in the database of venue names.
Claim 16
16. The method

according to claim
14, further
comprising:
providing rules and
heuristics for
searching; and
determining an
optimum
permutation based
on the rules and
heuristics.
Ex. 46 - 50
Infringement
Siri interprets at least one of the potential candidate parse results according to the relative probability that the
candidate interprets the user's intent. The ordering is determined by the application of rules (such as a priori
precedence ordering) or heuristics (such as minimizing path length between candidate nodes in the active ontology).
See, e.g.,
ontology 1050.
Claim 17
Ex. 46 - 51
Infringement
Siri adds new cases and keywords to a reference dictionary by, for example, learning about the user through user
instruction and usage. Siri also processes natural language input in the context of an ongoing dialog by adding
cases and keywords to the reference dictionary over the course of the dialog.
See, e.g.,
based on immediate context, conversational history, and personal information such as where you live. Siri doesnt forget
17. The method

according to claim
14, further
comprising adding,
as a result of user
input, new cases
and keywords to the
reference dictionary.
- Siri interprets this reply in the context of the existing conversation, using it to refine the user's request.
Claim 17
Ex. 46 - 52
Infringement
of the interaction.
- [0616] - [0623] In at least one embodiment, short term personal memory component(s) 1052 may be operable to perform . . .
Keep a history of the recent dialog between the embodiment of the assistant and the user, including the history of user inputs
and their interpretations . . .
- [0639] Long term personal memory entries may be acquired as a side effect of the user interacting with an embodiment of
assistant 1002. Any kind of interaction with the assistant may produce additions to the long term personal memory, including
browsing, searching, finding, shopping, scheduling, purchasing, reserving, communicating with other people via an assistant.
Ex. 46 - 53
Claim 18
Infringement
Siri determines an optimal interpretation of the natural language input by candidate parse results according to the
relative probability that the candidate interprets the user's intent. The ordering is determined by the application of
rules (such as a priori precedence ordering) or heuristics (such as minimizing path length between candidate nodes
in the active ontology).
18. The method

further comprising
performing a search
of a group of
possible
interpretations to
determine an
optimal
interpretation of the
natural language
input.
See, e.g.,
ontology 1050.
Claim 19
19. The method
further comprising
accepting the
natural language
input, wherein the
action of accepting
the natural language
input accepts a freeformat input.
Infringement
Siri accepts free-format input.
See, e.g.,
No. You can speak to Siri as you would to a person in a natural voice with a conversational tone. If you want to know what
the weather will be like tomorrow, simply say What will the weather be like tomorrow? Or Does it look like rain tomorrow?
Or even Will I need an umbrella tomorrow? No matter how you ask, Siri will tell you the forecast.
Claim 20
20. The method

wherein of
interpretation of
natural language
input is based on
other than the
results of analyzing
syntax of the natural
language query
input.
Claim 21
21. The method
further comprising
representing
graphically elements
of the metadata
database.
Ex. 46 - 54
Infringement
Siri processes the natural language input by generating parse results by other than analyzing the input's syntax. Siri
is not restricted to association of user input with data objects that represent syntactic parts of speech. Siri generates
parse candidates by associating the user input with data objects that represent concepts, relationships, properties,
entities, quantities, propositions, and/or other representations of meaning and user intent.
See, e.g.,
- [0384] - [0387] [Language interpreter may be operable to] [a]nalyze user input and identify a set of parse results. . . . Parse
results are associations of data in the user input with concepts, relationships, properties, instances, and/or other nodes and/or
data structures in models, databases, and/or other representations of user intent and/context. Parse result associations can
be complex mappings from sets and sequences of words, signals, and other elements of user input to one or more associated
concepts, relations, properties, instances, other nodes, and/or data structures described herein.
- [0384] - [0388] [Language interpreter may be operable to] [a]nalyze user input and identify a set of syntactic parse results,
which are parse results that associate data in the user input with structures that represent syntactic parts of speech, clauses
and phrases including multiword names, sentence structure, and/ or other grammatical graph structures.
- [0384] - [0389] [Language interpreter may be operable to] [a]nalyze user input and identify a set of semantic parse results,
which are parse results that associate data in the user input with structures that represent concepts, relationships, properties,
entities, quantities, propositions, and/or other representations of meaning and user intent. In one embodiment, these
representations of meaning and intent are represented by sets of and/or elements of and/or instances of models or databases
and/or nodes in ontologies, as described in element 220 of natural language processing procedure described in connection
with FIG. 28.
Infringement
Siri represents its candidate parses graphically by, for example, representing them in an active ontology.
See, e.g.,
- [0031] FIG. 8 is a block diagram depicting a fragment of an active ontology according to one embodiment.
- [0420] In at least one embodiment, the connections between nodes in an active ontology that provide evidential support for
disambiguating among candidate semantic parse results 222 are directed arcs, forming an inference lattice, in which
matching nodes provide evidence for nodes to which they are connected by directed arcs.

Exhibit 46 - DA&#39;s Infringement Contentions - Detailed

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Exhibit 46 - DA&#39;s Infringement Contentions - Detailed

Transféré par

Droits d'auteur :

Formats disponibles

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 1 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 2 of 55

U.S. Pat. Publ'n No. 2012/0016678 (Exhibit B)

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 3 of 55

U.S. Pat. Publ'n No. 2012/0016678 (Exhibit B)

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 4 of 55

Gruber, Siri, a Virtual Personal Assistant (Exhibit E)

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 5 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 6 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 7 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 8 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 9 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 10 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 11 of 55

database values; and

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 12 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 13 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 14 of 55

identifying, for the

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 15 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 16 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 17 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 18 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 19 of 55

information models; and

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 20 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 21 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 22 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 23 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 24 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 25 of 55

8. The method of claim

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 26 of 55

U.S. Pat. Publ'n No. 2012/0016678 (Exhibit B)

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 27 of 55

U.S. Pat. Publ'n No. 2012/0016678 (Exhibit B)

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 28 of 55

providing from said

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 29 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 30 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 31 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 32 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 33 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 34 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 35 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 36 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 37 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 38 of 55

10. The method

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 39 of 55

U.S. Pat. Publ'n No. 2012/0016678 (Exhibit B)

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 40 of 55

12. The method

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 41 of 55

14. The method

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 42 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 43 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 44 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 45 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 46 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 47 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 48 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 49 of 55

Case 1:13-cv-00633-DEP Document 227-36 Filed 05/21/15 Page 50 of 55

Exhibit 46 - DA's Infringement Contentions - Detailed

Exhibit 46 - DA's Infringement Contentions - Detailed