Field Query Mapping - Revision history

Paulproteus at 19:24, 30 August 2010

2010-08-30T19:24:50Z

← Older revision		Revision as of 19:24, 30 August 2010
Line 4:		Line 4:
	\|status=In Development		\|status=In Development
	}}		}}
	The people who run a DiscoverEd may wish to let users search specific metadata easily. For example, http://discovered.creativecommons.org~~/search~~/ lets users search for works "tagged" with "banana" by searching for tag:banana. (In particular, the predicate for "tag" is the term "subject" as specified by the Dublin Core.)		The people who run a DiscoverEd may wish to let users search specific metadata easily. For example, http://discovered.labs.creativecommons.org/ lets users search for works "tagged" with "banana" by searching for tag:banana. (In particular, the predicate for "tag" is the term "subject" as specified by the Dublin Core.)

	These prefixes, like tag:, are stored in the DiscoverEd code right now. This feature aims to move those into a configuration file.		These prefixes, like tag:, are stored in the DiscoverEd code right now. This feature aims to move those into a configuration file.
Line 19:		Line 19:

	== How to use ==		== How to use ==

			'''Note''': There is an implementation of this in the current version of DiscoverEd (as of 2010-08-30), but it ignores the ''excludecurator'' argument.

	Let's say you want to allow users to perform this query:		Let's say you want to allow users to perform this query:
Line 58:		Line 60:
	== Next steps ==		== Next steps ==

	* ~~Testing and deployment~~.		* Rewriting this to be compatible with ''excludecurator''.

Paulproteus at 18:18, 24 August 2010

2010-08-24T18:18:21Z

← Older revision		Revision as of 18:18, 24 August 2010
Line 54:		Line 54:
	* Added a sample configuration file		* Added a sample configuration file
	* Added code to our IndexFilter that looks for relevant triples and stores them in the Lucene document		* Added code to our IndexFilter that looks for relevant triples and stores them in the Lucene document
	* ~~Problem: The~~ Lucene ~~documents does not seem to show our column~~, so we~~'re going back to the drawing board and carefully reading~~ the ~~[http://wiki.apache~~.~~org/nutch/HowToMakeCustomSearch relevant Nutch documentation] to make sure we're using the APIs correctly~~		* We had a problem with these columns not appearing in Lucene, but we fixed the underlying bug that caused that.

	== ~~Deferred until later~~ ==		== Next steps ==

	* ~~Handling provenance with regard to this. Based on the current plan for how to handle curator exclusion,~~ and using the above example, we have to make sure that instead of adding merely the column "method", we add something like "curator1:method", "curator2:method", and so on. (This may be out of date; see the spec for Excluding curators.)`		* Testing and deployment.

Dithyramble: /* How to use */

2010-06-29T15:41:32Z

How to use

← Older revision		Revision as of 15:41, 29 June 2010
Line 22:		Line 22:
	Let's say you want to allow users to perform this query:		Let's say you want to allow users to perform this query:

	<blockquote>		<blockquote><pre>
	method:"Experiential learning"		method:"Experiential learning"
	</blockquote>		</pre></blockquote>

	and retrieve all web pages in your index that have a metadatum with predicate <http://purl.org/dc/terms/instructionalMethod> and value "Experiential learning".		and retrieve all web pages in your index that have a metadatum with predicate <http://purl.org/dc/terms/instructionalMethod> and value "Experiential learning".
Line 30:		Line 30:
	To do so, first edit <code>conf/nutch-site.xml</code>. Add this XML inside the <configuration> block.		To do so, first edit <code>conf/nutch-site.xml</code>. Add this XML inside the <configuration> block.

	<blockquote>		<blockquote><pre>
	<property>		<property>
	<name>query.basic.method.boost</name>		<name>query.basic.method.boost</name>
	<value>1.0</value>		<value>1.0</value>
	</property>		</property>
	</blockquote>		</pre></blockquote>

	This block of XML tells Nutch to accept the "method:" prefix in search queries. The value of this property indicates the weight the search engine should assign to this term.		This block of XML tells Nutch to accept the "method:" prefix in search queries. The value of this property indicates the weight the search engine should assign to this term.
Line 41:		Line 41:
	Next, edit <code>conf/discovered-search-prefixes.xml</code>. Add this XML inside the <configuration> block.		Next, edit <code>conf/discovered-search-prefixes.xml</code>. Add this XML inside the <configuration> block.

	<blockquote>		<blockquote><pre>
	<property>		<property>
	<name>http://purl.org/dc/terms/instructionalMethod</name>		<name>http://purl.org/dc/terms/instructionalMethod</name>
	<value>method</value>		<value>method</value>
	</property>		</property>
	</blockquote>		</pre></blockquote>

	This block of XML tells DiscoverEd to copy data out of the Jena store and paste it into a format where Nutch's basic query parser can find it.		This block of XML tells DiscoverEd to copy data out of the Jena store and paste it into a format where Nutch's basic query parser can find it.

Dithyramble: /* How to use */

2010-06-29T15:40:25Z

How to use

← Older revision		Revision as of 15:40, 29 June 2010
Line 22:		Line 22:
	Let's say you want to allow users to perform this query:		Let's say you want to allow users to perform this query:

			<blockquote>
	method:"Experiential learning"		method:"Experiential learning"
			</blockquote>

	and retrieve all web pages in your index that have a metadatum with predicate <http://purl.org/dc/terms/instructionalMethod> and value "Experiential learning".		and retrieve all web pages in your index that have a metadatum with predicate <http://purl.org/dc/terms/instructionalMethod> and value "Experiential learning".
Line 28:		Line 30:
	To do so, first edit <code>conf/nutch-site.xml</code>. Add this XML inside the <configuration> block.		To do so, first edit <code>conf/nutch-site.xml</code>. Add this XML inside the <configuration> block.

			<blockquote>
	<property>		<property>
	<name>query.basic.method.boost</name>		<name>query.basic.method.boost</name>
	<value>1.0</value>		<value>1.0</value>
	</property>		</property>
			</blockquote>

	This block of XML tells Nutch to accept the "method:" prefix in search queries. The value of this property indicates the weight the search engine should assign to this term.		This block of XML tells Nutch to accept the "method:" prefix in search queries. The value of this property indicates the weight the search engine should assign to this term.
Line 37:		Line 41:
	Next, edit <code>conf/discovered-search-prefixes.xml</code>. Add this XML inside the <configuration> block.		Next, edit <code>conf/discovered-search-prefixes.xml</code>. Add this XML inside the <configuration> block.

			<blockquote>
	<property>		<property>
	<name>http://purl.org/dc/terms/instructionalMethod</name>		<name>http://purl.org/dc/terms/instructionalMethod</name>
	<value>method</value>		<value>method</value>
	</property>		</property>
			</blockquote>

	This block of XML tells DiscoverEd to copy data out of the Jena store and paste it into a format where Nutch's basic query parser can find it.		This block of XML tells DiscoverEd to copy data out of the Jena store and paste it into a format where Nutch's basic query parser can find it.

Dithyramble: Explain how to use field query mapping

2010-06-29T15:26:40Z

Explain how to use field query mapping

← Older revision		Revision as of 15:26, 29 June 2010
Line 17:		Line 17:

	Then, at search time, Nutch's built-in query parser handles the query, e.g., "method:yaddayadda".		Then, at search time, Nutch's built-in query parser handles the query, e.g., "method:yaddayadda".

			== How to use ==

			Let's say you want to allow users to perform this query:

			method:"Experiential learning"

			and retrieve all web pages in your index that have a metadatum with predicate <http://purl.org/dc/terms/instructionalMethod> and value "Experiential learning".

			To do so, first edit <code>conf/nutch-site.xml</code>. Add this XML inside the <configuration> block.

			<property>
			<name>query.basic.method.boost</name>
			<value>1.0</value>
			</property>

			This block of XML tells Nutch to accept the "method:" prefix in search queries. The value of this property indicates the weight the search engine should assign to this term.

			Next, edit <code>conf/discovered-search-prefixes.xml</code>. Add this XML inside the <configuration> block.

			<property>
			<name>http://purl.org/dc/terms/instructionalMethod</name>
			<value>method</value>
			</property>

			This block of XML tells DiscoverEd to copy data out of the Jena store and paste it into a format where Nutch's basic query parser can find it.

	== Implementation ==		== Implementation ==

Dithyramble at 15:05, 22 June 2010

2010-06-22T15:05:42Z

← Older revision		Revision as of 15:05, 22 June 2010
Line 12:		Line 12:
	== Requirements ==		== Requirements ==

	When DiscoverEd crawls feeds and resources and saves metadata such as the page title, it converts this information into RDF triples; those triples are eventually saved on disk in a triple store, Jena.		When DiscoverEd crawls feeds and resources and saves metadata such as the page title, it converts this information into RDF triples; those triples are eventually saved on disk in a triple store, namely Jena.

	We will create a new configuration file that stores a list of mappings from predicate URIs ~~(such as stating that~~ "method:" ~~will be~~ a shorthand for the RDF predicate http://purl.org/dc/terms/instructionalMethod, ~~AKA~~ dct:instructionalMethod). At indexing time, a Lucene column called "method" will be created in the Lucene documents corresponding to each resource that has the dct:instructionalMethod predicate set.		We will create a new configuration file that stores a list of mappings from predicate URIs. For example, we might list "method:" as a shorthand for the RDF predicate <http://purl.org/dc/terms/instructionalMethod>, a.k.a. "dct:instructionalMethod". At indexing time, a Lucene column called "method" will be created in the Lucene documents corresponding to each resource that has the dct:instructionalMethod predicate set in the Jena store.

	Then, at search time, Nutch's built-in query parser handles the query.		Then, at search time, Nutch's built-in query parser handles the query, e.g., "method:yaddayadda".

	== Implementation ==		== Implementation ==
Line 24:		Line 24:
	* Problem: The Lucene documents does not seem to show our column, so we're going back to the drawing board and carefully reading the [http://wiki.apache.org/nutch/HowToMakeCustomSearch relevant Nutch documentation] to make sure we're using the APIs correctly		* Problem: The Lucene documents does not seem to show our column, so we're going back to the drawing board and carefully reading the [http://wiki.apache.org/nutch/HowToMakeCustomSearch relevant Nutch documentation] to make sure we're using the APIs correctly

	==Deferred until later==		== Deferred until later ==

	* Handling provenance with regard to this.		* Handling provenance with regard to this. Based on the current plan for how to handle curator exclusion, and using the above example, we have to make sure that instead of adding merely the column "method", we add something like "curator1:method", "curator2:method", and so on. (This may be out of date; see the spec for Excluding curators.)`

Dithyramble: disambig "subject"

2010-06-22T14:58:47Z

disambig "subject"

← Older revision		Revision as of 14:58, 22 June 2010
Line 4:		Line 4:
	\|status=In Development		\|status=In Development
	}}		}}
	The people who run a DiscoverEd may wish to let users search specific metadata easily. For example, http://discovered.creativecommons.org/search/ lets users search for works "tagged" with "banana" ~~as a Dublin Core subject~~ by searching for tag:banana.		The people who run a DiscoverEd may wish to let users search specific metadata easily. For example, http://discovered.creativecommons.org/search/ lets users search for works "tagged" with "banana" by searching for tag:banana. (In particular, the predicate for "tag" is the term "subject" as specified by the Dublin Core.)

	These prefixes, like tag:, are stored in the DiscoverEd code right now. This feature aims to move those into a configuration file.		These prefixes, like tag:, are stored in the DiscoverEd code right now. This feature aims to move those into a configuration file.

Paulproteus: /* Requirements */

2010-06-21T20:06:36Z

Requirements

← Older revision		Revision as of 20:06, 21 June 2010
Line 12:		Line 12:
	== Requirements ==		== Requirements ==

	(~~still writing~~)~~he+%5B%5BDiscove~~		When DiscoverEd crawls feeds and resources and saves metadata such as the page title, it converts this information into RDF triples; those triples are eventually saved on disk in a triple store, Jena.

			We will create a new configuration file that stores a list of mappings from predicate URIs (such as stating that "method:" will be a shorthand for the RDF predicate http://purl.org/dc/terms/instructionalMethod, AKA dct:instructionalMethod). At indexing time, a Lucene column called "method" will be created in the Lucene documents corresponding to each resource that has the dct:instructionalMethod predicate set.

			Then, at search time, Nutch's built-in query parser handles the query.

			== Implementation ==

			* Added a sample configuration file
			* Added code to our IndexFilter that looks for relevant triples and stores them in the Lucene document
			* Problem: The Lucene documents does not seem to show our column, so we're going back to the drawing board and carefully reading the [http://wiki.apache.org/nutch/HowToMakeCustomSearch relevant Nutch documentation] to make sure we're using the APIs correctly

			==Deferred until later==

			* Handling provenance with regard to this.

Paulproteus: Created page with '{{DiscoverEd Specification |contact=Asheesh Laroia |project=AgShare |status=In Development }} The people who run a DiscoverEd may wish to let users search specific metadata easil…'

2010-06-21T18:49:08Z

Created page with '{{DiscoverEd Specification |contact=Asheesh Laroia |project=AgShare |status=In Development }} The people who run a DiscoverEd may wish to let users search specific metadata easil…'

New page

{{DiscoverEd Specification
|contact=Asheesh Laroia
|project=AgShare
|status=In Development
}}
The people who run a DiscoverEd may wish to let users search specific metadata easily. For example, http://discovered.creativecommons.org/search/ lets users search for works "tagged" with "banana" as a Dublin Core subject by searching for tag:banana.

These prefixes, like tag:, are stored in the DiscoverEd code right now. This feature aims to move those into a configuration file.

This feature was defined and developed during the [[DiscoverEd Sprint (June, 2010)|June 2010 DiscoverEd Sprint]]

== Requirements ==

(still writing)he+%5B%5BDiscove