View source for Metadata Scraper

{{Software Project
|Description=The Creative Commons Metadata Scraper is a simple crawler used by the license engine to detect metadata stored in pages.
|Bug tracker=https://github.com/cc-archive/metadata_scraper/issues/
|Code repository=https://github.com/cc-archive/metadata_scraper
|Mailing list=https://creativecommons.email/mailman/listinfo/cc-devel
}}

== Decommissioned ==

The Creative Commons Metadata Scraper was decommissioned on Monday 8th January 2018. Due to the changing nature of the web, in particular the widespread adoption of HTTPS, the Scraper had seen greatly reduced use. For potential future directions in metadata presentation, see Creative Commons' [https://github.com/creativecommons/open-ledger Open Ledger] project. 

== What it Did ==

The metadata scraper was used by the license deeds to extract embedded metadata from pages.  It scanned the page for [[RDFa]] using [http://pypi.python.org/pypi/rdfadict rdfadict] and [http://www.w3.org/2007/08/pyRdfa/ pyRdfa].

== When it Happened ==

When a creator or copyright holder [http://creativecommons.org/license selects a license], they have the opportunity to provide additional metadata about their work.  This includes information such as creator and medium, as well as how the creator would like to be attributed. If this information is provided, it is encoded in the HTML generated using [[RDFa]].  

In order to provide more context relevant information for visitors, the license deeds loaded a script which looks at the referring page for metadata.  If found, the metadata was used to update the deed and display additional attribution or CC+ details.

== Developer Information ==

Metadata Scraper was implemented in Python using the [http://cherrypy.org CherryPy] web framework.  The source code is available on GitHub at https://github.com/cc-archive/metadata_scraper .

== Contact ==

Contact [[User:CCID-rob|Rob Myers]] with questions about the scraper.

[[Category:Developer]]