Class MetadataScoringFilter
- java.lang.Object
-
- org.apache.nutch.scoring.AbstractScoringFilter
-
- org.apache.nutch.scoring.metadata.MetadataScoringFilter
-
- All Implemented Interfaces:
Configurable
,Pluggable
,ScoringFilter
public class MetadataScoringFilter extends AbstractScoringFilter
For documentation:org.apache.nutch.scoring.metadata
-
-
Field Summary
Fields Modifier and Type Field Description static String
METADATA_CONTENT
static String
METADATA_DATUM
static String
METADATA_PARSED
-
Fields inherited from interface org.apache.nutch.scoring.ScoringFilter
X_POINT_ID
-
-
Constructor Summary
Constructors Constructor Description MetadataScoringFilter()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description CrawlDatum
distributeScoreToOutlinks(Text fromUrl, ParseData parseData, Collection<Map.Entry<Text,CrawlDatum>> targets, CrawlDatum adjust, int allCount)
This will take the metadata that you have listed in your "scoring.parse.md" property, and looks for them inside the parseData object.void
passScoreAfterParsing(Text url, Content content, Parse parse)
Takes the metadata, which was lumped inside the content, and replicates it within your parse data.void
passScoreBeforeParsing(Text url, CrawlDatum datum, Content content)
Takes the metadata, specified in your "scoring.db.md" property, from the datum object and injects it into the content.void
setConf(Configuration conf)
handles conf assignment and pulls the value assignment from the "scoring.db.md", "scoring.content.md" and "scoring.parse.md" properties.-
Methods inherited from class org.apache.nutch.scoring.AbstractScoringFilter
generatorSortValue, getConf, indexerScore, initialScore, injectedScore, updateDbScore
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.nutch.scoring.ScoringFilter
orphanedScore
-
-
-
-
Field Detail
-
METADATA_DATUM
public static final String METADATA_DATUM
- See Also:
- Constant Field Values
-
METADATA_CONTENT
public static final String METADATA_CONTENT
- See Also:
- Constant Field Values
-
METADATA_PARSED
public static final String METADATA_PARSED
- See Also:
- Constant Field Values
-
-
Method Detail
-
distributeScoreToOutlinks
public CrawlDatum distributeScoreToOutlinks(Text fromUrl, ParseData parseData, Collection<Map.Entry<Text,CrawlDatum>> targets, CrawlDatum adjust, int allCount) throws ScoringFilterException
This will take the metadata that you have listed in your "scoring.parse.md" property, and looks for them inside the parseData object. If they exist, this will be propagated into your 'targets' Collection's ["outlinks"] attributes.- Specified by:
distributeScoreToOutlinks
in interfaceScoringFilter
- Overrides:
distributeScoreToOutlinks
in classAbstractScoringFilter
- Parameters:
fromUrl
- url of the source pageparseData
- ParseData instance, which stores relevant score value(s) in its metadata. NOTE: filters may modify this in-place, all changes will be persisted.targets
- <url, CrawlDatum> pairs. NOTE: filters can modify this in-place, all changes will be persisted.adjust
- a CrawlDatum instance, initially null, which implementations may use to pass adjustment values to the original CrawlDatum. When creating this instance, set its status toCrawlDatum.STATUS_LINKED
.allCount
- number of all collected outlinks from the source page- Returns:
- if needed, implementations may return an instance of CrawlDatum,
with status
CrawlDatum.STATUS_LINKED
, which contains adjustments to be applied to the original CrawlDatum score(s) and metadata. This can be null if not needed. - Throws:
ScoringFilterException
- there is a fatal error distributing score data from the current page to all of its outlinks- See Also:
ScoringFilter.distributeScoreToOutlinks(org.apache.hadoop.io.Text, org.apache.nutch.parse.ParseData, java.util.Collection<java.util.Map.Entry<org.apache.hadoop.io.Text, org.apache.nutch.crawl.CrawlDatum>>, org.apache.nutch.crawl.CrawlDatum, int)
-
passScoreBeforeParsing
public void passScoreBeforeParsing(Text url, CrawlDatum datum, Content content)
Takes the metadata, specified in your "scoring.db.md" property, from the datum object and injects it into the content. This is transfered to the parseData object.- Specified by:
passScoreBeforeParsing
in interfaceScoringFilter
- Overrides:
passScoreBeforeParsing
in classAbstractScoringFilter
- Parameters:
url
- url of the pagedatum
- source datum. NOTE: modifications to this value are not persisted.content
- instance of content. Implementations may modify this in-place, primarily by setting some metadata properties.- See Also:
ScoringFilter.passScoreBeforeParsing(org.apache.hadoop.io.Text, org.apache.nutch.crawl.CrawlDatum, org.apache.nutch.protocol.Content)
,passScoreAfterParsing(org.apache.hadoop.io.Text, org.apache.nutch.protocol.Content, org.apache.nutch.parse.Parse)
-
passScoreAfterParsing
public void passScoreAfterParsing(Text url, Content content, Parse parse)
Takes the metadata, which was lumped inside the content, and replicates it within your parse data.- Specified by:
passScoreAfterParsing
in interfaceScoringFilter
- Overrides:
passScoreAfterParsing
in classAbstractScoringFilter
- Parameters:
url
- page urlcontent
- original content. NOTE: modifications to this value are not persisted.parse
- target instance to copy the score information to. Implementations may modify this in-place, primarily by setting some metadata properties.- See Also:
passScoreBeforeParsing(org.apache.hadoop.io.Text, org.apache.nutch.crawl.CrawlDatum, org.apache.nutch.protocol.Content)
,ScoringFilter.passScoreAfterParsing(org.apache.hadoop.io.Text, org.apache.nutch.protocol.Content, org.apache.nutch.parse.Parse)
-
setConf
public void setConf(Configuration conf)
handles conf assignment and pulls the value assignment from the "scoring.db.md", "scoring.content.md" and "scoring.parse.md" properties.- Specified by:
setConf
in interfaceConfigurable
- Overrides:
setConf
in classAbstractScoringFilter
-
-