CoOccurrence

Class CoOccurrenceMatrix

java.lang.Object

java.util.AbstractMap<CoOccurrence,CoOccurrence>

CoOccurrence.CoOccurrenceMatrix

All Implemented Interfaces:

java.util.Map<CoOccurrence,CoOccurrence>

public class CoOccurrenceMatrix
extends java.util.AbstractMap<CoOccurrence,CoOccurrence>
implements java.util.Map<CoOccurrence,CoOccurrence>

Author:

Nina Tahmasebi This class represents the co-occurrence matrix that is built based on a Dictionary and a collection of documents. The matrix is represented as a HashMap where all CoOccurrence objects found so far in the collection are stored.

Associated with a CoOccurrenceMatrix is also an index. For each Term present in the matrix, the index contains an ArrayList of all co-occurrence where the term takes part. The index is synchronized with the matrix. Adding/removing a CoOccurrence to/from the matrix will result in an updated index.

The co-occurrence matrix can also be seen as a graph where terms in the matrix represent the nodes and the co-occurrences represent the edges. The index can be considered to be an Adjacency list (wiki-link) , Nina Tahmasebi, L3S Research Center

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	CoOccurrenceMatrix.CoOccComparatorFrq A comparator for CoOccurrence so that they can be sorted on their frequencies.

Nested classes/interfaces inherited from class java.util.AbstractMap

java.util.AbstractMap.SimpleEntry<K,V>, java.util.AbstractMap.SimpleImmutableEntry<K,V>

Nested classes/interfaces inherited from interface java.util.Map

java.util.Map.Entry<K,V>

Constructor Summary

Constructors

Constructor and Description
CoOccurrenceMatrix(Dictionary dictionary) Constructs an empty CoOccurrenceMatrix where the dictionary is set and the index is empty.

Method Summary

Methods

Modifier and Type	Method and Description
void	clear()
boolean	compareIndex(java.util.HashMap<Term,java.util.ArrayList<CoOccurrence>> index)
void	createXMLFile(java.lang.String pathToXMLFile)
java.util.ArrayList<java.lang.Integer>	degreeDistribution() Returns the degree distribution of this matrix.
java.util.Set	entrySet() Returns the entry set of this matrix.
boolean	equals(java.lang.Object ma) Compares two matrices.
CoOccurrenceMatrix	extractLimitedPartOfMatrix(int degree, int limit, java.util.ArrayList<java.lang.String> seed)
CoOccurrenceMatrix	extractPartOfMatrix(int degree, java.util.ArrayList<java.lang.String> seed) Starting from a seed set extractPartOfMatrix iteratively extracts a sub-matrix.
Dictionary	getDictionary() Returns the dictionary used for building this matrix
java.util.HashMap<Term,java.util.ArrayList<CoOccurrence>>	getIndex() Returns index representing the incidence matrix.
java.lang.String	getPeriod()
java.util.ArrayList<java.lang.String>	getTopTerms(int k, java.lang.String query) Method for returning the top k terms that co-occur with query term in the co-occurrence matrix.
java.util.Iterator<CoOccurrence>	iterator() Returns an iterator over all edges in the graph represented by the matrix.
java.util.Iterator<Term>	iteratorTerms() Returns an iterator over all nodes in the graph represented by the matrix.
CoOccurrenceMatrix	mergeMatrix(CoOccurrenceMatrix inputMatrix) Merges this matrix with inputMatrix and removes the duplicates.
void	normalizeMatrix()
CoOccurrence	put(CoOccurrence e, CoOccurrence d) The parameters e and d should be the same.
void	remove(CoOccurrence co)
boolean	removeCoOccur(CoOccurrence co, boolean removeLeaf)
void	removeLeafNodes() Removes all leaf nodes from this matrix, i.e.
void	removeLowFrequencyOccur(double i) Removes all CoOccurrence with a co-occurrence frequency less than or equal to i.
void	setDictionary(Dictionary dictionary) Changes dictionary for this matrix.
void	setIndex(java.util.HashMap<Term,java.util.ArrayList<CoOccurrence>> index)
void	setPeriod(java.lang.String period)
int	size() Returns the number of elements in this matrix
java.util.List<CoOccurrence>	sort() Method for sorting Matrix.
static java.util.List<CoOccurrence>	sortCoOccrCollection(java.util.Collection<CoOccurrence> keySet, java.util.Comparator<CoOccurrence> comp) Method for sorting a a Collection of CoOccurrence.
void	updateMatrix(CoOccurrence co) Increases the frequency for the co-occurrence co by one.
void	writeMatrixForClustering(double i, java.io.Writer writer) Output of the file is adapted to Beate Dorow's Clustering Algorithm.
void	writeMatrixForSynonyms(double i, java.io.Writer writer) Output of the file is adapted to Beate Dorow's Clustering Algorithm.
void	writeMatrixToFile(double i, java.io.Writer writer) Writes all occurrences of matrix that is larger that i to writer.

Methods inherited from class java.util.AbstractMap

containsKey, containsValue, get, hashCode, isEmpty, keySet, putAll, remove, toString, values

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.Map

containsKey, containsValue, get, hashCode, isEmpty, keySet, putAll, remove, values

Constructor Detail

CoOccurrenceMatrix

public CoOccurrenceMatrix(Dictionary dictionary)

Constructs an empty CoOccurrenceMatrix where the dictionary is set and the index is empty.

Parameters:

dictionary - the dictionary associated with the matrix

Method Detail

clear

public void clear()

Specified by:

clear in interface java.util.Map<CoOccurrence,CoOccurrence>

Overrides:

clear in class java.util.AbstractMap<CoOccurrence,CoOccurrence>

compareIndex

public boolean compareIndex(java.util.HashMap<Term,java.util.ArrayList<CoOccurrence>> index)

entrySet

public java.util.Set entrySet()

Returns the entry set of this matrix.

Specified by:

entrySet in interface java.util.Map<CoOccurrence,CoOccurrence>

Specified by:

entrySet in class java.util.AbstractMap<CoOccurrence,CoOccurrence>

Returns:

The entry set of the CoOccurrenceMatrix

setPeriod

public void setPeriod(java.lang.String period)

getPeriod

public java.lang.String getPeriod()

remove

public void remove(CoOccurrence co)

equals

public boolean equals(java.lang.Object ma)

Compares two matrices. The matrices are considered equal if they contain the same co-occurrences with the same frequencies and their sizes are equal.

Specified by:

equals in interface java.util.Map<CoOccurrence,CoOccurrence>

Overrides:

equals in class java.util.AbstractMap<CoOccurrence,CoOccurrence>

extractPartOfMatrix

public CoOccurrenceMatrix extractPartOfMatrix(int degree,
                                     java.util.ArrayList<java.lang.String> seed)

Starting from a seed set extractPartOfMatrix iteratively extracts a sub-matrix. The CoOccurrence objects in the resulting matrix is at a distance less than or equal to degree from a node in seed. degree = 0 results in a sub-matrix based only on the seed set.

Parameters:

degree - determines the maximum distance that any CoOccurrence in the resulting sub-matrix will have from the nodes in seed

seed - is a list of starting nodes in the matrix

Returns:

A co-occurrence matrix representing a sub-matrix based on the seed list.

extractLimitedPartOfMatrix

public CoOccurrenceMatrix extractLimitedPartOfMatrix(int degree,
                                            int limit,
                                            java.util.ArrayList<java.lang.String> seed)

degreeDistribution

public java.util.ArrayList<java.lang.Integer> degreeDistribution()

Returns the degree distribution of this matrix. An array with the nr of co-occurrences for every degree

Returns:

list of integers representing the degree distribution of the matrix

getDictionary

public Dictionary getDictionary()

Returns the dictionary used for building this matrix

Returns:

The dictionary that this matrix is based on.

getIndex

public java.util.HashMap<Term,java.util.ArrayList<CoOccurrence>> getIndex()

Returns index representing the incidence matrix.

Returns:

The index that represents the matrix.

iterator

public java.util.Iterator<CoOccurrence> iterator()

Returns an iterator over all edges in the graph represented by the matrix.

Returns:

An iterator over all CoOccurrences present in the matrix

iteratorTerms

public java.util.Iterator<Term> iteratorTerms()

Returns an iterator over all nodes in the graph represented by the matrix.

Returns:

An iterator over all terms present in the matrix,

put

public CoOccurrence put(CoOccurrence e,
               CoOccurrence d)

The parameters e and d should be the same. Adds a CoOccurrence to the matrix, the index is automatically updated.

Specified by:

put in interface java.util.Map<CoOccurrence,CoOccurrence>

Overrides:

put in class java.util.AbstractMap<CoOccurrence,CoOccurrence>

removeCoOccur

public boolean removeCoOccur(CoOccurrence co,
                    boolean removeLeaf)

removeLeafNodes

public void removeLeafNodes()

Removes all leaf nodes from this matrix, i.e. all terms that only co-occur with one other term. index is automatically updated.

    a                         a 

   / \               =>      / \ 

  b - c - d - e             b - c 

removeLowFrequencyOccur

public void removeLowFrequencyOccur(double i)

Removes all CoOccurrence with a co-occurrence frequency less than or equal to i. Runs in O(N) where N = #CoOccurrence in this matrix. index is not synchronized with matrix. The items that should be removed from index are represented with null.

Parameters:

i - the minimum number of occurrences that the elements in the remaining matrix must have

setDictionary

public void setDictionary(Dictionary dictionary)

Changes dictionary for this matrix.

Parameters:

dictionary - the dictionary to use h

size

public int size()

Returns the number of elements in this matrix

Specified by:

size in interface java.util.Map<CoOccurrence,CoOccurrence>

Overrides:

size in class java.util.AbstractMap<CoOccurrence,CoOccurrence>

Returns:

The number of CoOccurrences in this matrix.

updateMatrix

public void updateMatrix(CoOccurrence co)

Increases the frequency for the co-occurrence co by one.

Parameters:

co - the co-occurrence to update

writeMatrixForClustering

public void writeMatrixForClustering(double i,
                            java.io.Writer writer)

Output of the file is adapted to Beate Dorow's Clustering Algorithm. Each line will be in the form: term term #coOccurrences normalized coOccurrence , where the items are tab-separated.

Parameters:

i - the threshold to be used

writer - specifies where the output should be written

sortCoOccrCollection

public static java.util.List<CoOccurrence> sortCoOccrCollection(java.util.Collection<CoOccurrence> keySet,
                                                java.util.Comparator<CoOccurrence> comp)

Method for sorting a a Collection of CoOccurrence. This method uses the inherent comparing of CoOccurrence with a specific comparator Unit. The comparator could specify sorting units based on number of internal clusters, or longest spanning units etc. Note that the units need not to be unique!

Parameters:

keySet - Set of CoOccurrence to be sorted.

comp - Comparator to be used for sorting CoOccurrence.

Returns:

list of CoOccurrences sorted.

getTopTerms

public java.util.ArrayList<java.lang.String> getTopTerms(int k,
                                                java.lang.String query)

Method for returning the top k terms that co-occur with query term in the co-occurrence matrix.

Parameters:

k - - the top number of terms to return

query - - the query term with which the top terms should co-occur

Returns:

a list of top k terms that co-occur with query term

sort

public java.util.List<CoOccurrence> sort()

Method for sorting Matrix.

writeMatrixForSynonyms

public void writeMatrixForSynonyms(double i,
                          java.io.Writer writer)

Output of the file is adapted to Beate Dorow's Clustering Algorithm. Each line will be in the form: term term #coOccurrences normalized coOccurrence , where the items are tab-separated.

Parameters:

i - the threshold to be used

writer - specifies where the output should be written

writeMatrixToFile

public void writeMatrixToFile(double i,
                     java.io.Writer writer)

Writes all occurrences of matrix that is larger that i to writer.

Parameters:

i - the threshold to be used

writer - specifies where the output should be written

normalizeMatrix

public void normalizeMatrix()

mergeMatrix

public CoOccurrenceMatrix mergeMatrix(CoOccurrenceMatrix inputMatrix)
                               throws java.lang.CloneNotSupportedException

Merges this matrix with inputMatrix and removes the duplicates. The frequency of a CoOccurrences<\tt> that exists in both inputMatrix and this<\tt> it the sum of the frequencies of the two merged tt>CoOccurrences<\tt>. The output is a new CoOccurrenceMatrix with the merged matrix.

Parameters:

inputMatrix - the matrix which should be merged

Returns:

Merged matrix where all duplicates are removed

Throws:

java.lang.CloneNotSupportedException

createXMLFile

public void createXMLFile(java.lang.String pathToXMLFile)
                   throws java.io.IOException

Throws:

java.io.IOException

setIndex

public void setIndex(java.util.HashMap<Term,java.util.ArrayList<CoOccurrence>> index)