SequenceFile.Sorter (Apache Hadoop Common 3.5.0 API)
Description: declaration: package: org.apache.hadoop.io, class: SequenceFile, class: Sorter

Package org.apache.hadoop.io

# Class SequenceFile.Sorter

java.lang.Object
org.apache.hadoop.io.SequenceFile.Sorter

Enclosing class:
  SequenceFile

public static class SequenceFile.Sorter extends Object
Sorts key/value pairs in a sequence-format file.

For best performance, applications should make sure that the `Writable.readFields(DataInput)` implementation of their keys is very efficient. In particular, it should avoid allocating memory.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static interface`
  `SequenceFile.Sorter.RawKeyValueIterator`
  The interface to iterate over raw keys/values of SequenceFiles.
  `class`
  `SequenceFile.Sorter.SegmentDescriptor`
  This class defines a merge segment.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `Sorter(FileSystem fs, Class<? extends WritableComparable> keyClass, Class valClass, Configuration conf)`
  Sort and merge files containing the named classes.
  `Sorter(FileSystem fs, RawComparator comparator, Class keyClass, Class valClass, Configuration conf)`
  Sort and merge using an arbitrary `RawComparator`.
  `Sorter(FileSystem fs, RawComparator comparator, Class keyClass, Class valClass, Configuration conf, SequenceFile.Metadata metadata)`
  Sort and merge using an arbitrary `RawComparator`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `SequenceFile.Writer`
  `cloneFileAttributes(Path inputFile, Path outputFile, Progressable prog)`
  Clones the attributes (like compression of the input file and creates a corresponding Writer
  `int`
  `getFactor()`
  `int`
  `getMemory()`
  `SequenceFile.Sorter.RawKeyValueIterator`
  `merge(List<SequenceFile.Sorter.SegmentDescriptor> segments, Path tmpDir)`
  Merges the list of segments of type `SegmentDescriptor`
  `SequenceFile.Sorter.RawKeyValueIterator`
  `merge(Path[] inNames, boolean deleteInputs, int factor, Path tmpDir)`
  Merges the contents of files passed in Path[]
  `SequenceFile.Sorter.RawKeyValueIterator`
  `merge(Path[] inNames, boolean deleteInputs, Path tmpDir)`
  Merges the contents of files passed in Path[] using a max factor value that is already set
  `void`
  `merge(Path[] inFiles, Path outFile)`
  Merge the provided files.
  `SequenceFile.Sorter.RawKeyValueIterator`
  `merge(Path[] inNames, Path tempDir, boolean deleteInputs)`
  Merges the contents of files passed in Path[]
  `void`
  `setFactor(int factor)`
  Set the number of streams to merge at once.
  `void`
  `setMemory(int memory)`
  Set the total amount of buffer memory, in bytes.
  `void`
  `setProgressable(Progressable progressable)`
  Set the progressable object in order to report progress.
  `void`
  `sort(Path[] inFiles, Path outFile, boolean deleteInput)`
  Perform a file sort from a set of input files into an output file.
  `void`
  `sort(Path inFile, Path outFile)`
  The backwards compatible interface to sort.
  `SequenceFile.Sorter.RawKeyValueIterator`
  `sortAndIterate(Path[] inFiles, Path tempDir, boolean deleteInput)`
  Perform a file sort from a set of input files and return an iterator.
  `void`
  `writeFile(SequenceFile.Sorter.RawKeyValueIterator records, SequenceFile.Writer writer)`
  Writes records from RawKeyValueIterator into a file represented by the passed writer.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### Sorter

    public Sorter(FileSystem fs, Class<? extends WritableComparable> keyClass, Class valClass, Configuration conf)
    Sort and merge files containing the named classes.
    Parameters:
      `fs` - input FileSystem.
      `keyClass` - input keyClass.
      `valClass` - input valClass.
      `conf` - input Configuration.

  - ### Sorter

    public Sorter(FileSystem fs, RawComparator comparator, Class keyClass, Class valClass, Configuration conf)
    Sort and merge using an arbitrary `RawComparator`.
    Parameters:
      `fs` - input FileSystem.
      `comparator` - input RawComparator.
      `keyClass` - input keyClass.
      `valClass` - input valClass.
      `conf` - input Configuration.

  - ### Sorter

    public Sorter(FileSystem fs, RawComparator comparator, Class keyClass, Class valClass, Configuration conf, SequenceFile.Metadata metadata)
    Sort and merge using an arbitrary `RawComparator`.
    Parameters:
      `fs` - input FileSystem.
      `comparator` - input RawComparator.
      `keyClass` - input keyClass.
      `valClass` - input valClass.
      `conf` - input Configuration.
      `metadata` - input metadata.

- ## Method Details

  - ### setFactor

    public void setFactor(int factor)
    Set the number of streams to merge at once.
    Parameters:
      `factor` - factor.

  - ### getFactor

    public int getFactor()
    Returns:
      Get the number of streams to merge at once.

  - ### setMemory

    public void setMemory(int memory)
    Set the total amount of buffer memory, in bytes.
    Parameters:
      `memory` - buffer memory.

  - ### getMemory

    public int getMemory()
    Returns:
      Get the total amount of buffer memory, in bytes.

  - ### setProgressable

    public void setProgressable(Progressable progressable)
    Set the progressable object in order to report progress.
    Parameters:
      `progressable` - input Progressable.

  - ### sort

    public void sort(Path[] inFiles, Path outFile, boolean deleteInput) throws IOException
    Perform a file sort from a set of input files into an output file.
    Parameters:
      `inFiles` - the files to be sorted
      `outFile` - the sorted output file
      `deleteInput` - should the input files be deleted as they are read?
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### sortAndIterate

    public SequenceFile.Sorter.RawKeyValueIterator sortAndIterate(Path[] inFiles, Path tempDir, boolean deleteInput) throws IOException
    Perform a file sort from a set of input files and return an iterator.
    Parameters:
      `inFiles` - the files to be sorted
      `tempDir` - the directory where temp files are created during sort
      `deleteInput` - should the input files be deleted as they are read?
    Returns:
      iterator the RawKeyValueIterator
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### sort

    public void sort(Path inFile, Path outFile) throws IOException
    The backwards compatible interface to sort.
    Parameters:
      `inFile` - the input file to sort.
      `outFile` - the sorted output file.
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### merge

    public SequenceFile.Sorter.RawKeyValueIterator merge(List<SequenceFile.Sorter.SegmentDescriptor> segments, Path tmpDir) throws IOException
    Merges the list of segments of type `SegmentDescriptor`
    Parameters:
      `segments` - the list of SegmentDescriptors
      `tmpDir` - the directory to write temporary files into
    Returns:
      RawKeyValueIterator
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### merge

    public SequenceFile.Sorter.RawKeyValueIterator merge(Path[] inNames, boolean deleteInputs, Path tmpDir) throws IOException
    Merges the contents of files passed in Path[] using a max factor value that is already set
    Parameters:
      `inNames` - the array of path names
      `deleteInputs` - true if the input files should be deleted when unnecessary
      `tmpDir` - the directory to write temporary files into
    Returns:
      RawKeyValueIteratorMergeQueue
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### merge

    public SequenceFile.Sorter.RawKeyValueIterator merge(Path[] inNames, boolean deleteInputs, int factor, Path tmpDir) throws IOException
    Merges the contents of files passed in Path[]
    Parameters:
      `inNames` - the array of path names
      `deleteInputs` - true if the input files should be deleted when unnecessary
      `factor` - the factor that will be used as the maximum merge fan-in
      `tmpDir` - the directory to write temporary files into
    Returns:
      RawKeyValueIteratorMergeQueue
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### merge

    public SequenceFile.Sorter.RawKeyValueIterator merge(Path[] inNames, Path tempDir, boolean deleteInputs) throws IOException
    Merges the contents of files passed in Path[]
    Parameters:
      `inNames` - the array of path names
      `tempDir` - the directory for creating temp files during merge
      `deleteInputs` - true if the input files should be deleted when unnecessary
    Returns:
      RawKeyValueIteratorMergeQueue
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### cloneFileAttributes

    public SequenceFile.Writer cloneFileAttributes(Path inputFile, Path outputFile, Progressable prog) throws IOException
    Clones the attributes (like compression of the input file and creates a corresponding Writer
    Parameters:
      `inputFile` - the path of the input file whose attributes should be cloned
      `outputFile` - the path of the output file
      `prog` - the Progressable to report status during the file write
    Returns:
      Writer
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### writeFile

    public void writeFile(SequenceFile.Sorter.RawKeyValueIterator records, SequenceFile.Writer writer) throws IOException
    Writes records from RawKeyValueIterator into a file represented by the passed writer.
    Parameters:
      `records` - the RawKeyValueIterator
      `writer` - the Writer created earlier
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### merge

    public void merge(Path[] inFiles, Path outFile) throws IOException
    Merge the provided files.
    Parameters:
      `inFiles` - the array of input path names
      `outFile` - the final output file
    Throws:
      `IOException` - raised on errors performing I/O.