CopyCommandWithMultiThread (Apache Hadoop Common 3.5.0 API)
Description: declaration: package: org.apache.hadoop.fs.shell, class: CopyCommandWithMultiThread

Package org.apache.hadoop.fs.shell

# Class CopyCommandWithMultiThread

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.hadoop.fs.shell.Command
org.apache.hadoop.fs.shell.FsCommand
org.apache.hadoop.fs.shell.CopyCommandWithMultiThread

All Implemented Interfaces:
  `Configurable`

public abstract class CopyCommandWithMultiThread extends FsCommand
Abstract command to enable sub copy commands run with multi-thread.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final int`
  `DEFAULT_QUEUE_SIZE`
  `protected PathData`
  `dst`
  `protected static final org.slf4j.Logger`
  `LOG`

  ### Fields inherited from class org.apache.hadoop.fs.shell.Command

  `args, COMMAND_DESCRIPTION_FIELD, COMMAND_NAME_FIELD, COMMAND_USAGE_FIELD, err, exceptions, exitCode, name, numErrors, out, recursive`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `CopyCommandWithMultiThread()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `protected void`
  `copyFileToTarget(PathData src, PathData target)`
  Copies the source file to the target.
  `protected void`
  `copyStreamToTarget(InputStream in, PathData target)`
  If direct write is disabled ,copies the stream contents to a temporary file "target.
  `protected ThreadPoolExecutor`
  `getExecutor()`
  `protected void`
  `getLocalDestination(LinkedList<String> args)`
  The last arg is expected to be a local path, if only one argument is given then the destination will be the current directory
  `protected void`
  `getRemoteDestination(LinkedList<String> args)`
  The last arg is expected to be a remote path, if only one argument is given then the destination will be the remote user's directory
  `protected PathData`
  `getTargetPath(PathData src)`
  `protected int`
  `getThreadCount()`
  `protected int`
  `getThreadPoolQueueSize()`
  `protected boolean`
  `isMultiThreadNecessary(LinkedList<PathData> args)`
  `protected void`
  `preserve(org.apache.hadoop.fs.shell.CommandWithDestination.FileAttribute fileAttribute)`
  Add file attributes that need to be preserved.
  `protected void`
  `preserveAttributes(PathData src, PathData target, boolean preserveRawXAttrs)`
  Preserve the attributes of the source to the target.
  `protected void`
  `processArguments(LinkedList<PathData> args)`
  Processes the command's list of expanded arguments.
  `protected void`
  `processPath(PathData src)`
  Hook for commands to implement an operation to be applied on each path for the command.
  `protected void`
  `processPath(PathData src, PathData dst)`
  Called with a source and target destination pair
  `protected void`
  `processPathArgument(PathData src)`
  This is the last chance to modify an argument before going into the (possibly) recursive `Command.processPaths(PathData, PathData...)` -> `Command.processPath(PathData)` loop.
  `protected void`
  `recursePath(PathData src)`
  Gets the directory listing for a path and invokes `Command.processPaths(PathData, PathData...)`
  `protected void`
  `setDirectWrite(boolean flag)`
  `protected void`
  `setLazyPersist(boolean flag)`
  `protected void`
  `setOverwrite(boolean flag)`
  This method is used to enable the force(-f) option while copying the files.
  `protected void`
  `setPreserve(boolean preserve)`
  If true, the last modified time, last access time, owner, group and permission information of the source file will be preserved as far as target `FileSystem` implementation allows.
  `protected void`
  `setThreadCount(String optValue)`
  set thread count by option value, if the value less than 1, use 1 instead.
  `protected void`
  `setThreadPoolQueueSize(String optValue)`
  set thread pool queue size by option value, if the value less than 1, use DEFAULT_QUEUE_SIZE instead.
  `protected void`
  `setVerifyChecksum(boolean flag)`
  `protected void`
  `setWriteChecksum(boolean flag)`

  ### Methods inherited from class org.apache.hadoop.fs.shell.FsCommand

  `getCommandName, processRawArguments, registerCommands, run, runAll`

  ### Methods inherited from class org.apache.hadoop.fs.shell.Command

  `displayError, displayError, displayWarning, exitCodeForError, expandArgument, expandArguments, getCommandFactory, getDepth, getDescription, getListingGroupSize, getName, getReplacementCommand, getUsage, isDeprecated, isPathRecursable, isRecursive, isSorted, postProcessPath, processArgument, processNonexistentPath, processOptions, processPaths, processPaths, run, run, setCommandFactory, setName, setRecursive`

  ### Methods inherited from class org.apache.hadoop.conf.Configured

  `getConf, setConf`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### DEFAULT_QUEUE_SIZE

    public static final int DEFAULT_QUEUE_SIZE
    See Also:
      - Constant Field Values

  - ### LOG

    protected static final org.slf4j.Logger LOG

  - ### dst

    protected PathData dst

- ## Constructor Details

  - ### CopyCommandWithMultiThread

    public CopyCommandWithMultiThread()

- ## Method Details

  - ### setThreadCount

    protected void setThreadCount(String optValue)
    set thread count by option value, if the value less than 1, use 1 instead.
    Parameters:
      `optValue` - option value

  - ### setThreadPoolQueueSize

    protected void setThreadPoolQueueSize(String optValue)
    set thread pool queue size by option value, if the value less than 1, use DEFAULT_QUEUE_SIZE instead.
    Parameters:
      `optValue` - option value

  - ### getThreadCount

    @VisibleForTesting protected int getThreadCount()

  - ### getThreadPoolQueueSize

    @VisibleForTesting protected int getThreadPoolQueueSize()

  - ### getExecutor

    @VisibleForTesting protected ThreadPoolExecutor getExecutor()

  - ### processArguments

    protected void processArguments(LinkedList<PathData> args) throws IOException
    Description copied from class: `Command`
    Processes the command's list of expanded arguments. `Command.processArgument(PathData)` will be invoked with each item in the list. The loop catches IOExceptions, increments the error count, and displays the exception.
    Parameters:
      `args` - a list of `PathData` to process
    Throws:
      `IOException` - if anything goes wrong...

  - ### isMultiThreadNecessary

    @VisibleForTesting protected boolean isMultiThreadNecessary(LinkedList<PathData> args) throws IOException
    Throws:
      `IOException`

  - ### copyFileToTarget

    protected void copyFileToTarget(PathData src, PathData target) throws IOException
    Copies the source file to the target.
    Parameters:
      `src` - item to copy
      `target` - where to copy the item
    Throws:
      `IOException` - if copy fails

  - ### setOverwrite

    protected void setOverwrite(boolean flag)
    This method is used to enable the force(-f) option while copying the files.
    Parameters:
      `flag` - true/false

  - ### setLazyPersist

    protected void setLazyPersist(boolean flag)

  - ### setVerifyChecksum

    protected void setVerifyChecksum(boolean flag)

  - ### setWriteChecksum

    protected void setWriteChecksum(boolean flag)

  - ### setDirectWrite

    protected void setDirectWrite(boolean flag)

  - ### setPreserve

    protected void setPreserve(boolean preserve)
    If true, the last modified time, last access time, owner, group and permission information of the source file will be preserved as far as target `FileSystem` implementation allows.
    Parameters:
      `preserve` - preserve.

  - ### preserve

    protected void preserve(org.apache.hadoop.fs.shell.CommandWithDestination.FileAttribute fileAttribute)
    Add file attributes that need to be preserved. This method may be called multiple times to add attributes.
    Parameters:
      `fileAttribute` - - Attribute to add, one at a time

  - ### getLocalDestination

    protected void getLocalDestination(LinkedList<String> args) throws IOException
    The last arg is expected to be a local path, if only one argument is given then the destination will be the current directory
    Parameters:
      `args` - is the list of arguments
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### getRemoteDestination

    protected void getRemoteDestination(LinkedList<String> args) throws IOException
    The last arg is expected to be a remote path, if only one argument is given then the destination will be the remote user's directory
    Parameters:
      `args` - is the list of arguments
    Throws:
      `PathIOException` - if path doesn't exist or matches too many times
      `IOException`

  - ### processPathArgument

    protected void processPathArgument(PathData src) throws IOException
    Description copied from class: `Command`
    This is the last chance to modify an argument before going into the (possibly) recursive `Command.processPaths(PathData, PathData...)` -> `Command.processPath(PathData)` loop. Ex. ls and du use this to expand out directories.
    Overrides:
      `processPathArgument` in class `Command`
    Parameters:
      `src` - a `PathData` representing a path which exists
    Throws:
      `IOException` - if anything goes wrong...

  - ### processPath

    protected void processPath(PathData src) throws IOException
    Description copied from class: `Command`
    Hook for commands to implement an operation to be applied on each path for the command. Note implementation of this method is optional if earlier methods in the chain handle the operation.
    Overrides:
      `processPath` in class `Command`
    Parameters:
      `src` - a `PathData` object
    Throws:
      `IOException` - if anything else goes wrong in the user-implementation

  - ### processPath

    protected void processPath(PathData src, PathData dst) throws IOException
    Called with a source and target destination pair
    Parameters:
      `src` - for the operation
      `dst` - for the operation
    Throws:
      `IOException` - if anything goes wrong

  - ### recursePath

    protected void recursePath(PathData src) throws IOException
    Description copied from class: `Command`
    Gets the directory listing for a path and invokes `Command.processPaths(PathData, PathData...)`
    Overrides:
      `recursePath` in class `Command`
    Parameters:
      `src` - `PathData` for directory to recurse into
    Throws:
      `IOException` - if anything goes wrong...

  - ### getTargetPath

    protected PathData getTargetPath(PathData src) throws IOException
    Throws:
      `IOException`

  - ### copyStreamToTarget

    protected void copyStreamToTarget(InputStream in, PathData target) throws IOException
    If direct write is disabled ,copies the stream contents to a temporary file "target._COPYING_". If the copy is successful, the temporary file will be renamed to the real path, else the temporary file will be deleted. if direct write is enabled , then creation temporary file is skipped.
    Parameters:
      `in` - the input stream for the copy
      `target` - where to store the contents of the stream
    Throws:
      `IOException` - if copy fails

  - ### preserveAttributes

    protected void preserveAttributes(PathData src, PathData target, boolean preserveRawXAttrs) throws IOException
    Preserve the attributes of the source to the target. The method calls `shouldPreserve(FileAttribute)` to check what attribute to preserve.
    Parameters:
      `src` - source to preserve
      `target` - where to preserve attributes
      `preserveRawXAttrs` - true if raw.* xattrs should be preserved
    Throws:
      `IOException` - if fails to preserve attributes