Shell (Apache Hadoop Common 3.5.0 API)
Description: declaration: package: org.apache.hadoop.util, class: Shell

Package org.apache.hadoop.util

# Class Shell

java.lang.Object
org.apache.hadoop.util.Shell

Direct Known Subclasses:
  `DF`, `Shell.ShellCommandExecutor`, `Stat`

@Public @Evolving public abstract class Shell extends Object
A base class for running a Shell command. `Shell` can be used to run shell commands like `du` or `df`. It also offers facilities to gate commands by time-intervals.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static interface`
  `Shell.CommandExecutor`
  `static class`
  `Shell.ExitCodeException`
  This is an IOException with exit code added.
  `static enum`
  `Shell.OSType`
  `static class`
  `Shell.ShellCommandExecutor`
  A simple shell command executor.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `ENV_HADOOP_HOME`
  Environment variable for Hadoop's home dir: "HADOOP_HOME".
  `static final String`
  `ENV_NAME_REGEX`
  Regular expression for environment variables: "[A-Za-z_][A-Za-z0-9_]*".
  `static final boolean`
  `FREEBSD`
  `protected boolean`
  `inheritParentEnv`
  Indicates if the parent env vars should be inherited or not
  `static final boolean`
  `isSetsidAvailable`
  Flag which is true if setsid exists.
  `static final String`
  `LINK_COMMAND`
  a Unix command to create a link: "ln".
  `static final boolean`
  `LINUX`
  `static final org.slf4j.Logger`
  `LOG`
  `static final boolean`
  `MAC`
  `static final Shell.OSType`
  `osType`
  Get the type of the operating system, as determined from parsing the `os.name` property.
  `static final boolean`
  `OTHER`
  `static final boolean`
  `PPC_64`
  `static final String`
  `READ_LINK_COMMAND`
  a Unix command to get a link target: "readlink".
  `static final String`
  `SET_GROUP_COMMAND`
  a Unix command to set the change user's groups list: "chgrp".
  `static final String`
  `SET_OWNER_COMMAND`
  a Unix command to set owner: "chown".
  `static final String`
  `SET_PERMISSION_COMMAND`
  a Unix command to set permission: "chmod".
  `static final boolean`
  `SOLARIS`
  `static final String`
  `SYSPROP_HADOOP_HOME_DIR`
  System property for the Hadoop home directory: "hadoop.home.dir".
  `protected long`
  `timeOutInterval`
  Time after which the executing script would be timedout.
  `static final String`
  `TOKEN_SEPARATOR_REGEX`
  Token separator regex used to parse Shell tool outputs.
  `static final String`
  `USER_NAME_COMMAND`
  a Unix command to get the current user's name: "whoami".
  `static final boolean`
  `WINDOWS`
  `static final int`
  `WINDOWS_MAX_SHELL_LENGHT`
  Deprecated.
  use the correctly spelled constant.
  `static final int`
  `WINDOWS_MAX_SHELL_LENGTH`
  Maximum command line length in Windows KB830473 documents this as 8191
  `static final Object`
  `WindowsProcessLaunchLock`
  Windows `CreateProcess` synchronization object.
  `static final String`
  `WINUTILS`
  Deprecated.
  use one of the exception-raising getter methods, specifically `getWinUtilsPath()` or `getWinUtilsFile()`

- ## Constructor Summary

  Constructors
  Modifier
  Constructor
  Description
  `protected`
  `Shell()`
  Create an instance with no minimum interval between runs; stderr is not merged with stdout.
  `protected`
  `Shell(long interval)`
  Create an instance with a minimum interval between executions; stderr is not merged with stdout.
  `protected`
  `Shell(long interval, boolean redirectErrorStream)`
  Create a shell instance which can be re-executed when the `run()` method is invoked with a given elapsed time between calls.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `static File`
  `appendScriptExtension(File parent, String basename)`
  Returns a File referencing a script with the given basename, inside the given parent directory.
  `static String`
  `appendScriptExtension(String basename)`
  Returns a script file name with the given basename.
  `static String`
  `bashQuote(String arg)`
  Quote the given arg so that bash will interpret it as a single value.
  `static boolean`
  `checkIsBashSupported()`
  `static void`
  `checkWindowsCommandLineLength(String... commands)`
  Checks if a given command (String[]) fits in the Windows maximum command line length Note that the input is expected to already include space delimiters, no extra count will be added for delimiters.
  `static void`
  `destroyAllShellProcesses()`
  Static method to destroy all running `Shell` processes.
  `static String`
  `execCommand(String... cmd)`
  Static method to execute a shell command.
  `static String`
  `execCommand(Map<String,String> env, String... cmd)`
  Static method to execute a shell command.
  `static String`
  `execCommand(Map<String,String> env, String[] cmd, long timeout)`
  Static method to execute a shell command.
  `static Set<Shell>`
  `getAllShells()`
  Static method to return a Set of all `Shell` objects.
  `static String[]`
  `getCheckProcessIsAliveCommand(String pid)`
  Return a command for determining if process with specified pid is alive.
  `String`
  `getEnvironment(String env)`
  Get an environment variable.
  `static String`
  `getEnvironmentVariableRegex()`
  Return a regular expression string that match environment variables.
  `protected abstract String[]`
  `getExecString()`
  return an array containing the command name and its parameters.
  `int`
  `getExitCode()`
  get the exit code.
  `static String[]`
  `getGetPermissionCommand()`
  Return a command to get permission information.
  `static String[]`
  `getGroupsCommand()`
  a Unix command to get the current user's groups list.
  `static String[]`
  `getGroupsForUserCommand(String user)`
  A command to get a given user's groups list.
  `static String[]`
  `getGroupsIDForUserCommand(String user)`
  A command to get a given user's group id list.
  `static String`
  `getHadoopHome()`
  Get the Hadoop home directory.
  `static Long`
  `getMemlockLimit(Long ulimit)`
  Static method to return the memory lock limit for datanode.
  `Process`
  `getProcess()`
  get the current sub-process executing the given command.
  `static File`
  `getQualifiedBin(String executable)`
  Fully qualify the path to a binary that should be in a known hadoop bin location.
  `static String`
  `getQualifiedBinPath(String executable)`
  Fully qualify the path to a binary that should be in a known hadoop bin location.
  `static String[]`
  `getReadlinkCommand(String link)`
  Return a command to read the target of the a symbolic link.
  `static String[]`
  `getRunScriptCommand(File script)`
  Returns a command to run the given script.
  `static String[]`
  `getSetOwnerCommand(String owner)`
  Return a command to set owner.
  `static String[]`
  `getSetPermissionCommand(String perm, boolean recursive)`
  Return a command to set permission.
  `static String[]`
  `getSetPermissionCommand(String perm, boolean recursive, String file)`
  Return a command to set permission for specific file.
  `static String[]`
  `getSignalKillCommand(int code, String pid)`
  Return a command to send a signal to a given pid.
  `static String[]`
  `getSymlinkCommand(String target, String link)`
  Return a command to create symbolic links.
  `static String[]`
  `getUsersForNetgroupCommand(String netgroup)`
  A command to get a given netgroup's user list.
  `Thread`
  `getWaitingThread()`
  get the thread that is waiting on this instance of `Shell`.
  `static File`
  `getWinUtilsFile()`
  Get a file reference to winutils.
  `static String`
  `getWinUtilsPath()`
  Locate the winutils binary, or fail with a meaningful exception and stack trace as an RTE.
  `static boolean`
  `hasWinutilsPath()`
  Predicate to indicate whether or not the path to winutils is known.
  `static boolean`
  `isJava7OrAbove()`
  Deprecated.
  This call isn't needed any more: please remove uses of it.
  `static boolean`
  `isJavaVersionAtLeast(int version)`
  Query to see if major version of Java specification of the system is equal or greater than the parameter.
  `boolean`
  `isTimedOut()`
  To check if the passed script to shell command executor timed out or not.
  `protected abstract void`
  `parseExecResult(BufferedReader lines)`
  Parse the execution result.
  `protected void`
  `run()`
  Check to see if a command needs to be executed and execute if needed.
  `protected void`
  `setEnvironment(Map<String,String> env)`
  Set the environment for the command.
  `protected void`
  `setWorkingDirectory(File dir)`
  Set the working directory.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### LOG

    public static final org.slf4j.Logger LOG

  - ### SYSPROP_HADOOP_HOME_DIR

    public static final String SYSPROP_HADOOP_HOME_DIR
    System property for the Hadoop home directory: "hadoop.home.dir".
    See Also:
      - Constant Field Values

  - ### ENV_HADOOP_HOME

    public static final String ENV_HADOOP_HOME
    Environment variable for Hadoop's home dir: "HADOOP_HOME".
    See Also:
      - Constant Field Values

  - ### WINDOWS_MAX_SHELL_LENGTH

    public static final int WINDOWS_MAX_SHELL_LENGTH
    Maximum command line length in Windows KB830473 documents this as 8191
    See Also:
      - Constant Field Values

  - ### WINDOWS_MAX_SHELL_LENGHT

    @Deprecated public static final int WINDOWS_MAX_SHELL_LENGHT
    Deprecated.
    use the correctly spelled constant.
    mis-spelling of `WINDOWS_MAX_SHELL_LENGTH`.
    See Also:
      - Constant Field Values

  - ### USER_NAME_COMMAND

    public static final String USER_NAME_COMMAND
    a Unix command to get the current user's name: "whoami".
    See Also:
      - Constant Field Values

  - ### WindowsProcessLaunchLock

    public static final Object WindowsProcessLaunchLock
    Windows `CreateProcess` synchronization object.

  - ### osType

    public static final Shell.OSType osType
    Get the type of the operating system, as determined from parsing the `os.name` property.

  - ### WINDOWS

    public static final boolean WINDOWS

  - ### SOLARIS

    public static final boolean SOLARIS

  - ### MAC

    public static final boolean MAC

  - ### FREEBSD

    public static final boolean FREEBSD

  - ### LINUX

    public static final boolean LINUX

  - ### OTHER

    public static final boolean OTHER

  - ### PPC_64

    public static final boolean PPC_64

  - ### ENV_NAME_REGEX

    public static final String ENV_NAME_REGEX
    Regular expression for environment variables: "[A-Za-z_][A-Za-z0-9_]*".
    See Also:
      - Constant Field Values

  - ### SET_PERMISSION_COMMAND

    public static final String SET_PERMISSION_COMMAND
    a Unix command to set permission: "chmod".
    See Also:
      - Constant Field Values

  - ### SET_OWNER_COMMAND

    public static final String SET_OWNER_COMMAND
    a Unix command to set owner: "chown".
    See Also:
      - Constant Field Values

  - ### SET_GROUP_COMMAND

    public static final String SET_GROUP_COMMAND
    a Unix command to set the change user's groups list: "chgrp".
    See Also:
      - Constant Field Values

  - ### LINK_COMMAND

    public static final String LINK_COMMAND
    a Unix command to create a link: "ln".
    See Also:
      - Constant Field Values

  - ### READ_LINK_COMMAND

    public static final String READ_LINK_COMMAND
    a Unix command to get a link target: "readlink".
    See Also:
      - Constant Field Values

  - ### timeOutInterval

    protected long timeOutInterval
    Time after which the executing script would be timedout.

  - ### inheritParentEnv

    protected boolean inheritParentEnv
    Indicates if the parent env vars should be inherited or not

  - ### WINUTILS

    @Deprecated public static final String WINUTILS
    Deprecated.
    use one of the exception-raising getter methods, specifically `getWinUtilsPath()` or `getWinUtilsFile()`
    Location of winutils as a string; null if not found.

    Important: caller must check for this value being null. The lack of such checks has led to many support issues being raised.

  - ### isSetsidAvailable

    public static final boolean isSetsidAvailable
    Flag which is true if setsid exists.

  - ### TOKEN_SEPARATOR_REGEX

    public static final String TOKEN_SEPARATOR_REGEX
    Token separator regex used to parse Shell tool outputs.

- ## Constructor Details

  - ### Shell

    protected Shell()
    Create an instance with no minimum interval between runs; stderr is not merged with stdout.

  - ### Shell

    protected Shell(long interval)
    Create an instance with a minimum interval between executions; stderr is not merged with stdout.
    Parameters:
      `interval` - interval in milliseconds between command executions.

  - ### Shell

    protected Shell(long interval, boolean redirectErrorStream)
    Create a shell instance which can be re-executed when the `run()` method is invoked with a given elapsed time between calls.
    Parameters:
      `interval` - the minimum duration in milliseconds to wait before re-executing the command. If set to 0, there is no minimum.
      `redirectErrorStream` - should the error stream be merged with the normal output stream?

- ## Method Details

  - ### isJava7OrAbove

    @Deprecated public static boolean isJava7OrAbove()
    Deprecated.
    This call isn't needed any more: please remove uses of it.
    query to see if system is Java 7 or later. Now that Hadoop requires Java 7 or later, this always returns true.
    Returns:
      true, always.

  - ### isJavaVersionAtLeast

    public static boolean isJavaVersionAtLeast(int version)
    Query to see if major version of Java specification of the system is equal or greater than the parameter.
    Parameters:
      `version` - 8, 9, 10 etc.
    Returns:
      comparison with system property, always true for 8

  - ### checkWindowsCommandLineLength

    public static void checkWindowsCommandLineLength(String... commands) throws IOException
    Checks if a given command (String[]) fits in the Windows maximum command line length Note that the input is expected to already include space delimiters, no extra count will be added for delimiters.
    Parameters:
      `commands` - command parts, including any space delimiters
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### bashQuote

    @Private public static String bashQuote(String arg)
    Quote the given arg so that bash will interpret it as a single value. Note that this quotes it for one level of bash, if you are passing it into a badly written shell script, you need to fix your shell script.
    Parameters:
      `arg` - the argument to quote
    Returns:
      the quoted string

  - ### getGroupsCommand

    public static String[] getGroupsCommand()
    a Unix command to get the current user's groups list.
    Returns:
      group command array.

  - ### getGroupsForUserCommand

    public static String[] getGroupsForUserCommand(String user)
    A command to get a given user's groups list. If the OS is not WINDOWS, the command will get the user's primary group first and finally get the groups list which includes the primary group. i.e. the user's primary group will be included twice.
    Parameters:
      `user` - user.
    Returns:
      groups for user command.

  - ### getGroupsIDForUserCommand

    public static String[] getGroupsIDForUserCommand(String user)
    A command to get a given user's group id list. The command will get the user's primary group first and finally get the groups list which includes the primary group. i.e. the user's primary group will be included twice. This command does not support Windows and will only return group names.
    Parameters:
      `user` - user.
    Returns:
      groups id for user command.

  - ### getUsersForNetgroupCommand

    public static String[] getUsersForNetgroupCommand(String netgroup)
    A command to get a given netgroup's user list.
    Parameters:
      `netgroup` - net group.
    Returns:
      users for net group command.

  - ### getGetPermissionCommand

    public static String[] getGetPermissionCommand()
    Return a command to get permission information.
    Returns:
      permission command.

  - ### getSetPermissionCommand

    public static String[] getSetPermissionCommand(String perm, boolean recursive)
    Return a command to set permission.
    Parameters:
      `perm` - permission.
      `recursive` - recursive.
    Returns:
      set permission command.

  - ### getSetPermissionCommand

    public static String[] getSetPermissionCommand(String perm, boolean recursive, String file)
    Return a command to set permission for specific file.
    Parameters:
      `perm` - String permission to set
      `recursive` - boolean true to apply to all sub-directories recursively
      `file` - String file to set
    Returns:
      String[] containing command and arguments

  - ### getSetOwnerCommand

    public static String[] getSetOwnerCommand(String owner)
    Return a command to set owner.
    Parameters:
      `owner` - owner.
    Returns:
      set owner command.

  - ### getSymlinkCommand

    public static String[] getSymlinkCommand(String target, String link)
    Return a command to create symbolic links.
    Parameters:
      `target` - target.
      `link` - link.
    Returns:
      symlink command.

  - ### getReadlinkCommand

    public static String[] getReadlinkCommand(String link)
    Return a command to read the target of the a symbolic link.
    Parameters:
      `link` - link.
    Returns:
      read link command.

  - ### getCheckProcessIsAliveCommand

    public static String[] getCheckProcessIsAliveCommand(String pid)
    Return a command for determining if process with specified pid is alive.
    Parameters:
      `pid` - process ID
    Returns:
      a `kill -0` command or equivalent

  - ### getSignalKillCommand

    public static String[] getSignalKillCommand(int code, String pid)
    Return a command to send a signal to a given pid.
    Parameters:
      `code` - code.
      `pid` - pid.
    Returns:
      signal kill command.

  - ### getEnvironmentVariableRegex

    public static String getEnvironmentVariableRegex()
    Return a regular expression string that match environment variables.
    Returns:
      environment variable regex.

  - ### appendScriptExtension

    public static File appendScriptExtension(File parent, String basename)
    Returns a File referencing a script with the given basename, inside the given parent directory. The file extension is inferred by platform: `".cmd"` on Windows, or `".sh"` otherwise.
    Parameters:
      `parent` - File parent directory
      `basename` - String script file basename
    Returns:
      File referencing the script in the directory

  - ### appendScriptExtension

    public static String appendScriptExtension(String basename)
    Returns a script file name with the given basename. The file extension is inferred by platform: `".cmd"` on Windows, or `".sh"` otherwise.
    Parameters:
      `basename` - String script file basename
    Returns:
      String script file name

  - ### getRunScriptCommand

    public static String[] getRunScriptCommand(File script)
    Returns a command to run the given script. The script interpreter is inferred by platform: cmd on Windows or bash otherwise.
    Parameters:
      `script` - File script to run
    Returns:
      String[] command to run the script

  - ### getHadoopHome

    public static String getHadoopHome() throws IOException
    Get the Hadoop home directory. Raises an exception if not found
    Returns:
      the home dir
    Throws:
      `IOException` - if the home directory cannot be located.

  - ### getQualifiedBin

    public static File getQualifiedBin(String executable) throws FileNotFoundException
    Fully qualify the path to a binary that should be in a known hadoop bin location. This is primarily useful for disambiguating call-outs to executable sub-components of Hadoop to avoid clashes with other executables that may be in the path. Caveat: this call doesn't just format the path to the bin directory. It also checks for file existence of the composed path. The output of this call should be cached by callers.
    Parameters:
      `executable` - executable
    Returns:
      executable file reference
    Throws:
      `FileNotFoundException` - if the path does not exist

  - ### getQualifiedBinPath

    public static String getQualifiedBinPath(String executable) throws IOException
    Fully qualify the path to a binary that should be in a known hadoop bin location. This is primarily useful for disambiguating call-outs to executable sub-components of Hadoop to avoid clashes with other executables that may be in the path. Caveat: this call doesn't just format the path to the bin directory. It also checks for file existence of the composed path. The output of this call should be cached by callers.
    Parameters:
      `executable` - executable
    Returns:
      executable file reference
    Throws:
      `FileNotFoundException` - if the path does not exist
      `IOException` - on path canonicalization failures

  - ### hasWinutilsPath

    public static boolean hasWinutilsPath()
    Predicate to indicate whether or not the path to winutils is known. If true, then `WINUTILS` is non-null, and both `getWinUtilsPath()` and `getWinUtilsFile()` will successfully return this value. Always false on non-windows systems.
    Returns:
      true if there is a valid path to the binary

  - ### getWinUtilsPath

    public static String getWinUtilsPath()
    Locate the winutils binary, or fail with a meaningful exception and stack trace as an RTE. This method is for use in methods which don't explicitly throw an `IOException`.
    Returns:
      the path to `WINUTILS_EXE`
    Throws:
      `RuntimeException` - if the path is not resolvable

  - ### getWinUtilsFile

    public static File getWinUtilsFile() throws FileNotFoundException
    Get a file reference to winutils. Always raises an exception if there isn't one
    Returns:
      the file instance referring to the winutils bin.
    Throws:
      `FileNotFoundException` - on any failure to locate that file.

  - ### checkIsBashSupported

    public static boolean checkIsBashSupported() throws InterruptedIOException
    Throws:
      `InterruptedIOException`

  - ### setEnvironment

    protected void setEnvironment(Map<String,String> env)
    Set the environment for the command.
    Parameters:
      `env` - Mapping of environment variables

  - ### setWorkingDirectory

    protected void setWorkingDirectory(File dir)
    Set the working directory.
    Parameters:
      `dir` - The directory where the command will be executed

  - ### run

    protected void run() throws IOException
    Check to see if a command needs to be executed and execute if needed.
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### getExecString

    protected abstract String[] getExecString()
    return an array containing the command name and its parameters.
    Returns:
      exec string array.

  - ### parseExecResult

    protected abstract void parseExecResult(BufferedReader lines) throws IOException
    Parse the execution result.
    Parameters:
      `lines` - lines.
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### getEnvironment

    public String getEnvironment(String env)
    Get an environment variable.
    Parameters:
      `env` - the environment var
    Returns:
      the value or null if it was unset.

  - ### getProcess

    public Process getProcess()
    get the current sub-process executing the given command.
    Returns:
      process executing the command

  - ### getExitCode

    public int getExitCode()
    get the exit code.
    Returns:
      the exit code of the process

  - ### getWaitingThread

    public Thread getWaitingThread()
    get the thread that is waiting on this instance of `Shell`.
    Returns:
      the thread that ran runCommand() that spawned this shell or null if no thread is waiting for this shell to complete

  - ### isTimedOut

    public boolean isTimedOut()
    To check if the passed script to shell command executor timed out or not.
    Returns:
      if the script timed out.

  - ### execCommand

    public static String execCommand(String... cmd) throws IOException
    Static method to execute a shell command. Covers most of the simple cases without requiring the user to implement the `Shell` interface.
    Parameters:
      `cmd` - shell command to execute.
    Returns:
      the output of the executed command.
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### execCommand

    public static String execCommand(Map<String,String> env, String[] cmd, long timeout) throws IOException
    Static method to execute a shell command. Covers most of the simple cases without requiring the user to implement the `Shell` interface.
    Parameters:
      `env` - the map of environment key=value
      `cmd` - shell command to execute.
      `timeout` - time in milliseconds after which script should be marked timeout
    Returns:
      the output of the executed command.
    Throws:
      `IOException` - on any problem.

  - ### execCommand

    public static String execCommand(Map<String,String> env, String... cmd) throws IOException
    Static method to execute a shell command. Covers most of the simple cases without requiring the user to implement the `Shell` interface.
    Parameters:
      `env` - the map of environment key=value
      `cmd` - shell command to execute.
    Returns:
      the output of the executed command.
    Throws:
      `IOException` - on any problem.

  - ### destroyAllShellProcesses

    public static void destroyAllShellProcesses()
    Static method to destroy all running `Shell` processes. Iterates through a map of all currently running `Shell` processes and destroys them one by one. This method is thread safe

  - ### getAllShells

    public static Set<Shell> getAllShells()
    Static method to return a Set of all `Shell` objects.
    Returns:
      all shells set.

  - ### getMemlockLimit

    public static Long getMemlockLimit(Long ulimit)
    Static method to return the memory lock limit for datanode.
    Parameters:
      `ulimit` - max value at which memory locked should be capped.
    Returns:
      long value specifying the memory lock limit.