U  g@sdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Zddl m Z mZmZmZGdddeZGdddeZGdd d eZGd d d eZGd d d eZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZddZddZddZGd d!d!e Z!Gd"d#d#e!Z"Gd$d%d%e!Z#Gd&d'd'e!Z$Gd(d)d)e!Z%Gd*d+d+e Z&Gd,d-d-e&Z'dS).a!Implements the produtil.run: provides the object tree for representing shell commands. Do not load this module directly except for type checking (instanceof(o,produtil.prog.Runner)). It is meant to be used only by the produtil.run module. This module is part of the implementation of a shell-like syntax for running programs. The rest of the implementation is in the produtil.run and produtil.pipeline modules. MPI programs are implemented by the produtil.mpiprog and produtil.mpi_impl. This module implements a shell-like syntax of running shell programs from Python. This module should not be used directly: the produtil.run implements critical parts of the functionality. Specifically, this module implements the Runner and ImmutableRunner classes. It also knows how to convert them to produtil.pipeline.Pipeline objects for actual execution. * Runner --- This class represents a process that could be run. It keeps track of all possible aspects of running a process, including the command, arguments, environment variables, stdout stream, stderr stream, stdin stream, and a list of functions or callable objects to run before executing the problem. Provides public functions to modify the Runner. * ImmutableRunner --- A Runner that cannot be changed: when modifying the Runner, it returns a new object. This is to implement shell aliases. For example, one could make an ImmutableRunner for program to index GRIB2 files. All the user would have to do is add the GRIB2 file as an argument, and capture the output. Note that the actual work of creating the Runner or ImmutableRunner, or turning them into Pipeline objects done by the produtil.run module. Turning MPI programs into Runner objects is done by the produtil.mpiprog module and produtil.mpi_impl package, with the public interface in produtil.run. Hence, nobody would ever load this module directly, except for type checking (ie.: to see if your argument is a Runner before passing it to produtil.run.checkrun)..N)launchmanagePIPEERR2OUTc@seZdZdZdS)ProgSyntaxErrorzY!Base class of exceptions raised when a Runner is given arguments that make no sense.N__name__ __module__ __qualname____doc__r r :/lfs/h1/ops/prod/packages/hmon.v3.2.7/ush/produtil/prog.pyr/src@seZdZdZdS)OverspecifiedStreamzp!Raised when one tries to specify the stdout, stderr or stdin to go to, or come from, more than one locationNrr r r r r2src@seZdZdZdS) MultipleStdinzT!Raised when the caller specifies more than one source for the stdin of a RunnerNrr r r r r5src@seZdZdZdS)MultipleStdoutzU!Raised when the caller specifies more than one destination for a Runner's stdoutNrr r r r r8src@seZdZdZdS)MultipleStderrzU!Raised when the caller specifies more than one destination for a Runner's stderrNrr r r r r;src@seZdZdZdS)InvalidPipelinezt!Raised when the caller specifies an invalid input or output when piping a Runner into or out of another object.Nrr r r r r>src@seZdZdZdS)NotValidPosixShz!Base class of exceptions that are raised when converting a Runner or pipeline of Runners to a POSIX sh command, if the Runner cannot be expressed as POSIX sh.Nrr r r r rBsrc@seZdZdZdS)NoSuchRedirectionz!Raised when trying to convert a pipeline of Runners to a POSIX sh string, if a redirection in the pipeline cannot be expressed in POSIX sh.Nrr r r r rFsrc@seZdZdZdS)NotValidPosixShStringz!Raised when converting a Runner or pipeline of Runners to a POSIX sh string. If a string is sent to a program's stdin, this is raised when that string cannot be expressed in POSIX sh.Nrr r r r rJsrc@seZdZdZdS)EqualInExecutablez!Raised when converting a Runner or pipeline of Runners to a posix sh string if a Runner's executable contains an equal ("=") sign.Nrr r r r rNsrc@seZdZdZdS) EqualInEnvz!Raised when converting a Runner or pipeline of Runners to a POSIX sh string if there is an equal ("=") sign in an environment variable name.Nrr r r r rRsrcCstd|rdSdSdS)z!Returns True if the specified environment variable name is a valid POSIX sh variable name, and False otherwise. @param s an environment variable namez\A[A-Za-z][A-Za-z0-9_]*\zTFNresearchsr r r shvarokWs rcCstd|rdSdSdS)z{!Returns True if the specified string can be expressed as a POSIX sh string, and false otherwise. @param s a stringz6\A[a-zA-Z0-9 !"#$%&?()*+,./:;<=>?@^_`{|}~\\\]\[\'-]*\ZTFNrrr r r shstrok`s rcCs@t|stdt|ftd|rr selfr r r for_input{szStreamGenerator.for_inputcCsdt|fSr$r%r&r r r for_output~szStreamGenerator.for_outputcCs|S)z!Returns the stderr value. The default implementation returns repr_for_out(), causing stderr to receive whatever stdout receives. repr_for_outr&r r r repr_for_errszStreamGenerator.repr_for_errN)rr r r r(r)r,r r r r r#vsr#c@s^eZdZdZdddZddZddZed d Zd d Z d dZ ddZ ddZ ddZ dS) FileOpenera!This is part of the internal implementation of Runner, used to convert it to a produtil.pipeline.Pipeline for execution. It represents stdin, stdout or stderr being connected to an open file. It instructs the Runner to open the file before starting the process.FcCs||_||_||_dS)z!FileOpener constructor @param filename the name of the file being opened @param mode how it is being opened @param err if True, this is for stderrN)filenamemodeerr)r'r.r/r0r r r __init__szFileOpener.__init__cCst|j|j|jS)z'!Creates a shallow copy of this object.)r-r.r/r0r&r r r copyszFileOpener.copycCstd}|jrd}|jdkr*d|t|jfS|jdkrFd|t|jfS|jdkr`dt|jfStd |jfd S) z`!Creates a POSIX sh representation of the part of the command that requests redirection.2abz%s>> %swbz%s> %srbz cat %s | z:Cannot convert file open mode %s to a POSIX sh redirectionN)r0r/r"r.r)r'morer r r to_shells   zFileOpener.to_shellcCsld}ddkstd|jkr"tj}n:d|jkr@tjtjBtjB}nd|jkr\tjtjBtjB}|dk sht|S)z8!Returns an integer version of mode suitable for os.openNrr7wa)AssertionErrorr/osO_RDONLYO_WRONLYO_CREATO_TRUNCO_APPEND)r'intmoder r r rDs     zFileOpener.intmodecCsdt|j|jddfS)zZ!Returns a tuple (None,stream,None,True) where "stream" is the opened file object.NT)r>openr.rDr&r r r _gen_streamszFileOpener._gen_streamcCsdt|jt|jfS)M!Returns a string representation of this object as valid Python code.zFileOpener(%s,%s)r r.r/r&r r r __repr__szFileOpener.__repr__cCst|jdS)zf!Part of the implementation of Runner.__repr__, this returns the filename and ",string=False".z ,string=False)r r.r&r r r repr_for_inszFileOpener.repr_for_incCsdt|jt|jdkfS)z!Part of the implementation of Runner.__repr__, this returns the filename and ",string=False". It also appends ",append=X" where X is the true/false flag for appending to the file.z %s,append=%sr5rHr&r r r r+szFileOpener.repr_for_outcCs|S)z!Same as repr_for_out.r*r&r r r r,szFileOpener.repr_for_errN)F)rr r r r1r2r9propertyrDrFrIrJr+r,r r r r r-s   r-c@s@eZdZdZddZddZddZdd Zd d Zd d Z dS) StringInputz2!Represents sending a string to a process's stdin.cCs ||_dS)zw!Creates a StringInput that sends the specified object to stdin. @param obj the object to send to stdinNobjr'rNr r r r1szStringInput.__init__cCs t|jS)z'!Returns a shallow copy of this object.)rLrNr&r r r r2szStringInput.copycCs|jdddfS)zr!Returns a tuple containing (O,None,None,None) where O was the object sent to the StringInput constructor.NrMr&r r r rFszStringInput._gen_streamcCsdt|jfS)rGzStringInput(%s)r rNr&r r r rIszStringInput.__repr__cCsdt|jS)zX!Converts this object, if possible, to an echo command followed by a pipe ("|").z echo %s | )r"rNr&r r r r9szStringInput.to_shellcCs>t|jdkr*dt|jdddfSdt|jfSdS)z!Part of the implementation of Runner.__repr__. If possible, this creates valid Python code to represent specifying sending the given string to the stdin of a Runner. If the string is too long, it is abbreviated.(z%s...,string=Truer%z...z%s,string=TrueN)lenrNr r&r r r rJszStringInput.repr_for_inN) rr r r r1r2rFrIr9rJr r r r rLsrLc@s@eZdZdZddZddZddZdd Zd d Zd d Z dS) StreamReuserz]!Arranges for a stream-like object to be sent to the stdout, stderr or stdin of a Runner.cCs ||_dS)zq!Creates a StreamReuser for the specified stream-like object. @param obj the stream-like object to reuse.NrMrOr r r r1szStreamReuser.__init__cCs t|jS)zq!Returns a shallow copy of this object. Note that means that the underlying stream object is not copied.)rTrNr&r r r r2szStreamReuser.copycCs tddS)z^!Raises NotValidPosixSh to indicate that the stream cannot be represented as POSIX sh.z=Python streams cannot be passed to remote POSIX sh processes.N)rr&r r r r9szStreamReuser.to_shellcCsdd|jdfS)z\!Returns a tuple (None,None,obj,False) where obj is the provided stream-like object.NFrMr&r r r rFszStreamReuser._gen_streamcCs t|jSzE!Returns repr(obj) where obj is the given stream-like object.rPr&r r r rJszStreamReuser.repr_for_incCs t|jSrUrPr&r r r r+ szStreamReuser.repr_for_outN) rr r r r1r2r9rFrJr+r r r r rTsrTc@sHeZdZdZddZddZddZdd Zd d Zd d Z ddZ dS) OutIsErrorz,!Instructs a Runner to send stderr to stdoutcCsdS)z!OutIsError constructor.Nr r&r r r r1szOutIsError.__init__cCstS)z!!Returns a new OutIsError object.)rVr&r r r r2szOutIsError.copycCsdS)z!Returns "2>&1" z2>&1r r&r r r r9szOutIsError.to_shellcCs ddtdfS)z>!Returns a tuple containing (None,None,pipeline.ERR2OUT,False)NF)rr&r r r rFszOutIsError._gen_streamcCsdS)z7!This should never be called. It returns ".err2out()". .err2out()r r&r r r rJszOutIsError.repr_for_incCsdS)z!Part of the representation of Runner.__repr__. Returns ".err2out()" which instructs a Runner to send stderr to stdout.rWr r&r r r r+ szOutIsError.repr_for_outcCs t|tS)zU!Is the other object an OutIsError? @param other the other object to analyze.) isinstancerVr'otherr r r __eq__%szOutIsError.__eq__N) rr r r r1r2r9rFrJr+r[r r r r rVsrVc@sTeZdZdZddZddZddZdd Zeeeed Z ed d Z d dZ ddZ ddZ ddZddZddZddZddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*Zd+d,Zd-d.Zd/d0ZdOd2d3Zd4d5Zd6d7Zd8d9Z d:d;Z!dd?Z#d@dAZ$dBdCZ%dPdEdFZ&dQdGdHZ'dIdJZ(dRdKdLZ)dSdMdNZ*d1S)TRunnera!Represents a single stage of a pipeline to execute. This is a linked list class used to store information about a program or pipeline of programs to be run. It has the capability of converting itself to a Pipeline object (run(Runner)), or converting itself to a POSIX sh command (Runner.to_shell()). Note that some commands cannot be represented in POSIX sh, such as commands with non-ASCII characters or commands that have Python streams as their stdout or stdin. Those commands can still be run with a Pipeline, but trying to convert them to a POSIX sh command will throw NotValidPosixSh or a subclass thereof.cKsDd|_|_|_|_|_d|_|_d|_t|t r|}t |j |_ |jdk r`|j |_|jdk rv|j |_|jdk r|j |_|jdk r|j |_|jdk rt |j|_|jdk rt |j|_|j|_n^t|t stdt|jt|ft|dts*tdt|djt|f||_ d|_d|krR|drR|d|krjt |d|_d|kr||dkd |kr|t|d >d |kr||d kd |kr||d ?d |kr|d |kr|j|d ddd|kr|j|dddd|kr(||dd|kr@||ddS)a!Creates a new Runner. The only non-keyword argument can be one of three things: 1. A Runner to copy. Every aspect of the Runner that can be copied will be. Note that if a stream-like object is connected to stdin, stdout or stderr, it will NOT be copied. 2. A list of strings. This will be used as the command path, and arguments. Many options can be set via keyword arguments: * clearenv=True - the environment should be cleared before running this command. Any arguments set by the env= keyword or the .env(...) member function ignore this. Also, PATH, USER, LOGNAME and HOME are retained since most programs cannot run without them. * env=dict(var=value,...) - a dict of environment variables to set before running the Runner. Does NOT affect this parent's process, only the child process. * in=filename - a file to send to stdin. * instr=str - a string to send to stdin * out=filename - a file to connect to stdout. Will truncate the file. * outa=filename - same as "out=filename," but appends to the file. * err2out - redirects stderr to stdout * err=filename - a file to connect to stderr. Will truncate the file. * erra=filename - same as "err=filename," but appends to the file. * prerun=[obj,anotherobj,...] - sent to self.prerun, this is a list of functions or callable objects to run before executing the process. The objects are not called until execution is requested via self._gen. @param args the arguments to the program @param kwargs other settings (see constructor description).Nz.The args argument must be a list, not a %s %s.rz8The first element of args must be a string, not a %s %s.TclearenvenvininstroutZoutaerr2outr0FappendZerracdprerun)_stdin_stdout_stderr_prev_env_prerun_cd_threadsrXr\list_argsr2dict _copy_env TypeErrortyperr strr]rbr0rerf)r'argskwargsr:r r r r18sj.                      zRunner.__init__cCs|jS)z9!Returns the number of threads requested by this program.rnr&r r r getthreadsszRunner.getthreadscCst||_|jSz6!Sets the number of threads requested by this program.)intrn)r'nthreadsr r r setthreadss zRunner.setthreadscCs d|_dS)z!!Removes the request for threads.Nrxr&r r r delthreadsszRunner.delthreadszThe number of threads per rank.cCs|jdkr|S|jjS)z+!Returns the first Runner in this pipeline.N)rjfirstr&r r r rs z Runner.firstcCs|jrt|_|S)zG!Removes all prerun objects. @see prerun() @return self)rlror&r r r remove_prerunszRunner.remove_preruncCs$|jdkr|g|_n |j||S)a!Adds a function or callable object to be called before running the program. The callables should be very fast operations, and are executed by self._gen when creating the Pipeline. They take, as an argument, the Runner and an optional "logger" keyword argument that is either None, or a logging.Logger to use to log messages. @param arg a callable object that takes self as an argument, and an optional keyword argument "logger" with a logging.Logger for log messagesN)rlrdr'argr r r rfs   z Runner.preruncCsBt|trd|fSt|tr(d|fSt|tr6|St|SdS)a!Returns a string representation of the given argument to convert it to an input to produtil.pipeline.Pipeline. Conversions: * float --- converted via %g * int --- converted via %d * basestring --- no conversion; used directly * all others --- str(arg) @param arg the argument to convert @returns a string version of argz%gz%dN)rXfloatr{rurr r r _stringify_args     zRunner._stringify_argcsPt|tst|tst|tr2j|njfdd|DS)a!Add one or more arguments to the executable. Can ONLY accept strings, ints, floats or iterables (tuple, list). Strings, ints and floats are sent to _stringify_args, and the result is added to the end of the list of arguments to the command to run. For iterables (tuple, list), adds all elements to the list of arguments, passing each through _stringify_args. @param args one or more arguments to add @returns selfcsg|]}|qSr )r.0xr&r r sz&Runner.__getitem__..)rXrurr{rprdrextendr'rvr r&r __getitem__s zRunner.__getitem__cCs|S)z!Alias for __repr__())rIr&r r r __str__szRunner.__str__cCs|jdk rdt|jf}nd}t|jdkr8|d7}n|dt|jd7}t|jdkr|dd d d |jddDd 7}|jdk r|d |jf7}|jdk r|d|jf7}|j dk rt |j t r|d7}n|d|j f7}|j s|d7}|jdk r8|dddd |jDd7}|jdk r^|ddd |jD7}|jdk r|dt|jd7}|S)a/!Attempts to produce valid Python code to represent this Runnable. Generally, that can be done, unless an input string is too long, or a stream is connected to a Python object. In those cases, human-readable representations are given, which are not exactly Python code.Nz%s | r3rz exe()zexe(%s)[,cSsg|] }t|qSr r%rr r r rsz#Runner.__repr__..]z.in(%s)z.out(%s)rWz.err(%s)z .clearenv()z.env(z, cSsg|]\}}d||fqS)%s=%sr )rkvr r r rs)cSsg|]}dt|fqS)z .prerun(%s)r%rr r r rsz.cd()rjr rSrpjoinrgrJrhr+rirXrVr,rrrkitemsrlrm)r'rr r r rIs:  *        zRunner.__repr__cCsft|tr^|j|jko\|j|jko\|j|jko\|j|jko\|j|jko\|j|jko\|j|jkSt SdS)z!Returns True if the other object is a Runner that is equal to this one, and False otherwise. @param other the object to compareN) rXr\rprrrgrhrirkrlNotImplementedrYr r r r[s        z Runner.__eq__cCs<|jdko:|jdko:|jdko:|jdko:|jdko:|jdkS)z!Returns true if this is simply an executable with arguments (no redirection, no prerun objects, no environment modification, no piping), and False otherwise.N)rgrhrlrirkrjr&r r r isplainexeszRunner.isplainexecCs ||_|S)z!Requests that this process run in the specified directory. The directory must already exist before the program starts. @param dirpath the directory to cd into, which must already exist. @returns self)rm)r'dirpathr r r resz Runner.cdcCs|j|ddS)z}!Connects the given object to stdin, via inp(stdin,string=False). @param stdin the stdin object @returns selfFstringinpr'stdinr r r __lt__$sz Runner.__lt__cCs|j|ddS)z!Connects the given object to stdout, truncating it if it is a file. Same as out(stdout,append=False). @param stdout the stdout object @returns selfFrcrar'stdoutr r r __gt__)sz Runner.__gt__cCs|j|ddS)z!Sends the specified string into stdin. Same as inp(stdin,string=True). @param stdin the stdin file @returns selfTrrrr r r __lshift__/szRunner.__lshift__cCs|j|ddS)z!Appends stdout to the specified file. Same as out(stdout,append=True). @param stdout the stdout file @returns selfTrcrrr r r __rshift__5szRunner.__rshift__cCs|S)zB!Sends stderr to stdout. Same as err2out(). @returns self)rbr&r r r __pos__;szRunner.__pos__cCs|j|ddS)z!Redirects stderr and stdout to the specified file, truncating it. Same as err2out().out(filename,append=False) @param outerr the stdout and stderr file @returns selfFrc)rbra)r'Zouterrr r r __ge__?sz Runner.__ge__cCs ||S)z!Pipes this Runner to the other Runner. Same as pipeto(other). @returns other @param other the other runner to pipe into)pipetorYr r r __or__Esz Runner.__or__cCs|j|||S)a!Inserts the specified argument before the given index. This function is intended for internal use only. It is used to implement threading on Cray, where arguments relating to threading have to be added after the Runner is generated. @warning It is generally not safe to call this function outside the produtil.mpi_impl subpackage since its modules may generate completely different commands than you asked in order to execute your requested programs. @param arg a string argument to add @param index the index to insert before @note Index 0 is the executable, while later indices are arguments.)rpinsertr'indexrr r r arginsKsz Runner.arginsccs|jD] }|VqdS)z;!Iterates over the executable and arguments of this commandN)rprr r r rv^s z Runner.argsNcCs|dkr t}|dk st|t|j}|j|_|jdk rD|j|_|jdk rZ|j|_|jdk rp|j|_|j dk rt |j |_ |j dk r|j |_ |j dk rt|_ |j D]}|j |q|dk stt||st|S)a!Returns a deep copy of this object, almost. If stdin, stdout or stderr are connected to streams instead of files or strings, then the streams are not copied. Instead, the exact same stream objects are connected to the same unit in the new Runner. @param typeobj the type of the new object or None for Runner. Do not set this unless you know what you're doing. @returns the new objectN)r\r=rorprrrgr2rhrirkrqrjrlrdrX)r'typeobjr:pr r r r2cs,          z Runner.copycCs d|_|S)zp!Instructs this command to duplicate the parent process environment (the default). @returns selfT)rrr&r r r copyenv~szRunner.copyenvcCsd|_i|_|S)z!Instructs this command to start with an empty environment except for certain critical variables without which most programs cannot run. (Retains PATH, USER, LOGNAME and HOME.) @returns selfF)rrrkr&r r r r]szRunner.clearenvcCsx|jdkr|jrdSi}|jr*ttj}n&i}dD]}|tjkr2tj|||<q2|jdk rt|jD]}|j|||<q`|S)an!This internal function generates information about the environment variables to be input to this process. If the parent environment is to be passed unmodified, None is returned. Otherwise, this routine returns dict of environment variables calculated from os.environ and internal settings. @returns the new environment dictNPATHUSERLOGNAMEHOME)rkrrrqr>environ)r'r^keyr r r _impl_make_envs    zRunner._impl_make_envcCs|jdkrt||j|S)N)rkKeyErrorrr r r getenvs z Runner.getenvcKs4|jdkri|_|D]}t|||jt|<q|S)z!Sets environment variables for this Runner. The variables should be specified as keyword arguments. @param kwargs varname=value arguments @returns selfN)rkru)r'rwrr r r r^s  z Runner.envcCs|jdk r|jd}n|jdk r0|j}nd}|jdk rT|dt|jd7}|jdk sf|jstd|j drt d|j df|d 7}|js|d 7}|jD]<}td|rt d |f|d td ||j|f7}q|jsdD]}||jkr|d||f7}q|d 7}|d dd|j D7}|j dk rR|d |j 7}|jdk rp|d |j7}|jdk r|d7}|S)z!Returns a string that expresses this object as a POSIX sh shell command if possible, or raises a subclass of NotValidPosixSh if not.Nz | r3z( set -e ; cd z ; exec =rzM%s: cannot have an "=" in the executable name when modifying the environment.r^z -iz!%s: variable name contains an "=" rrz "%s=$%s"cSsg|] }t|qSr )r"rr r r rsz#Runner.to_shell..z ))rjr9rgrmr"rkrrrrrprrrrhri)r'rrr r r r9sH            zRunner.to_shellcCs|S)z!Returns self if self is modifiable, otherwise returns a modifiable copy of self. This is intended to be used to implement unmodifiable subclasses of Runner @returns selfr r&r r r runnersz Runner.runnercCst|tstd|jdk r$td|jdk r6td|jdk rR|jdk rRtd|}||_|jdk r|| |jd|_|S)a:!Specifies that this Runner will send its stdout to the other runner's stdin. This will raise MultipleStdout if this Runner's stdout target is already specified, or MultipleStdin if the other's stdin is already specified. @param other the runner to pipe into @returns otherzXAttempting to pipe a Runner into something that is not a Runner (likely a syntax error).NzFAttempted to pipe more than one process into stdin of the same processz-More than one stdout is detected in prog|progz,More than one stdin is detected in prog|prog) rXr\rrjrrhrrgrr)r'rZZrotherr r r rs      z Runner.pipetoFcCs|jdk r|j|||S|jdk r.tdt|tsLt|tsLt|trT||_n:|rhtt||_n&t|trtt|d|_n t||_|S)a!Specifies that the first Runner in this pipeline takes input from the given file or string specified by stdin. If string=True, then stdin is converted to a string via str(), otherwise it must be a filename or a stream. Raises MultipleStdin if the stdin source is already specified. @param stdin the input file or string @param string if True, stdin is a string. Otherwise, it is a file. @returns selfNz*More than one stdin detected in Runner.inpr7) rjrrgrrXrLr-rTrur'rrr r r rs    z Runner.inpcCsR|jdk rtdt|trD|r2tt|d|_qNtt|d|_n t||_|S)a!Specifies that this process sends output from its stdout stream to the given file or stream. The stdout object must be a string filename, or a stream. If append=False, and the stdout is a filename, the file will be truncated, if append=True then it is appended. Raises MultipleStdout if the stdout location is already specified @param stdout the stdout file @param append if True, append to the file, otherwise truncate @returns selfNz3More than one stdout detected in call to Runner.outr5r6)rhrrXrur-rTr'rrdr r r ras   z Runner.outcCs|jdk rtdt|_|S)z-!Sends stderr to stdout @returns selfN3More than one stderr detected in call to Runner.err)rirrVr&r r r rb0s  zRunner.err2outcCsV|jdk rtdt|trH|r4tt|dd|_qRtt|dd|_n t||_|S)a!Specifies that this process sends output from its stderr stream to the given file or stream. The stderr object must be a string filename, or a stream. If append=False, and the stderr is a filename, the file will be truncated, if append=True then it is appended. Raises MultipleStderr if the stderr location is already specified. @param stderr the stderr output file @param append if True, append to the file otherwise truncate @returns selfNrr5Tr6)rirrXrur-rTr'stderrrdr r r r09s   z Runner.errc Cs|jdk r|jj|||dn$|dk rB|dt|t|jf|jdk rd|jD]}|||dqR|dk r|dt|t|t|jdfi}|jdk r|j\}}}} |dk r||d<|dk r||d<|dk r||d <| dk r| |d <|j dk rT|j \}}}} |dkst |dk r0||d <|dk rB||d <| dk rT| |d <|j dk r|j \}}}} |dkst |dk r||d<|dk r||d<| dk r| |d<|j dk r| |d<|dk r||d<|j|j|dkfd|ji|dS)a.!Populates a Pipeline object with information from this Runner. This is a recursive function that starts at the last element of the pipeline (output element) and walks back to the first (input element). The "next" parameter points to the next (output-direction) element of the pipeline. The optional logger parameter is where to send log messages. @param[out] pipeline the produtil.pipeline.Pipeline @param logger a logging.Logger for log messages @param next the next Runner in the pipelineN)loggernextzGEN %s: recurse to %s)rzGEN %s: gen to %s with cmd=%srinstringrsendincloseinrsendoutcloseoutrsenderrcloseerrr^rre)rj_gendebugr rlrurprgrFrhr=rirkr _impl_addrm) r'pipelinerrrfrwrstreamsendcloser r r rOsd                z Runner._gen)N)F)F)F)NN)+rr r r r1ryr}r~rKthreadsrrrfrrrrIr[rrerrrrrrrrrvr2rr]rrr^r9rrrrarbr0rr r r r r\,sP X $   $   r\c@seZdZdZddZddZd,ddZd d Zd d Zd dZ ddZ ddZ ddZ ddZ d-ddZd.ddZd/ddZddZd d!Zd"d#Zd$d%Zd0d&d'Zd(d)Zd*d+ZdS)1ImmutableRunnera!An copy-on-write version of Runner. This subclass of Runner is unmodifiable. It is meant to be used for re-usable exe()-like objects. For example, if one wants an object lsl that runs exe('ls')['-l'] with optional extra arguments, one could do: lsl=ImmutableRunner(Runner('ls')['-l']) and then every time one does run(lsl[argument list]), it generates a new object without modifying the original lsl, ensuring later calls to lsl will have the same effect: lsl['/'] lsl['~'] lsl['/'] # prints the same as the first This is implemented by a copy-on-write method: if a modification is requested, a Runner is returned with the requested modifications.cKs>z0d|_tj||f||jdk r.t|j|_W5d|_XdS)z!Creates a new ImmutableRunner. All arguments to this constructor have the same meanings as the Runner constructor. @param args,kwargs passed to Runner.__init__FTN)_initr\r1rjr)r'rvrwr r r r1s  zImmutableRunner.__init__cCs |S)z2!Removes all prerun objects. @see prerun()) _init_runnerrr&r r r rszImmutableRunner.remove_prerunNcCs|dkr t}t||S)aH!Creates a deep copy of this runner, except if stream objects are connected to stdin, stdout or stderr. In that case, those same stream objects are still connected. @param typeobj the type of the output object. Do not use this unless you know what you're doing @returns a copy of selfN)rr\r2)r'rr r r r2szImmutableRunner.copycCs |tS)z;!Returns a modifiable version of this object (as a Runner).)r2r\r&r r r rszImmutableRunner.runnercCs2|jr |}n|}|dk s tt|ts.t|S)z!Do not call this function: it is an internal implementation function. It returns self if self.__init__ is still being run, otherwise it returns self.runner().N)rrr=rXr\)r'rr r r rs  zImmutableRunner._init_runnercCs |S)z!Creates a new Runner that is like self in all ways except that it uses the parent process environment. @returns the new Runner)rrr&r r r rszImmutableRunner.copyenvcCs |S)a!Creates a new Runner which is like self in all ways except that it uses an empty environment except for a few critical variables without which most programs cannot run. (Retains PATH, USER, LOGNAME and HOME.) @returns a new Runner)rr]r&r r r r]szImmutableRunner.clearenvcCs||S)a!Returns a new Runner that is like self, except that it cd's to the target directory before running. The directory must already exist before the program starts. @param cd the directory to cd into, which must already exist. @returns the new Runner)rre)r'rer r r reszImmutableRunner.cdcKs|jf|S)z!Returns a new Runner that is like self in all ways except that the specified environment variables are set. @param kwargs varname=value arguments of environment variables to set @returns the new Runner)rr^)r'rwr r r r^szImmutableRunner.envcCs||S)z!Returns a new Runner that is like self in all ways, except that it has been piped into the other Runner. @returns the new Runner @param other the Runner to pipe into.)rrrYr r r rszImmutableRunner.pipetoFcCs|||S)z!Returns a new Runner that is like self in all ways except that it has a different stdin @param stdin the stdin string or filename @param string if True, stdin is a string)rrrr r r rszImmutableRunner.inpcCs|||S)z!Returns a new Runner that is like self in all ways except with a different stdout. @param stdout the stdout filename @param append if True, append to the file, otherwise truncate)rrarr r r raszImmutableRunner.outcCs|||S)z!Returns a new Runner that is like self in all ways except with a different stderr. @param stderr the stderr filename @param append if True, append to the file, otherwise truncate)rr0rr r r r0szImmutableRunner.errcCs |S)zd!Returns a new Runner that is like self in all ways except that stderr is piped into stdout.)rrbr&r r r rbszImmutableRunner.err2outcCs||S)z!Returns a new Runner that is like self in all ways except that a new prerun function has been added. @param arg the new prerun function @sa Runner.prerun())rrfrr r r rfszImmutableRunner.preruncCst||S)z!Returns a new Runner that is like self in all ways except with new arguments. @param args the new argument or arguments @sa Runner.__getitem__)r\rrrr r r rszImmutableRunner.__getitem__cCs|||S)z!Returns a new Runner that is like self in all ways, except with the specified argument inserted. @param index the index to insert before @param arg the argument to insert)rrrr r r rszImmutableRunner.arginscCs||||S)a0!Creates a Runner object that is a duplicate of this ImmutableRunner, and calls its _gen function. @param pipeline the produtil.pipeline.Pipeline to generate @param logger a logging.Logger for log messages @param next the next Runner in the chain @sa Runner._gen())rr)r'rrrr r r r szImmutableRunner._gencCs|}||_|Srzrr)r'r|r:r r r r}szImmutableRunner.setthreadscCs|}|`|S)z9!Removes the request for threads. Same as self.threads=1r)r'r:r r r r~szImmutableRunner.delthreads)N)F)F)F)NN)rr r r r1rr2rrrr]rer^rrrar0rbrfrrrr}r~r r r r rs*       r)(r produtil.sigsafetyprodutilioselectrtimefcntlr>loggingsignalprodutil.mpi_implprodutil.pipelinerrrr Exceptionrrrrrrrrrrrrrr"objectr#r-rLrTrVr\rr r r r s:'H   G#Z