gc"@sdZddddddddd d d d d ddddddddddddddddddd d!d"g"Zd#d$lZd#d$lZd#d$lZd#d$lZd#d$lZd#d$lZd#d$lZd#d$l Z d#d$l Z d#d$l Z d#d$l Z d#d$l Z d#d$lZd#d$lZe jd%Zdefd&YZdefd'YZdefd(YZdefd)YZdefd*YZdefd+YZdefd,YZdefd-YZd.efd/YZd efd0YZd efd1YZd2Zd$d3Z!d$d4Z"d5Z#d6Z$d7d$d8Z%e&d$d9Z'd:Z(e)d;Z*d<Z+e&e)d=d$d>e&d$e&e&d$d$e&e&d$d?Z,d$e&d@Z-dAd$dBZ.e)d$d$e)dCZ/e)d$dDdEZ0d$dDdFZ1d$dGZ2d$dHZ3e)d$d$dIZ4d$d$e)dJZ5d$e)dKZ6d$dLZ7dMd$dNZ8d$d$d$d$d$dOZ9ddVdPYZ:d$dQdDdRdSd$d$dTe&dU Z;d$S(Ws*!This module provides a set of utility functions to do filesystem operations. It replaces or improves upon several os, stat, and sys module functions by working around Python bugs, providing an API layer that allows forward compatibility to future Python versions, and adding logging capabilities.t FileOpErrort FileOpErrorstCannotLinkMultitUnexpectedAbsolutePathtInvalidExecutabletFindExeInvalidExeNamet CannotFindExetRelativePathErrortDeliveryFailedtVerificationFailedtrealcwdtchdirtmakedirst remove_filetrmallt lstat_statt isnonemptyt check_filet deliver_filetmake_symlinks_intfind_exet make_symlinktreplace_symlinktunblocktfortcopytnorm_expand_patht norm_abs_pathtcheck_last_linestwait_for_filest FileWaitert call_fcntrltgribvert netcdfverttouchiNsprodutil.fileopcBs,eZdZgdZdZdZRS(sg!This is the superclass of several exceptions relating to multi-file operations in produtil.fileop.cCs||_||_||_dS(s<!FileOpError constructor @param message the error message @param filename the name of the problematic file @param more a list of tuples, (from,to,message) where from is the source file, to is the destination file and message is a description of the problem with that pair.N(tmessagetfilenametmore(tselfR"R#R$((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyt__init__s  cCsd|j|jfS(s%!A string description of the problem.s%s: %s(R#R"(R%((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyt__str__4sccs/x(|jD]\}}}|||fVq WdS(s!Iterates over a list of tuples, (from,to,message) where from is the source file, to is the destination file and message is a description of the problem with that pair.N(R$(R%tfromfilettofileR"((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyt__iter__7s(t__name__t __module__t__doc__R&R'R*(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRs  cBseZdZRS(sn!This exception is raised when an operation that processes multiple files catches more than one exception.(R+R,R-(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR=scBseZdZRS(s!This exception is raised when the caller tries to create multiple symlinks in a single target, but the target is not a directory.(R+R,R-(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR@scBseZdZRS(sk!This exception indicates that the renamer function sent to make_symlinks_in returned an absolute path.(R+R,R-(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRDscBseZdZRS(s!Thrown when a find_exe fails.(R+R,R-(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRGscBseZdZRS(sU!Thrown when find_exe is given an executable name that contains a directory path.(R+R,R-(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRIscBseZdZRS(sR!Thrown when find_exe cannot find an executable in the path or directory list.(R+R,R-(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRLscBseZdZRS(sL!Raised when a relative path is given, but an absolute path is expected.(R+R,R-(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyROst WrongSymlinkcBseZdZRS(s!Raised when os.symlink makes a symlink to a target other than the one that was requested. This is present to detect a bug in Cray where os.symlink randomly makes a symlink to the wrong place.(R+R,R-(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR.SscBs)eZdZdZdZdZRS(s:!This exception is raised when a file cannot be delivered.cCs||_||_||_dS(s!DeliveryFailed constructor. @param message why the delivery failed @param fromfile what was being delivered @param tofile delivery destinationN(R"R(R)(R%R"R(R)((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR&[s  cCsd|j|j|jfS(s*!Human-readable description of this error.s %s: cannot deliver (from %s): %s(R)R(R"(R%((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR'lscCs,dt|jt|jt|jfS(s'!Pythonic representation of this error.sDeliveryFailed(%s,%s,%s)(treprR"R(R)(R%((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyt__repr__ps(R+R,R-R&R'R0(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRYs  cBs)eZdZdZdZdZRS(s\!This exception is raised when a copy of a file has different content than the original.cCs#tj||||||_dS(s!VerificationFailed constructor. @param message why the delivery failed @param fromfile what was being delivered @param tofile delivery destination @param verifyfile temporary file that failed verificationN(RR&t verifyfile(R%R"R(R)R1((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR&xscCs d|j|j|j|jfS(s*!Human-readable description of this error.s:%s: verification failed on temporary file %s (from %s): %s(R)R1R(R"(R%((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR'scCs8dt|jt|jt|jt|jfS(s'!Pythonic representation of this error.sVerificationFailed(%s,%s,%s,%s)(R/R"R(R)R1(R%((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR0s(R+R,R-R&R'R0(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR us cCstjjtjS(sI!Returns the current working directory, expanding any symbolic links.(tostpathtrealpathtgetcwd(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR scCs{t|}y1|dk r/|j|dntj|Wn7tk rv}|j|dt|dtnXdS(s!Changes to the specified directory. Please use produtil.cd.NamedDir instead. This is generally not a good idea since you will not cd back if an unhandled exception is raised. It is better to use the produtil.cd module, which provides ways to enter a directory in a "with" block and optionally delete it afterwards. Such functionality could also be implemented via a try...finally block. @param path the path to cd to @param logger a logging.Logger for log messagess : cd heres : cannot cd: texc_infoN(tstrtNonetinfoR2R tEnvironmentErrortwarningtTrue(R3tloggerte((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR s  !cCs*t|dtj||WdQXdS(sT!Open the file for append and set mtime and atime. Opens the specified file in append mode, but writes nothing. Sets the access and modification times. @param filename the string filename @param times A 2-tuple of numbers, of the form (atime, mtime). These are UNIX epoch times (seconds since 1970 began in UTC).taN(topenR2tutime(R#ttimes((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR!s cCs~t|di}|jd}t|dkr7d S|dd!}|dkrTdS|dkrddS|d krtd SWd QXd S( s!What is the NetCDF version of this file? Returns one of three strings based on the NetCDF version of the given file, or returns None if the file is not NetCDF: * "CDF1" = NetCDF classic format * "CDF2" = NetCDF 64-bit offset format * "HDF5" = HDF5 file, and hence possibly a NetCDF4 file. * None = Not NetCDF and not HDF5 @param filename the name of the file to testtrbiiisCDFtCDF1sCDFtCDF2sHDF  tHDF5N(R@treadtlenR8(R#tfteighttfour((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR s      cCst|ts7tdt|jt|fnt|sGd St|dA}|j d}|dkrxdS|dd!dkrd Sd SWd QXd S( s!What is the GRIB version of this file? Returns the GRIB file version: 1 or 2. If the file is not a GRIB file, or if the answer is indeterminate, returns None. Only the first GRIB record is tested. @param filename the path to the file to testsJThe first argument to gribver should be a filename. You provided a %s %s.RCisGRIBiiitGRIBiN( t isinstanceR7t TypeErrorttypeR+R/RR8R@RG(R#RIRJ((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRs"  i cCsxt|D]}yFtjj|sX|dk rH|j|dntj|nWq tk r}tjj|rtStjj |rn||dkrq nq Xq WdS(s!Make a directory tree, working around filesystem bugs. This makedirs implementation works around a common bug: if two processes try to recursively make a directory tree simultaneously, makedirs can fail when two processes make the same path component at the same time. This implementation automatically retries in that situation. @param filename the directory path @param numtries the number of times to retry @param logger a logging.Logger for log messagess: make directory and parentsiN( trangeR2R3tisdirR8R9R R:R<texists(R#tnumtriesR=tnR>((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR s  cCs|dks+|dks+tjj| r/dSy4|dk rU|jd|fntj|Wntk r}|jtjkr|dk r|j d|t |fdt nn|dk r|r|jn|j }|d|fqnXdS(s!Deletes the specified file. Does nothing if the filename is None, is the empty string or already does not exist. Otherwise, the file is deleted. @param filename The file to delete. @param info Optional: indicates that warnings about a file already not existing should be sent to the logger at INFO level (info=True) instead of WARNING (info=False). @param logger the logging.Logger for messagestNs%s: remove files%s: cannot remove: %sR6s"%s: cannot remove; does not exist.( R8R2R3RRR9tunlinkR:terrnotENOENTR;R7R<(R#R9R=R>tlog((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR s +    c Os_|jdd}|dk r;|jdt|fnt}xK|D]C}yt||WqKtk r}|j|d|fqKXqKWt|dkr|ddn}t|dkr2d}|dk r|j|nt |dj |g|D]$\}}} ||t | f^qn|dk r[|jdt|fndS( s!Deletes the specified list of files. Deletes files listed in "args". Each one is passed to remove_file. Exceptions that derive from EnvironmentError are collected, and will be raised at the end, thus allowing removal of later files to continue if earlier ones failed. If only one file causes an exception, that exception will be raised, otherwise FileOpErrors will be raised @param args The files to delete. @param kwargs Keyword arguments passed to remove_file().R=sRemoving %d files...iis9Multiple exceptions caught while deleting files in rmall.t,sDone removing %d files...N( tgetR8R9RHtlistR R:tappendR;RtjoinR7( targstkwargsR=textargR>tmsgR?tbtc((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRs&     7 cCs|dk std\}}y>tj|}tj|jsL||fStj|}Wn1tk r}|s|jtj krqnX||fS(s!Runs lstat and stat on a file as efficiently as possible. Returns (lstat(filename),stat(filename)) where each is None if it fails due to non-existence. Does this in as few filesystem metadata operations as possible. Will raise an exception if the stat fails for any reason other than non-existence of a file, or if the file or linked file is non-existent and raise_nonexist=True. @param filename The file to test. @param raise_nonexist Should we raise an exception if the file does not exist? @returns a tuple (L,S) where L is the lstat return value, and S is the stat return value. Each will be None if the file or link target do not exist.N(NN( R8tAssertionErrorR2tlstattstattS_ISLNKtst_modeR:RWRX(R#traise_nonexisttxlstattxstatR>((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR4s   cCsi|dkrdSt|}|dks4|dkr8dSt|\}}|dk ob|jdk}|S(s!Returns True if the filename refers to an existent file that is non-empty, and False otherwise. @param filename The file to test.RUiN(R8R7Rtst_size(R#tsfiletltstret((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRNs  iic Cs| d krtjj } n| d kr=tjj} n|d k rRt}n|d k sdt|d k svttjj |}t |dt\}}t j |j rtd||n|}d }t |\}}d \}}|d k rut j |j ru|}tjj||}||}}|d k r]|jd||fnt |\}}qun|d kr7tjj|}t|dkrd}n|d k r|jd||fnt |dt\}}|d kr td||nt j |j s7td||q7n|d k rp| rp|d k rp|jd |fqpn|d k rt j|j r|d k r|jd |fqn?tjj||r|d k r|jd ||fnd S|d k r<|jd ||fq<n |d k r<|jd|n|j|jk}|r| r| rt j|j r|d k r|jd|fqq|d k r|jd||fnytj||d SWqtk r}|d k r|jd|t|fqqXnd }d }zy:|d krKd|d}ntjd|dtd|}|j}|d k r|jd||fn|d krt|d}tj ||d|Wd QXn|||||j!d }|rN|d k r!|jd||fnt"j#||sNt$d|||qNn|d k rt|jd||fn| rytj%|d|j&Wqt't(fk r}|d k r|j)d||t|fqqXn| rtj*||j |@n|d k r)|jd||fn| rNtj+||j,|j-fntj||d }WnBt.k r}|d k r|j/d|t|fnnXWd y?|d k r|j!n|r|d k rtj0|nWntk r}nXXd S(!s0 !This moves or copies the file "infile" to "outfile" in a unit operation; outfile will never be seen in an incomplete state. If the caller specifies keep=False (default is True) and moveok=True, and the source and destination are on the same filesystem then the delivery is done with a simple move. Otherwise a copy is done to a temporary file on the same filesystem as the target. If verification is requested (verify=True) then the temporary file is verified by filecmp.cmp, before moving the temporary file to the final location. When requested, and when possible, the permissions and ownership are preserved. Both copy_acl and preserve_group have defaults set by the produtil.cluster module. If the cluster uses access control lists for data restriction classes, then copy_acl will be set to True, otherwise it is false. If group quotas are enabled, preserve_group is False, otherwise it is True. @note The original file is never deleted, but it may be moved to the target if keep=False. If a copy is done instead, the original file is still present. @param infile the origin file @param outfile the destination file or its parent directory @param keep If False, the original file is no longer needed. If False and moveok=True, the file might be delivered by a "mv" operation, avoiding any data duplication (no "cp"). @param verify If a "cp" is done, reopen the target and source and verify they are the same. Note that providing a copier will break the verification functionality if the copier changes the contents of the destination file (such as a copier that compresses). @param blocksize block size during copy operations @param tempprefix Prefix for temporary files during copy operations. Do not include directory paths in the tempprefix. @param permmask Permission bits to remove Default: world write (002) @param removefailed If True, delete temporary files if the delivery fails @param logger the logging.Logger for log messages. @param preserve_perms If True, copy the old file's permissions to the new file @param preserve_times If True, copy the old file's timestamps to the new file @param preserve_group If True, copy the old file's group ID to the new file @param copy_acl If True, copy the access control lists from one file to another @param moveok If True, delivery by "mv" is allowed. Must also set keep=False. @param force If False, delivery will be aborted (raise TargetFileExists) if the target file already exists. @param copier If present, this function or callable object is used to copy data from the source file to the temporary file before moving it to the target. The copier is called as copier(infile,temp_file_name,temp_file_object) Where the temp_file_name is the name of the destination file and the temp_file_object is an object that can be used to write to the file. The copier should NOT close the temp_file_object. Rks+This subroutine cannot deliver directories.s%s: is a directory; file is %sit.s6%s: exists, so parent %s must exist and be a directorys;Target does not exist, and parent of target does not exist.s?Target does not exist, and parent of target is not a directory.sA%s: exists and overwrite (force) is disabled. Aborting delivery.s5%s: destination is a link, will recopy as a non-link.s%s: same as %sNs%s: exists, replacing with %ss%s: does not exists:%s: cannot deliver via "os.rename" since source is a link.s%s: move from %ss&%s: could not deliver by os.rename: %sstmp.s.part.tprefixtdeletetdirs%s: copy to temporary %sRCtlengths%s: verify copy %ssfilecmp.cmp returned Falses'%s: copy group ID and permissions to %sis!%s: cannot copy groupid to %s: %ss%s: delivery failed: %s(NN(1R8tprodutiltclustert group_quotastuse_acl_for_rstdataR<RfR2R3tbasenameRRhtS_ISDIRRjRR^tdebugtdirnameRHRiR9tsamestattst_devtrenameR:R7ttempfiletNamedTemporaryFiletFalsetnameR@tshutilt copyfileobjtclosetfilecmptcmpR tchowntst_gidtIOErrortOSErrorR;tchmodRAtst_atimetst_mtimet ExceptionterrorRV(tinfiletoutfiletkeeptverifyt blocksizet tempprefixtpermmaskt removefailedR=tpreserve_permstpreserve_timestpreserve_grouptcopy_acltmoveoktforcetcopiertinbasetilstattistattactual_outfiletoutdirtoflstattofstattodlstattodstattsamefsR>ttempttempnametindata((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRZs=                                      !       cCstjj|}||kr7td|f|n|dkr\tjdjd}nxc|D][}|dkr~d}ntjj||}tjj|rctj |tj rc|SqcW|sdSt d|dS(s!Searches the $PATH or a specified iterable of directory names to find an executable file with the given name. Returns the executable's location. If the executable cannot be found, and raise_missing=True, raises CannotFindExe, otherwise returns None. Raises FindExeInvalidExeName if "name" is not the same as its os.path.basename. @param name The name of the executable to find. @param dirlist The list of directories to search, or None to search $PATH @param raise_missing If True, the CannotFindExe exception is raised for executables that cannot be found. If False, return None in that situation.sIexecutable name is not the same as its basename in find_exe (basename=%s)tPATHt:RURsscannot find executableN( R2R3R|RR8tenvirontsplitR^tisfiletaccesstX_OKR(Rtdirlistt raise_missingtbnRtexename((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR4s     'i(c CsqyWt|dB}|j|}|dk rP|jd|t|fnWdQXWntk rl}nXdS(sb!Opens the specified file for reading and attempts to read data to it. Logs the process. Will NOT raise any I/O or system errors; they are ignored. This is a workaround for a bug in Cray: symlinks to recently created files cannot be read by the compute node unless the batch node reads from them first (or unless you wait a while).trts %s: read %sN(R@RGR8R9R/R:(R#treadsizeR=RItbufR>((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pytsymlink_read_testSs *c Cs t|dt\}}tj|js<td|ng}x|D]} d } y)|d k r|| } | d krwIntjj | rt dt ntjj || } n!tjj |tjj | } |rdtjj| r;|rtjj| rtj| nt| | dtd|q}|d k r}|jd| | fq}nt| | d|d|WqItk r} |j| t| t| f|d k r|jt| dtqqIXqIWt|d kr td ||nd S( s!Creates symbolic links from a set of source files to a target directory. If "force" is True, then any existing files will first be deleted. The "renamer" can be a function that generates paths of the symlinks, relative to targetdir, for each symlink in "sources". If the return value from "renamer" is an absolute path, an exception will be thrown. If the return value is None, then no link will be made. Example: make_symlinks_in(['/path/to/a','/path/to/b'],'.', renamer=lambda s: os.path.basename(s)+'.linkified') will create a.linkified, linked to /path/to/a, and b.linkified, linked to /path/to/b in directory "." @param sources The list of files to link to. @param targetdir The directory in which to place the links. @param force Remove existing files if needed. @param renamer Function to generate link names. @param logger A logging.Logger for log messages. @param copy If True, files are copied instead of linked.Rkstarget is not a directorysrenamed path is absoluteRR=s:%s: skip. Am in copy mode, and source does not exist (%s)RR6iscannot link filesN(RR<RhR}RjRR8R2R3tisabsRtrenamedR^R|RRRVRR;RR:R]R7RHR( tsourcest targetdirRtrenamerR=tcopyttlstatttstatterrorstsourcettargetR>((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRbs@    ! " $icCs|dkrt}n|dk r;|jd||fntjj|s}|dk rm|jd|ntjdntjj |rtjj |tjj |}|dk r|jd|fqnyptj ||tj |}||krEd|||f}|dk r3|j|nt||nWn_tk r}|jtjk sr| rxn|dk r|jdnt||d|SXdS( s!Creates a symbolic link "target" that points to "source". If the target already exists and is NOT a directory, then the file will be replaced. The replacement is done in a unit operation so that the target will always exist (unless the operation fails). @param source The file to link to. @param target The name of the link. @param force If True, and target exists, delete it first. @param logger a logging.Logger for log messages.s link %s -> %ss-FATAL ERROR: source: "%s" not exist, Exiting.is-Target is a directory. Symlink to %s insteads_FATAL ERROR: FILESYSTEM FAILURE: Cannot symlink "%s" -> "%s". Instead, the symlink is to "%s".s-target exists - using replace_symlink insteadR=N(R8t module_loggerR9R2R3RRtcriticaltsystexitRQR^R|tsymlinktreadlinkR.R:RWtEEXISTR(RRRR=t max_triestcontentRcR>((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRs6   !    cCs|dkrt}ntjjtjj|dtjj|tjdtjdf}y|dk r|j d||fntj ||tj |}||krd|||f}|dk r|j |nt ||n|dk r|j d||fntj||Wnvtk r}t|t rTny4|dk rz|j d|fntj|Wntk rnX|nXdS(s|!Do not call this routine directly: you want make_symlink instead. This routine creates a new symbolic link and renames that link to "target." That always replaces target with a symbolic link to source, even if target did not already exist. @param source the file to link from @param target the file to link to @param logger a logging.Logger for messagesstmp.%s.%06x.%06x.tmpi s link %s -> %ssRFILESYSTEM FAILURE: Cannot symlink "%s" -> "%s". Instead, the symlink is to "%s".srename %s to %ssfailed: delete %sN(R8RR2R3R^RR|trandomt getrandbitsR9RRRR.RRRMtremoveR:(RRR=RRRRcR>((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRs8 "      cCsttjd|dS(s+!Attempts to modify the given stream to be non-blocking. This only works with streams that have an underlying POSIX fileno, such as those from open. Will re-raise any exception received, other than AttributeError and EnvironmentError. Hence, I/O errors and attempts to make a non-fileno stream non-blocking will produce a False return value, while anything else will raise an exception. @param stream the stream to unblock @param logger a logging.Logger for log messages @returns True on success, False otherwise.iN(RR2t O_NONBLOCK(tstreamR=((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRs cCs y(t|tr|}n |j}WnRttfk r|}|dk rx|jdt|t|fdt nt SXy<t j |t j }t j |t j ||B|@t SWnLtk r}|dk r|jdt|t|fdt nt SXdS(s!Internal function that implements unblock() @param stream the stream to modify @param on flags to turn on @param off flags to turn off @param logger a logging.Logger for messages @returns True on success, False otherwise.s?%s: stream has no fileno, cannot switch to non-blocking I/O: %sR6s)%s: cannot switch to non-blocking I/O: %sN(RMtinttfilenotAttributeErrorR:R8R;R/R7R<RtfcntltF_GETFLtF_SETFLR(RtontoffR=tfdteetflags((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRs$   #  #cCs|dk rD|jdt|t|t|t|fnx|jD]q\}}t|tsrtdt|f}|dk rtj j |t }nt ||d|d|qQWdS(s!This is a convenience routine that makes many symbolic links to fort.N files for various integers N using make_symlink. It works similarly to fortcopy. The optional basedir is the relative directory. The optional force argument is passed on to make_symlink and has the usual meaning: replace existing files. Call like this: @code fortlink({ 15:"/usr/local/share/file1", 23:"./file2"}) @endcode And you will create these symbolic links: @code{.unformatted} ./fort.15 -> /usr/local/share/file1 ./fort.23 -> ./file2 @endcode as with other symlink routines in this module, set force=True to remove target fort.N files if they already exist. @param forts Mapping from Fortran unit number to link target. @param force Remove target files if they exist. @param basedir Where to make the links instead of the current directory. @param logger A logging.Logger for log messages.s3in fortlink, forts=%s force=%s basedir=%s logger=%ssfort.%dRR=N( R8R~R/titemsRMR7RfRR2R3R^twhereR(tfortsRtbasedirR=tiR#tlink((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pytfortlinks  / c Ksx|jD]\}}dt|f}|dk rPtjj|t}nyt||d||Wq tk r}|dk r|j d||t |f|sqqq Xq WdS(sA!A convenience function for copying files to local fort.N files for various integers N using deliver_file(...,keep=True). It works similarly to fortlink. The force= argument tells fortcopy to overwrite existing files. Otherwise, an exception will be raised if the destination file already exists. The optional basedir argument is the parent directory of the fort.N. Call like this: @code fortcopy({ 15:"/usr/local/share/file1", 23:"./file2"}) @endcode And you will create files: @code{.unformatted} ./fort.15 (copied from /usr/local/share/file1) ./fort.23 (copied from ./file2) @endcode All other keyword arguments are sent to deliver_file. @param forts Mapping from Fortran unit number to copy target. @param basedir Where to put the files instead of the current directory. @param logger A logging.Logger for log messages. @param only_log_errors Only log failed operations instead of logging everything. @param kwargs All other keyword arguments are passed to deliver_file()sfort.%dR=s%%s: fortcopy could not copy to %s: %sN( RRR8R2R3R^RRR:R;R7( RRR=tonly_log_errorsR`RR#tnewfileR((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR5s    cCs|dkrtj}ntjjtjj|}|r`tjjtjj|}ntjj|st d||fn|S(s}!Normalizes path and expand home directories. Calls os.path.normpath and os.path.expanduser on its argument, or on os.getcwd() if no argument is supplied (or if path=None). This removes extraneous a/./b, a/../b, expands ~username and ~, and other system-specific expansions. See the Python documentation of normpath and expanduser for details. Will also call realpath and normcase if fullnorm=True. Raises RelativePathError if the resulting path is not absolute. @param path the path to expand @param fullnorm If True, call os.path.normcase() and os.path.realpath() normapth and expanduser.s2%s: path is relative, not absolute (expands to %s)N( R8R2R5R3tnormpatht expandusertnormcaseR4RR(R3tfullnormR((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR_s !cCstjj|t|S(s!Return relative path. This routine generates relative file paths (using os.path.relpath) that are relative to the specified "from" directory fromdir. The fromdir will be first sent through norm_expand_path to eliminate system-specific weirdness, such as a/./b, a/../b, ~username and so on. This will raise RelativePathError if the resulting path is not absolute. @param rel the path @param fromdir the directory from which we want the relative path(R2R3trelpathR(treltfromdir((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRws i'c Cstt|d}y|j| tjWn?tk rq}|dk rr|jd|t|fqrnXd}x1|D])}|d7}|j|dkrt SqW|dk r|jd||fnt SWdQXdS(s!Checks the last few bytes of a file to see if the specified search string is present. Returns True if the string is present or False if the file existed but the string was not present. Will raise an exception if the file is non-existent or cannot be read. @param filename The file to search (a string). @param searchstr The string to search for. Must not contain end-of-line chars. @param lastbytes The number of bytes at the end of the file to check. Can be larger than the file size. @param logger A logging.Logger for log messages. @returns True if the file contains the given string, False otherwiseRs%s: probably not an error: %siis%s: read %d linesN( R@R7tseekR2tSEEK_ENDR:R8R9tfindR<R(R#t searchstrt lastbytesR=RIR>Rtline((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRs      c Csytj|}|j|krH|dk rD|jd|fntS|dk sl|dk sl|dk r\ttj}|dk r||j|ks|dk r|jd|fntSn|dk r||j |ks|dk r |jd|fntSn|dk r\||j |ksY|dk rR|jd|fntSq\n|dk r|jd|fnt SWnOt k r}|j t jkr|dk r|jd|fntSnXdS(s!Determines whether the specified file exists, and meets additional requirements. @param filename The file to analyze. @param min_size If present, the file must be at least this many bytes. @param min_mtime_age If specified, the file must have been modified more than this many seconds in the past. @param min_atime_age if specified, the file atime must be at least this many seconds old. The meaning of atime varies, but usually means the last access time. @param min_ctime_age If specified, the file ctime must be at least this many seconds old. The meaning of ctime varies between platforms and file types, but usually means the file creation or inode change time. See stat(2) for details. @param logger a logging.Logger for log messages. @note This routine can also be used on directories, but one should avoid the min_size option when doing that. @returns True if requirements are met, False otherwise. s %s: too smalls&%s: not old enough (modification time)s %s: not old enough (access time)s&%s: not old enough (inode change time)s%s: file meets requirementss%s: does not exist (ENOENT)N(R2RhRnR8R9RRttimeRRtst_ctimeR<R:RWRX( R#tmin_sizet min_mtime_aget min_atime_aget min_ctime_ageR=RqtnowR>((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRsH              cBsweZdZd d d d d ddZdZd dZdZdZdZ dZ d d d e d Z RS( s8!A class that waits for files to meet some requirements.g?cCswt|_t|_t|_||_||_||_||_t ||_ |dk rs|j |ndS(s!Constructor for the FileWaiter. Most arguments have the same meaning as check_file() @param flist the file or list of files to wait for. This is simply sent into self.add. @param min_size minimum file size @param min_mtime_age minimum modification time age, @param min_atime_age minimum access time age. @param min_ctime_age time since last file status change (see stat(2)) @param min_fraction the minimum fraction of the provided files that must match the above requirements in order for FileWaiter.wait to return True. Default is 1.0, which means all of them.N( R\t_flisttsett_fsett_foundRRRRtfloatt min_fractionR8tadd(R%tflistRRRRR((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyR&s        cCsgt|trE||jkr"dS|jj||jj|nx|D]}|j|qLWdS(s!Adds a file, or iterable that iterates over files, to the list of files to wait for. If the same filename is received a second time, it is ignored. @param flist a filename (string) or list of filenamesN(RMR7RRR]R(R%Rtfile((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRs cCs(t||j|j|j|jd|S(s!Checks to see if one file meets the requirements set in the constructor. This default implementation calls check_file. This is in a separate member function so that a subclass can override the file checking method. @returns True if the file is "ready," and False if it is not. @param filename the path to the file to check @param logger a logging.Logger for messagesR=(RRRRR(R%R#R=((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pytcheckscCst|_dS(sF!Resets internal information about which files have been seen.N(RR(R%((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pytreset sccsx|jD] }|Vq WdS(s)!Iterates over all files that were found.N(R(R%R#((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyt iterfound%scCs t|jS(s-!Returns the number of files that were found.(RHR(R%((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyt countfound)scCst|jt|jS(s1!Returns the number of files that were NOT found.(RHRR(R%((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyt countmissing,siicCst|}ttj}|}t}|r9|}nd}xddkrt|jdkr|dk r|jdntSt|jt|j} ttj}t|j} t|j} t| | } t j |j | } | |j dkr|jdtS|||kr:|jdt S|s9t dt||||d}|dkr|jdt S|dk r|jdt|jt|j|j d | | dkrd nd f|d kr|jn|j}|d t|fntj||dk r9|dq9nt }x|jD]~}||jkrdqIn|j|d|rI|jj||dk r|jd|t|jt|jfqqIqIWqBWt|jt|jkS(sw!Looks for the requested files. Will loop, checking over and over up to maxwait seconds, sleeping sleeptime seconds between checks. @param maxwait maximum seconds to wait @param sleeptime sleep time in seconds between checks @param logger a logging.Logger for messages @param log_each_file log messages about each file checkedisNo files to check.gh㈵>s Have required fraction of files.sWaited too long. Giving up.igMbP?sCStill need files: have %d of %d, but need %g%% of them (%g file%s).gY@RqRUisSleeping %g seconds...sDone sleeping.R=s$%s: found this one (%d of %d found).N(RRR<R8RHRR9RRtmathtceilRRtmaxtminR~tsleepRR R(R%tmaxwaitt sleeptimeR=t log_each_filetstartRtfirsttfloggertlefttnfilestnfoundtfract needfilestsleepnowtlogfunR#((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyt checkfiles0sd     $         "N( R+R,R-R8R&RR R R R RR<R!(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRs %     iiig?c Cs1t||||||} | j|||| S(s !Waits for files to meet requirements. This is a simple wrapper around the FileWaiter class for convenience. It is equivalent to creating a FileWaiter with the provided arguments, and calling its checkfiles routine. @param flist the file or list of files to wait for. This is simply sent into self.add. @param logger a logging.Logger for messages @param maxwait maximum seconds to wait @param sleeptime sleep time in seconds between checks @param min_size minimum file size @param min_mtime_age minimum modification time age, @param min_atime_age minimum access time age. @param min_ctime_age time since last file status change (see stat(2)) @param min_fraction the minimum fraction of the provided files that must match the above requirements in order for FileWaiter.wait to return True. Default is 1.0, which means all of them. @param log_each_file log messages about each file checked (RR!( RR=RRRRRRRRtwaiter((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pyRss ((<R-t__all__R2RRRRhRRWRRRRtloggingtprodutil.clusterRxtprodutil.pipelinet getLoggerRRRRRRRRRRR.RR R R8R R!R RR R<R RRRRRRRRRRRRRRRRRRRR(((s</lfs/h1/ops/prod/packages/hafs.v2.0.7/ush/produtil/fileop.pytsv        !            9&$  "* !7