Class NutchJob

    • Method Detail

      • cleanupAfterFailure

        public static void cleanupAfterFailure​(Path tempDir,
                                               FileSystem fs)
                                        throws IOException
        Clean up the file system in case of a job failure.
        Parameters:
        tempDir - The temporary directory which needs to be deleted/cleaned-up.
        fs - The FileSystem on which the tempDir resides.
        Throws:
        IOException - Occurs if there is fatal I/O error whilst performing the cleanup.
      • cleanupAfterFailure

        public static void cleanupAfterFailure​(Path tempDir,
                                               Path lock,
                                               FileSystem fs)
                                        throws IOException
        Clean up the file system in case of a job failure.
        Parameters:
        tempDir - The temporary directory which needs to be deleted/cleaned-up.
        lock - A lockfile if one exists.
        fs - The FileSystem on which the tempDir resides.
        Throws:
        IOException - Occurs if there is fatal I/O error whilst performing the cleanup.
        See Also:
        LockUtil.removeLockFile(FileSystem, Path)
      • getJobFailureLogMessage

        public static String getJobFailureLogMessage​(String name,
                                                     Job job)
                                              throws IOException,
                                                     InterruptedException
        Method to return job failure log message. To be used across all Jobs
        Parameters:
        name - Name/Type of the job
        job - Job Object for Job details
        Returns:
        job failure log message
        Throws:
        IOException - Can occur during fetching job status
        InterruptedException - Can occur during fetching job status