任务调度Azkaban(二)任务流

Command任务

  1. 创建tpye=command的job文件
    silence-pro:command silence$ ll
    total 40
    -rw-r--r--  1 silence  wheel  61  7  3 21:38 command1.job
    -rw-r--r--  1 silence  wheel  74  7  6 17:56 command2.job
    -rw-r--r--  1 silence  wheel  68  7  6 17:59 command3.job
    -rw-r--r--  1 silence  wheel  67  7  6 17:57 command4.job
    -rw-r--r--  1 silence  wheel  74  7  6 18:00 command5.job
    silence-pro:command silence$ cat command1.job
    type=command
    command=echo "create directory before get data"
    silence-pro:command silence$ cat command2.job
    type=command
    command=echo "get data from logserver"
    dependencies=command1
    silence-pro:command silence$ cat command3.job
    type=command
    command=echo "get data from ftp"
    dependencies=command1
    silence-pro:command silence$ cat command4.job
    type=command
    command=echo "insert into hive"
    dependencies=command5
    silence-pro:command silence$ cat command5.job
    type=command
    command=echo "upload to hdfs"
    dependencies=command3,command2
    silence-pro:command silence$
    
  2. 将文件夹压缩为zip包
    silence-pro:usercase silence$ ll
    total 8
    drwxr-xr-x  7 silence  wheel   224  7  6 18:00 command
    drwxr-xr-x  4 silence  wheel   128  7  6 17:48 java
    -rw-r--r--  1 silence  wheel  3021  7  6 17:48 java.zip
    silence-pro:usercase silence$ zip -r command.zip command/
      adding: command/ (stored 0%)
      adding: command/command2.job (deflated 14%)
      adding: command/command3.job (deflated 15%)
      adding: command/command1.job (deflated 7%)
      adding: command/command4.job (deflated 15%)
      adding: command/command5.job (deflated 20%)
    silence-pro:usercase silence$ ll
    total 16
    drwxr-xr-x  7 silence  wheel   224  7  6 18:00 command
    -rw-r--r--  1 silence  wheel  1301  7  6 18:04 command.zip
    drwxr-xr-x  4 silence  wheel   128  7  6 17:48 java
    -rw-r--r--  1 silence  wheel  3021  7  6 17:48 java.zip
    silence-pro:usercase silence$
    
  3. 在Azkaban系统创建项目并上传压缩包 Alt text Alt text Alt text Alt text

    command: 需要执行的命令 dependencies: job的依赖关系

Java任务

  1. 创建java.job
    silence-pro:java silence$ cat java.job
    type=javaprocess
    #指定类的全路径
    java.class=com.coocaa.ad.azkaban.demo.DemoAzkaban
    #指定执行jar包的路径
    classpath=lib/*
    silence-pro:java silence$
    
  2. 编写java文件
    package com.coocaa.ad.azkaban.demo;
    /**
     * <br>
     * <b>Function:</b><br>
     * <b>Author:</b>@author Silence<br>
     * <b>Date:</b>2018-07-04 16:54<br>
     * <b>Desc:</b>无<br>
     */
    public class DemoAzkaban {
    
     public static void main(String[] args) {
         System.out.println("hello world...");
     }
    }
    
  3. 讲上述文件打包成jar放到java.job 同级的lib文件夹下
    silence-pro:java silence$ ll
    total 8
    -rw-r--r--  1 silence  wheel  135  7  6 17:48 java.job
    drwxr-xr-x  3 silence  wheel   96  7  6 17:46 lib
    silence-pro:java silence$ cd lib/
    silence-pro:lib silence$ ll
    total 8
    -rw-r--r--  1 silence  wheel  3634  7  6 17:46 adx-azkaban-0.0.1-SNAPSHOT.jar
    silence-pro:lib silence$
    
  4. 创建Java-project项目并压缩成zip包并上传系统 Alt text
  5. 执行任务 Alt text

    type=javaprocess Java任务类型 java-type java.class=com.coocaa.ad.azkaban.demo.DemoAzkaban 配置类的全路径 classpath=lib/* 存放依赖包