任务调度Azkaban(一)搭建

Azkaband 搭建

获取压缩包tar.gz

  1. 拉取相应版本代码 https://github.com/azkaban/azkaban/releases git clone https://github.com/azkaban/azkaban.git
  2. 在项目根目录下一次执行
    • 清理一下./gradlew clean
    • 压缩成jar包./gradlew distTar
  3. 生成azkaban-web-server,azkaban-exec-server,azkaban-db等tar.gz包 ~/zakaban/azkaban-web-server/build/distributions/azkaban-web-server-3.49.0-4-g7efe67b.tar.gz ~/azkaban/azkaban-exec-server/build/distributions/azkaban-exec-server-3.49.0-4-g7efe67b.tar.gz ~/azkaban/azkaban-db/build/distributions/azkaban-db-3.49.0-4-g7efe67b.tar.gz
  4. 复制上面三个tar.gz到/opt/azkaban/目录,并解压重命名

初始化azkaban数据库

  1. 创建数据库azkaban create database azkaban;
  2. 创建用户和密码 CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban123456';
  3. 给用户赋予数据库权限 GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;
  4. 执行sql脚本初始化数据库 source /opt/azkaban/azkaban-db-3.49.0-4/create-all-sql-3.49.0-4-g7efe67b.sql
  5. 配置jetty ssl 密码:123456 Alt text

配置azkaban-web-server

  1. 在/opt/azkaban/azkaban-web-server-3.49.0-4目录下创建conf目录(如果目录下有conf目录则忽略,没有则从azkaban-solo-server目录下面拷贝过来,并拷贝log4j.properties 放到conf目录下)
  2. 编辑azkaban.properties文件,其中mysql数据库,用户密码根据上一步配置的进行调整,此处没有配置jetty ssl,jetty.use.ssl=false

silence-pro:conf silence$ cat azkaban.properties
# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai

# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml

# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
database.type=mysql
#h2.path=./h2
#h2.create.tables=true
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban123456
mysql.numconnections=100

# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081
jetty.ssl.port=8443
jetty.keystore=keystore
jetty.password=123456
jetty.keypassword=123456
jetty.truststore=keystore
jetty.trustpassword=123456
jetty.excludeCipherSuites=SSL_RSA_WITH_DES_CBC_SHA,SSL_DHE_RSA_WITH_DES_CBC_SHA,SSL_DHE_DSS_WITH_DES_CBC_SHA,SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA,SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA

# Azkaban Executor settings
executor.port=12321
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
#azkaban.jobtype.plugin.dir=plugins/jobtypes
  1. 配置azkaban-users.xml文件,测试使用默认设置
silence-pro:conf silence$ cat azkaban-users.xml
<azkaban-users>
  <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
  <user password="metrics" roles="metrics" username="metrics"/>

  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>
  1. 根据需要配置log4j.properties文件,测试使用默认
silence-pro:conf silence$ cat log4j.properties
log4j.rootLogger=INFO, Console
log4j.logger.azkaban=INFO, server
log4j.appender.server=org.apache.log4j.RollingFileAppender
log4j.appender.server.layout=org.apache.log4j.PatternLayout
log4j.appender.server.File=local/azkaban-webserver.log
log4j.appender.server.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n
log4j.appender.server.MaxFileSize=102400MB
log4j.appender.server.MaxBackupIndex=2
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
log4j.appender.Console.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n
silence-pro:conf silence$
  1. 启动azkaban-web-server

./bin/start-web.sh,关闭使用./bin/shutdown-web.sh

  1. 查看日志文件

tail -f ./local/azkaban-webserver.log

  1. 访问http://localhost:8081/index 初始密码为azkaban-user.xml里面配置的账号密码,默认为azkaban,azkaban

配置azkaban-exec-server

  1. 复制azkaban-web-server下的conf目录到azkaban-exec-server的conf目录下
mkdir conf
cp ../azkaban-web-server-3.49.0-4/conf/* ./conf/
  1. 编辑azkaban.properties
silence-pro:conf silence$ cat azkaban.properties
# Azkaban Personalization Settings
default.timezone.id=Asia/Shanghai
# Azkaban JobTypes Plugins
azkaban.jobtype.plugin.dir=plugins/jobtypes
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban123456
mysql.numconnections=100

# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
  1. 编辑log4j.propertis文件,根据需要调整,这里更换了一个日志文件名称azkaban-execserver.log
    silence-pro:conf silence$ cat log4j.properties
    log4j.rootLogger=INFO, Console
    log4j.logger.azkaban=INFO, server
    log4j.appender.server=org.apache.log4j.RollingFileAppender
    log4j.appender.server.layout=org.apache.log4j.PatternLayout
    log4j.appender.server.File=local/azkaban-execserver.log
    log4j.appender.server.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n
    log4j.appender.server.MaxFileSize=102400MB
    log4j.appender.server.MaxBackupIndex=2
    log4j.appender.Console=org.apache.log4j.ConsoleAppender
    log4j.appender.Console.layout=org.apache.log4j.PatternLayout
    log4j.appender.Console.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n
    
  2. 启动azkaban-exec-server ` ./bin/start-exec.sh,关闭使用./bin/shutdown-exec.sh`
  3. 查看日志 tail -f azkaban-execserver.log

测试任务执行

  1. 创建简单任务脚本
    silence-pro:usercase silence$ cat command.job
    #command.job
    type=command
    command=echo 'hello azkaban'
    silence-pro:usercase silence$
    
  2. 压缩成zip包
    silence-pro:usercase silence$ zip command.job.zip command.job
      adding: command.job (deflated 16%)
    silence-pro:usercase silence$ ll
    total 16
    -rw-r--r--  1 silence  wheel   55  6 28 17:25 command.job
    -rw-r--r--  1 silence  wheel  218  6 30 16:43 command.job.zip
    silence-pro:usercase silence$
    
  3. 在Azkaban系统创建测试project,并上传zip包进行测试 Alt text Alt text Alt text

搭建过程遇到的坑

  1. Missing required property ‘azkaban.native.lib’ 解决方案:设置execute.as.user=false
silence-pro:jobtypes silence$ pwd
/opt/azkaban/azkaban-exec-server-3.49.0-4/plugins/jobtypes
silence-pro:jobtypes silence$ cat commonprivate.properties
# set execute-as-user
execute.as.user=false
#execute.as.user=true
#azkaban.native.lib=/opt/azkaban/azkaban-web-server-3.49.0-4/bin
silence-pro:jobtypes silence$

安装gradle

https://docs.gradle.org/current/userguide/installation.html 配置环境变量

vim ~/.bash_profile
export Gradle_HOME=/Users/silence/Work/gradle/gradle-4.8.1
export PATH=$PATH:$Gradle_HOME/bin
gradle -v