Releases: azkaban/azkaban
Containerization for flow execution
This release has major changes related to containerization for the flow execution. It provides a way to isolate flow execution on containerized infrastructure like Kubernetes. For more detail, please refer to this design doc: https://azkaban.readthedocs.io/en/latest/containerization-design.html
9018fa2 (HEAD -> master, tag: 4.0.0, origin/master, origin/HEAD) Flow deterministic ramp-ups for imagetype and containerization
17bb970 Added jobtype based filtering for containerization ramp-up
0ae2163 Adding dependency for javax.el as it needs to be upgraded to 3.0 for hibernateValidator dependency (#2792)
adce749 Allow multiple copies of the same dependency file in Thin Zips. (#2754)
57286fb Merge Flow and server logs for containerized execution. (#2790)
d1a6efe ApiGateway support for ramp-up mechanism from POLL to CONTANERIZED dispatch
b285244 Give job props precedence over cluster props (#2786)
2ab81c1 Added mechanism to rampup Containerized from POLL dispatch
566bb1d Resolve symbolic links for jar files added to HadoopSecurityManagerClassloader (#2782)
7750f80 Marking flow as PREPARING from DISPATCHING as POD creation request is submitted (#2784)
ab50ddc Adding an option to provide flow container resource requests from flow parameter (#2780)
270defa Changed start status to READY instead of PREPARING for poll based dispatch
8024fad Add probes from the podTemplate to the podSpec
e10a022 Merge cf-engine-k8s to master (#2772)
f49d865 create-all-sql-*.sql is missing in some distributions
40625b1 Fix hyperlink to Containerization Design doc. Missed one leading "`" (#2767)
09e7292 Add Image Management APIs to index of documentation for easier access. (#2764)
de5aa52 Updated image management API documentation. (#2766)
4a12dde Delete unused "schedule options" files (#2755)
c2b1634 Marking flow as PREPARING from DISPATCHING as POD creation request is submitted. (#2759)
a9a8b8f Fix handling of null executionId in ExecutorApiGateway (#2752)
e6febcc Fetching all the hadoop tokens from single method and moving call to specific method as responsibility of each implementation (#2746)
8d219ee Fixed image link for flowchart in imageMgmtAPI.rst (#2745)
902c6bd Containerization Design Doc (#2727)
16a6fd5 Bring back "Flow override properties to also override job properties" (#2632)
f667821 Add pipeline Level 3: block all jobs until the previous flow has completed (#2515)
7d770b2 Flow container cherry-pick from cf-engine-k8s (#2742)
87b1d56 Build Azkaban against Hadoop 2.10. (#2729)
fb74282 Add executor api calls for containerized flows (#2715)
b3c5bb6 Rename polling DB lock to include schema name (#2720)
ce35c92 Add secret mounting to kubernetes dispatch (#2711)
e3d8e5b Prevent kill action from leaving executions stuck in KILLING status. (#2691)
635a3fc Restructuring code for HdfsAuth so that there can be multiple implementations of HdfsAuth and we can have an option to choose any implementation. (#2710)
670be39 Add ability to mount secrets in the PodSpec builder (#2702)
cc46cd3 Updated AzKubernetesV1SpecBuilder.java to add hostPath volumes (#2689)
8c7926c Add a date-time utility method and do a few refactors. (#2688)
e8b4b6a Moving common HadoopSecurityManager code to AbstractHadoopSecurityManager so that all the versions of HadoopSecurityManager can reuse this code. (#2685)
51fe9b2 Solidified FlowContainer logic with integration testing. (#2687)
7a18f7b Add TLS support to ExecutorApiGateway (#2679)
438930b Expose Azkaban Executor Server configuration to the cluster router (#2675)
4c9993f Fixing the name of Property class for Containerized Dispatch. Earlier it was ContainerizedExecutionManagerProperties. (#2680)
1d68caf Added version_set table DDL and JdbcVersionSetLoaderImpl class for interaction
0915422 Add periodic cleanup of containers (#2671)
5ca7d39 It contains integration with service spec builder and service creation. (#2674)
a10ff5d Added class for creation of kubernetes service object using template (#2667)
1c0642c It contains integration with pod spec builder and pod creation. It also contains delete methods for pod and service. (#2669)
6941dad Use CredentialProvider instead of CredentialProviderWithKeyStore unless needed. (#2664)
259b251 Remove duplication in embedded flow name. (#2661)
f7036b6 Job specific cluster components dependencies should take precedence over those of jobtype plugin. (#2660)
e33546d BDP-3878: Magnet on flow ramping regardless failure (#2652)
65b25bf Introducing version set and added skeleton for version set loader
d41c988 add a timer metric for flow startup delay (#2650)
7f86e68 Added builder class for creation of Kubernetes pod-spec
1121c83 Introducing FlowContainer. (#2647)
543797f Merge robin branch into master (#2644)
1e8c84c Adding dispatch logic for containerized environment (#2635)
1522278 Expand the length of some fields related to Ramp Feature (#2630)
b3c26fb Add a new CredentialProviderWithKeyStore interface which extends CredentialProvider interface with KeyStore specific APIs. (#2629)
8fb2667 Updaing dependency jopt-simple to 5.0.3
d0b519c Add get/set KeyStore methods in CredentialProvider interface. (#2625)
aee22b0 Removing duplicate code in ExecutorManager and ExecutionController (#2623)
8b65539 Fixed the PDT-->PST time change tests. They should pass until atleast 2029 (#2626)
5375629 Make MetricsManager more robust to changes in metrics reporter Ctor
43177fa Fix issue 917 and other startup issues
94f9ac8 Revert Flow Override feature #2241 (#2620)
b60eb05 Patch6 (#2616)
e9f1789 Add CSRF token validation for modifying project permissions
f5e294d Fix startup errors in some environments (#2614)
b2173ea Adding utility method to check PUSH and PUSH_CONTAINERIZED dispatch m… (#2607)
c05f233 Add flow execution source to flow properties (#2591)
3d07ead Add metric to measure the executor polling frequency. (#2599)
4386e77 Fixes failing WebUtils unit test (#2612)
fae5b51 Update Travis CI configs. (#2609)
55b8eb9 Fix grammatical errors (#2605)
603afd4 Fix grammatical errors (#2606)
b0aad21 Enable security logging via AzkabanEventReporter (#2608)
98eb6ca Restore flow lock message after project upload. (#2600)
9a50b73 Fix flaky queue duration condition in JobRunnerTest (#2604)
734121f Fix grammatical errors (#2601)
f8a72d0 Fix grammatical errors (#2602)
783f88d Adding enum for Dispatch Method. It can be PUSH, POLL or CONTAINERIZED (#2603)
0fc132f Formatting only: apply Save Actions on some files (#2596)
ad72f4a Flow override properties to also override job properties (#2241)
5b8eb08 Revert "Bring back Multiple Hadoop cluster (Robin) changes and fix LocalFileSystem leak. (#2585)" (#2592)
76537a3 Add job props to uniquely identify a job in mapreduce tags (#2593)
457957c [UI] Improve readability of long job property values (#2580)
953b872 Logging improvements for job callbacks (#2583)
efef003 Fix typos (quiting -> quitting) (#2581)
497d4cd Add metrics to API endpoint requests. (#2590)
2217f85 fix bug in calculating flow kill and pause durations,and job kill and preparation durations (#2578)
c79c8ce Remove null from startup-dependencies.json while writing and reading … (#2588)
1a78e1c fix typo for couple documents (#2584)
dd07532 Bring back Multiple Hadoop cluster (Robin) changes and fix LocalFileSystem leak. (#2585)
99daa7b UI and API changes to prepare Azkaban for OAuth SSO. (#2587)
991238b Revert "Add metrics to API endpoint requests" (#2582)
a041bcd (origin/limit_flow_retry_count) Revert "Pass cluster object around instead of cluster id during job parameter construction in JobTypeManager." (#2579)
ece3f08 Support sla settings in execute & schedule API actions (#2523)
d18bdc8 Formatting only: apply Save Actions on some files (#2562)
f2e3b68 Refactoring only: extract SLA parsing to new SlaRequestUtils (#2522)
31bd742 Add metric NumAgedQueuedFlows: Count flows that are in queue for a long time (#2549)
2343081 Fixed the thread-pool usage when downloading dependencies
31fa0b7 Pass cluster object around instead of cluster id during job parameter construction in JobTypeManager.
604b379 Add metrics to API endpoint requests
51eac65 Fix handling of Api failures during executor healthcheck
278a313 Closed the IO streams in the thinzip code path.
1eec4e4 Expose cluster routing exceptions to job logger for easy debugging.
7e3899e Migrated to PDL
ec24817 Fix jobrunner fireEvent timing.
5402abf Support routing jobs to a single Hadoop Cluster in Azkaban Exec Server.
c4a16c7 Introduce a per-job classloader during job execution.
6e3635c Revert "Introduce a per-job classloader during job execution."
5ca1100 Modify fields for Azkaban Event Reporter.
9dc4486 Add Azkaban project event data.
ee385ed Make HadoopSecurityManager_H_2_0 non-singleton.
519395c Introduce a per-job classloader during job execution.
48bc24f Revert "Introduce a per-job classloader during job execution."
3bd23cf Introduce a per-job classloader during job execution.
687bb99 Remove Teradata jobtype in Reportal, and add idea plugin to build.gradle
5669e77 Changed assertEquals orders and remove a unit test assertion to prevent build error.
6253a37 added attributes for job and flow life-cycle events
4c296d1 Allow jobtype plugins to process job properties before jobs start to run.
2466fbb [UI] Correctly indicate when a schedule has SLA rules defined in the Schedules page.
634e202 Extract API endpoint names and some other (web|exec) server configurations into constants.
84dd461 Introduce a concept of cluster component, through which jobs and jobtypes can declare their dependencies.
5a0f102 Change to Builder Pattern
1a98b4e Project Cache Abstraction
9255ef2 Handle exception during executor shutdown due to flow ramp disabled on clusters
dc12eba Remove direct instantiation and cast to Hdfs Distribute...
3.81.0: Add support for thin archives on web and executor server (#2363)
* Add thin archives feature to web and executor server. * Move from webserver persisting deps to CachedHttpFileSystem * Change CachedHttpFileSystem scheme to chttp:// * Allow caching in CachedHttpFileSystem to be disabled through config.
3.73.1
3.73.0
8ab89b9 (HEAD -> master, tag: 3.73.0, origin/master, origin/HEAD) Fix XmlUserManager bug: add back the mandatory constructor (#2258)
f2daeeb Fix flaky FileWatcherTest (#2259)
505d284 Auto expand RUNNING jobs (like FAILED and KILLED) in Flow Execution page, Job List tab. (#2257)
34b259e Fix HDFS viewer authentication issue. (#2255)
506a5fd FileWatcherTest fix (#2256)
05b71e4 Add logs to ProjectManager initialization at web server startup. (#2251)
320f3c0 XmlUserManagerTest to use mocks & avoid sleeping (#2244)
d69a0d1 Job callback to support multiple occurrences of the same token (#2237)
44d8153 Fix NPE in dynamically generating job link url. (#2250)
f0d6e0d Set maxParallelForks = 1 again in build.gradle (#2242)
054a54f Delete schedule of renamed/deleted flows on project upload. (#2226)
50f93f7 Add logging for the case when project cache still has enough buffer (#2239)
70ed1ea Add links to job logs in job context menu in execution graph (#2231)
11a1a56 Rename job logs link: Details -> Log (#2232)
2db6a8b Update CONTRIBUTING.md (#2214)
3.70.2
3.72.1
1c61d83 [BUGFIX] Backward Imcompatible to some JobTypes (#2228)
2e37113 Implement ability to check executor's capacity before polling for new Dispatching Logic (#2195)
137ba43 revert flowid capitalization in ProjectManagerServlet.java (#2229)
38a6fca Enable dynamic job link for ExecutionController. (#2227)
0455418 WatchService : Make sure to wait for 1 second before acting on event to avoid multiple reloads (#2218)
3.72.0
d258ea7 Add APIs for locking a flow (#2210)
3308072 make logging explicit when fetching tokens from other namenodes (#2217)
18ec507 use fully qualified job name for azkaban.link.attempt.url and azkaban.link.jobexec.url (#2219)
4f30ad3 [DEDUPE] Refactor and Deduplicate the common codes (remerged) (#2186)
80cdc1a Enable web metrics for new dispatching logic. (#2213)
2852010 Add directory name to exception message when failing to create temp prop file (#2208)
21c555b Add whitelist for specifying max number of concurrent runs for flows (#2201)
d56d7cd fixed SessionfindByIp TestCase (#2188)
d37cc8d Remove redundant UNIQUE INDEX declarations in CREATE TABLE statements (#2193)
3.71.1
3.71.0
b2212b9 AUto Reload Config : Fix race condition to make sure WatchService is started before its watcher thread starts. (#2184)
d9911be getProjectId api no longer returns IDs of deleted/inactive projects. (#2180)
ae5aff4 When flow trigger fails to be scheduled/unscheduled, throw exception to terminate the subsequent steps (#2174)
53f30e4 Auto reload azkaban-users.xml if there is any change in it. (#2172)
5e9e00e Reduce execution dir retention to 1 min (#2182)
ae546a7 overrideSuccessEmails and overrideFailureEmails have mixed up its values (#1734)
da7f2b0 Session monitoring enhancement (#2168)
9f99d73 Add web server host to event metadata (#2175)
e718b6e Refactoring for SlaOption and ExecutionOptions (#2121)
3309ba2 Change ExecutorManager to ExecutorManagerAdapter in Flow Trigger (#2171)
c2b00f5 Quartz scheduler refactor (#2161)
64cf66e Closes #1066. Fix HadoopSecurityManager_H_2_0 to use addDelegationTokens API instead of deprecated/private getDelegationToken (#2165)
4e212eb Refactor time related utility methods (#2164)