stepfunctions-tasks: emr-create-cluster - add support for AutoTerminationPolicy.IdleTimeout #29926
Labels
@aws-cdk/aws-emr
Related to Amazon EMR
@aws-cdk/aws-stepfunctions-tasks
effort/medium
Medium work item – several days of effort
feature-request
A feature should be added or improved.
p2
Describe the feature
This feature currently exists in the SDK RunJobFlow EMR API for creating EMR clusters.
"AutoTerminationPolicy: { "IdleTimeout": number }
This feature will terminate the cluster if it has been idle for a specified number of seconds.
Right now the CDK emrCreateCluster step function task is missing this option, but I believe it is possible to do if the entire step function is written in ASL JSON. There is a similar missing parameter issue for the ManagedScalingPolicy parameter as well.
Use Case
This feature is very useful for terminating clusters when no job has run in a while. Many times, I'll forget to terminate a cluster and this will cause a large bill. The auto termination policy helps prevent that. It is also useful for keeping a cluster available for a while if another job might be submitted later, instead of terminating it immediately and then having to create a new one. It also helps simplify my state machines by not having to add a terminateCluster step.
Proposed Solution
Add to the emrCreateCluster cdk construct. One workaround is by using ASL JSON directly with
DefinitionBody.fromString
.Other Information
Related Issue (Missing ManagedScalingPolicy)
Acknowledgements
CDK version used
2.138.0
Environment details (OS name and version, etc.)
MacOS
The text was updated successfully, but these errors were encountered: