Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(stepfunctions-tasks): add AutoTerminationPolicy.IdleTimeout to EmrCreateCluster #29954

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 15 additions & 0 deletions packages/aws-cdk-lib/aws-stepfunctions-tasks/README.md
Expand Up @@ -730,6 +730,21 @@ new tasks.EmrCreateCluster(this, 'Create Cluster', {
});
```

If you want to use an [auto-termination policy](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-auto-termination-policy.html),
you can specify the `autoTerminationPolicy` property. Set the `idleTimeout` as a `Duration` between 60 seconds and 7 days.
`autoTerminationPolicy` requires the EMR release label to be 5.30.0 or above.

```ts
new tasks.EmrCreateCluster(this, 'Create Cluster', {
instances: {},
name: 'ClusterName',
autoTerminationPolicy: {
idleTimeout: Duration.seconds(120),
},
});
```


### Termination Protection

Locks a cluster (job flow) so the EC2 instances in the cluster cannot be
Expand Down
@@ -1,6 +1,7 @@
import { Construct } from 'constructs';
import {
ApplicationConfigPropertyToJson,
AutoTerminationPolicyPropertyToJson,
BootstrapActionConfigToJson,
ConfigurationPropertyToJson,
InstancesConfigPropertyToJson,
Expand Down Expand Up @@ -67,6 +68,13 @@ export interface EmrCreateClusterProps extends sfn.TaskStateBaseProps {
* @default - A role will be created.
*/
readonly autoScalingRole?: iam.IRole;

/**
* An auto-termination policy for an Amazon EMR cluster.
*
* @default - None
*/
readonly autoTerminationPolicy?: EmrCreateCluster.AutoTerminationPolicyProperty;

/**
* A list of bootstrap actions to run before Hadoop starts on the cluster nodes.
Expand Down Expand Up @@ -227,6 +235,12 @@ export class EmrCreateCluster extends sfn.TaskStateBase {
}
}
}

if (this.props.autoTerminationPolicy !== undefined && !cdk.Token.isUnresolved(this.props.autoTerminationPolicy.idleTimeout)) {
if (this.props.autoTerminationPolicy.idleTimeout.toSeconds() < 60 || this.props.autoTerminationPolicy.idleTimeout.toSeconds() > 604800) {
throw new Error(`Idle Timeout must be between 60 and 604800 seconds but got ${this.props.autoTerminationPolicy.idleTimeout.toSeconds()}.`)
}
}
}

/**
Expand Down Expand Up @@ -279,6 +293,7 @@ export class EmrCreateCluster extends sfn.TaskStateBase {
AdditionalInfo: cdk.stringToCloudFormation(this.props.additionalInfo),
Applications: cdk.listMapper(ApplicationConfigPropertyToJson)(this.props.applications),
AutoScalingRole: cdk.stringToCloudFormation(this._autoScalingRole?.roleName),
AutoTerminationPolicy: this.props.autoTerminationPolicy ? AutoTerminationPolicyPropertyToJson(this.props.autoTerminationPolicy) : undefined,
BootstrapActions: cdk.listMapper(BootstrapActionConfigToJson)(this.props.bootstrapActions),
Configurations: cdk.listMapper(ConfigurationPropertyToJson)(this.props.configurations),
CustomAmiId: cdk.stringToCloudFormation(this.props.customAmiId),
Expand Down Expand Up @@ -1654,4 +1669,21 @@ export namespace EmrCreateCluster {
*/
readonly realm: string;
}

/**
* An auto-termination policy for an Amazon EMR cluster. An auto-termination policy defines the amount of idle time in seconds
* after which a cluster automatically terminates.
*
* @see hhttps://docs.aws.amazon.com/emr/latest/APIReference/API_AutoTerminationPolicy.html
*
*/
export interface AutoTerminationPolicyProperty {
/**
* Specifies the amount of idle time in seconds after which the cluster automatically terminates. You can specify a minimum
* of 60 seconds and a maximum of 604800 seconds (seven days).
*
* @default None
*/
readonly idleTimeout: cdk.Duration;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't this property needs to be optional?

}
}
Expand Up @@ -58,6 +58,17 @@ export function ApplicationConfigPropertyToJson(property: EmrCreateCluster.Appli
};
}

/**
* Render the AutoTerminationPolicyProperty as JSON
*
* @param property
*/
export function AutoTerminationPolicyPropertyToJson(property: EmrCreateCluster.AutoTerminationPolicyProperty) {
return {
IdleTimeout: cdk.numberToCloudFormation(property.idleTimeout.toSeconds()),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IdleTimeout is an optional property in cloudformation. In that case shouldn't we handle it as optional?

};
}

/**
* Render the ConfigurationProperty as JSON
*
Expand Down
Expand Up @@ -1733,3 +1733,55 @@ test('Task throws if WAIT_FOR_TASK_TOKEN is supplied as service integration patt
});
}).toThrow(/Unsupported service integration pattern. Supported Patterns: REQUEST_RESPONSE,RUN_JOB. Received: WAIT_FOR_TASK_TOKEN/);
});

test('Create Cluster with AutoTerminationPolicy', () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also add a test to throw error when the IdleTimeout is given outside the accepted seconds?

// WHEN
const task = new EmrCreateCluster(stack, 'Task', {
instances: {},
clusterRole,
name: 'Cluster',
serviceRole,
autoScalingRole,
autoTerminationPolicy: {
idleTimeout: cdk.Duration.seconds(120),
},
integrationPattern: sfn.IntegrationPattern.REQUEST_RESPONSE,
});

// THEN
expect(stack.resolve(task.toStateJson())).toEqual({
Type: 'Task',
Resource: {
'Fn::Join': [
'',
[
'arn:',
{
Ref: 'AWS::Partition',
},
':states:::elasticmapreduce:createCluster',
],
],
},
End: true,
Parameters: {
Name: 'Cluster',
Instances: {
KeepJobFlowAliveWhenNoSteps: true,
},
VisibleToAllUsers: true,
JobFlowRole: {
Ref: 'ClusterRoleD9CA7471',
},
ServiceRole: {
Ref: 'ServiceRole4288B192',
},
AutoScalingRole: {
Ref: 'AutoScalingRole015ADA0A',
},
AutoTerminationPolicy: {
IdleTimeout: 120,
},
},
});
});