[Improvement-15701][flink_task] supported variables of flink tasks #15708
Conversation
1. Exclude guava information from guava-retrying dependencies,Unable to download 'com.google.guava:guava-parent:pom:31.1.0.redhat-00001 'during packaging, this version of this library does not exist at all 2. Delete TaskExecutionContext definedParams member variable in this class, which has not been set,and modify other calls with this reference to prepareParamsMap 3. Modify the `getProperties` in the `Abstract Yarn` Task subclass to `prepareParamsMap`
bajiaolong
requested review from
caishunfeng,
SbloodyS and
zhuangchong
as code owners
SbloodyS
added
first time contributor
| @@ -67,7 +68,7 @@ public void before() throws Exception { | |||
| // "\",\"type\":\"VARCHAR\",\"value\":\"127.0.0.1:8080\"}]"); | |||
| Map<String, String> gloabParams = | |||
| Collections.singletonMap(PigeonTask.KEY_POOL_VAR_PIGEON_HOST, "127.0.0.1:8080"); | |||
| Mockito.when(taskExecutionContext.getDefinedParams()).thenReturn(gloabParams); | |||
| Mockito.when(ParameterUtils.convert(taskExecutionContext.getPrepareParamsMap())).thenReturn(gloabParams); | |||
There was a problem hiding this comment.
please fix the CI error
|
@SbloodyS Can this milestone be put in 3.2.2? Although this is an optimization item, it is also a bug. I will deal with it quickly. |
It's more like a feature instead of bug. Minor version updates do not include features. |
received, I want to know when the next version will be released and what are the Law? |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #15708 +/- ##
============================================
- Coverage 39.11% 39.11% -0.01%
+ Complexity 4887 4886 -1
============================================
Files 1326 1326
Lines 45206 45209 +3
Branches 4818 4818
============================================
Hits 17683 17683
- Misses 25635 25637 +2
- Partials 1888 1889 +1 ☔ View full report in Codecov by Sentry. |
There is currently no release plan for version 3.3.0. For version rules, you can take a look at |
ths |
| * definedParams | ||
| * // todo: we need to rename definedParams, prepareParamsMap, paramsMap, this is confusing | ||
| */ | ||
| private Map<String, String> definedParams; |
There was a problem hiding this comment.
hi @bajiaolong definedParams Is this parameter only used for Flink type tasks?
There was a problem hiding this comment.
hi @bajiaolong
definedParamsIs this parameter only used for Flink type tasks?
In addition to flink, there are also MapReduce and Pigeon tasks in use. But this parameter seems to have no effect, because I didn't see the task place setting it. The Pigeon task has test cases, so I revised them together.
| <exclusions> | ||
| <exclusion> | ||
| <groupId>com.google.guava</groupId> | ||
| <artifactId>guava</artifactId> | ||
| </exclusion> | ||
| </exclusions> |
There was a problem hiding this comment.
I'm not quite sure about the scope of this impact. cc @ruanwenjun
|
close #15701
Exclude guava information from guava-retrying dependencies,Unable to download 'com.google.guava:guava-parent:pom:31.1.0.redhat-00001 'during packaging, this version of this library does not exist at all
Delete TaskExecutionContext definedParams member variable in this class, which has not been set,and modify other calls with this reference to prepareParamsMap
Modify the
getPropertiesin theAbstract YarnTask subclass toprepareParamsMapPurpose of the pull request
does not take effect when flink uses parameter variables
Brief change log
1.Exclude guava information from guava-retrying dependencies,Unable to download 'com.google.guava:guava-parent:pom:31.1.0.redhat-00001 'during packaging, this version of this library does not exist at all
getPropertiesin theAbstract YarnTask subclass toprepareParamsMapVerify this pull request
This pull request is code cleanup without any test coverage.