[Bug] ClosedDate is set to Null and will revert to the current date on save!

[MrHinsh]

          Any update on a fix?

In both v14.4.6 and v14.4.7 I receive the additional warning logs on these "ClosedDateIsMigrationDate" work items:
2024-02-26 15:32:18.078 -06:00 [WRN] The field Microsoft.VSTS.Common.ClosedDate is set to Null and will revert to the current date on save! 2024-02-26 15:32:18.078 -06:00 [WRN] Source Closed Date [#nnnnnn][Rev4]: "2022-04-20T14:04:02.1230000-05:00"

The History tab on the source work items has the Closed revision data showing: Closed by, Closed Date, Board Column (new and old), Reason (new and old), Resolved By, and State (new and old).

Originally posted by @t-anderson in #1747 (comment)

Somehow between these two breakpoints the value is lost!

https://github.com/nkdAgility/azure-devops-migration-tools/blob/855c55cefb23b27ae62aedca45127a6430c65103/src/VstsSyncMigrator.Core/Execution/MigrationContext/WorkItemMigrationContext.cs

• 801 - Mapps the existing Source fields to the Target fields
• 846 - System.ClosedDate comes up Null!

---

HELP NEEDED - We are unable to replicate this! But it does happen!

Can someone with this error please run a specific work item that they find has this issue through the tool and do one of the following:

Option 1: Generate and attach Log - if you are non-technical this is your option. It might provide us with enough info to diagnose, but may only result in more asks.

1. In the Config file change LogLevel to Debug.
2. Find a work item ID that has this issue, and craft a query that only selects this work item.
3. Delete any target copy of that work item.
4. Then run the migration
5. Comment here with the full Log attached

Option 2: Debug in Visual Studio - This option is best but requires some developer experience.

1. Fork the Repo and Clone it to your local computer
2. Open the Solution in Visual Studio (we use 2022) and do a Full Rebuild
3. Edit the Launch Profile to point to your config.
   
4. Craft the query to only load the work item with the issue
5. Add the breakpoints detailed above
6. "Play" the Launch Profile that you created and

Create a comment below with the results of your investigation!

[TomBolen]

Hello, i can reproduced the issue.

The Problem is that the field Microsoft.VSTS.Common.ClosedDate is readonly at the target.

The check of the State in this Line (Current Master Branch Line No. 474 ):

if (!_ignore.Contains(f.ReferenceName) &&
(!newWorkItem.Fields[f.ReferenceName].IsChangedInRevision || newWorkItem.Fields[f.ReferenceName].IsEditable)
&& oldWorkItem.Fields[f.ReferenceName].Value != newWorkItem.Fields[f.ReferenceName].Value)

returns false and the Field in the target can't be set.

State Values:
newWorkItem.Fields[f.ReferenceName].IsChangedInRevision = true
newWorkItem.Fields[f.ReferenceName].IsEditable = false
oldWorkItem.Fields[f.ReferenceName].Value = 04/08/2019 12:05:04
newWorkItem.Fields[f.ReferenceName].Value = null

Source: DevOps Server on premise
Target: Azure DevOps Server

[MrHinsh]

This is great work. If IsEditable is false, then there is no way we can update it.

What we need to figure out is what is causing the IsEditable to be false. What are the previous revisions and what is causing the work item to be in a state where its not editable. It might be something that we can check for...

[MrHinsh]

Could you be clearer as to the source and target version? Which versions are TFS (on-prem) and which are ADO (cloud)...

p.s. I just call everything on-prem TFS as it avoids confusion...

[TomBolen]

On-Prem it is the latest Version of DevOps 2020.1.2 with Patch 9.
Target ist the current Version of Microsoft Azure DevOps environment. (dev.azure.com)
Both are running the default Agile Template of Microsoft with limited changes.
The biggest different is the language. Why ever i did that, but the local installation is german...
For the migration i made this translation in the json file:
"FieldMaps": [
{
"$type": "FieldValueMapConfig",
"WorkItemTypeName": "",
"sourceField": "System.State",
"targetField": "System.State",
"defaultValue": "New",
"valueMapping": {
"Neu": "New",
"Aktiv": "Active",
"Gelöst": "Resolved",
"Geschlossen": "Closed",
"Entfernt": "Removed"
}
},
{
"$type": "FieldValueMapConfig",
"WorkItemTypeName": "",
"sourceField": "Microsoft.VSTS.Common.Risk",
"targetField": "Microsoft.VSTS.Common.Risk",
"defaultValue": "3 - Low",
"valueMapping": {
"1 - Hoch": "1 - High",
"2 - Mittel": "2 - Medium",
"3 - Niedrig": "3 - Low"
}
},
{
"$type": "FieldValueMapConfig",
"WorkItemTypeName": "",
"sourceField": "Microsoft.VSTS.Common.Severity",
"targetField": "Microsoft.VSTS.Common.Severity",
"defaultValue": "3 - Medium",
"valueMapping": {
"1 - Kritisch": "1 - Critical",
"2 - Hoch": "2 - High",
"3 - Mittel": "3 - Medium",
"4 - Niedrig": "4 - Low"
}
},
{
"$type": "FieldValueMapConfig",
"WorkItemTypeName": "",
"sourceField": "Microsoft.VSTS.Common.ValueArea",
"targetField": "Microsoft.VSTS.Common.ValueArea",
"defaultValue": "Business",
"valueMapping": {
"Geschäft": "Business",
"Architektonisch": "Architectural"
}
},
{
"$type": "FieldValueMapConfig",
"WorkItemTypeName": "",
"sourceField": "Microsoft.VSTS.Common.Activity",
"targetField": "Microsoft.VSTS.Common.Activity",
"defaultValue": "",
"valueMapping": {
"Bereitstellung": "Deployment",
"Entwurf": "Design",
"Entwicklung": "Development",
"Dokumentation": "Documentation",
"Anforderung": "Requirements",
"Test": "Testing"
}
},
{
"$type": "FieldValueMapConfig",
"WorkItemTypeName": "",
"sourceField": "Microsoft.VSTS.Common.ResolvedReason",
"targetField": "Microsoft.VSTS.Common.ResolvedReason",
"defaultValue": "",
"valueMapping": {
"Korrigiert": "Fixed"
}
}
],

[TomBolen]

And this is the test config of my WorkItemMigrationConfig Processor:
{
"$type": "WorkItemMigrationConfig",
"Enabled": true,
"UpdateCreatedDate": true,
"UpdateCreatedBy": true,
"WIQLQuery": "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = @teamproject AND [System.ID] >= 160 AND [System.ID] <= 165 AND [System.WorkItemType] NOT IN ('Codeüberprüfungsanforderung', 'Codeüberprüfungsantwort', 'Testsuite', 'Testfall', 'Testplan','Shared Steps','Shared Parameter','Feedback Request') ORDER BY [System.Id] asc",
"AttachmentMigration": true,
"AttachmentWorkingPath": "c:\devopsmigrate\WorkItemAttachmentWorkingFolder\",
"FixHtmlAttachmentLinks": false,
"SkipToFinalRevisedWorkItemType": false,
"WorkItemCreateRetryLimit": 5,
"FilterWorkItemsThatAlreadyExistInTarget": false,
"PauseAfterEachWorkItem": false,
"AttachmentMaxSize": 480000000,
"AttachRevisionHistory": false,
"LinkMigrationSaveEachAsAdded": false,
"GenerateMigrationComment": false,
"WorkItemIDs": null,
"MaxGracefulFailures": 0,
"SkipRevisionWithInvalidIterationPath": false,
"SkipRevisionWithInvalidAreaPath": false
},

[Slawisch]

Hello, I hope all is well! I wanted to check in and see if there's been any progress on the bug.
I also ran into it, couple more details from my attempts:
Not all items are affected, just some of them, and it doesn't seem to matter what type of work item it is – we can't find any pattern.
Also, when we skip migrating the work item's history, it always sets today's date, but that might just be how it's supposed to work.
I'm on version 15.0.0, by the way.

Any updates or tips would be super helpful and much appreciated.

[MrHinsh]

@Slawisch thanks for getting in touch. We have not been able to identify a pattern either... I've raised it with the product team, and added additional logging to see what the issue is.

It would require some debugging in Visual Studio to really find the issue. There are some instructions above if you can!

[Slawisch]

Hello again. I think I found why it happens.
Pattern - when change to 'Closed' state is in the last revision. When WI's state is 'Closed' before it, 'Closed Date' is set in the next revision.

Why it happens:
On the line 470: newWorkItem.State = oldWorkItem.State; it changes the state of target item. And if isEditable = true - it gets reset to false (I think that's some inner logic of WorkItem type).
So, if, for example, it is the last revision, we set State to 'Closed', isEditable becomes false and it's impossible to update Closed Date.
But, if it is the last revision, and we already have State set to Closed, then state isn't updated on that line and isEditable remains true. And date is changed successfully.

[MrHinsh]

Im not sure I 100% follow your workflow there... could you update it to make it clearer? It will help me understand if I can 1) make a code change to fix it, or 2) replicate it locally.

Can you specify exactly what the order of revisions is on the source?

if I can replicate it here, I can try some cod-itzu (code+jujitzu) to get this resolved.

[Slawisch]

_n - number of revisions, so that Rev.: n, for example, is the last one.

Scenario when bug happens:

Rev.: n-2
State: Active
Closed Date: not set

Rev.: n-1
State: Active
Closed Date: not set

Rev.: n
State: Closed
Closed Date: any date on source

Result: closed date not set on target

Scenario when bug does not happen:

Rev.: n-2
State: Active
Closed Date: not set

Rev.: n-1
State: Closed
Closed Date: any date on source

Rev.: n
State: Closed
Closed Date: any date on source

Result: closed date set on target only on last revision

[MrHinsh]

OK, so based on @Slawisch's awesome work here it looks like we could work around this by injecting a new revision in the Bug Senario to turn it into No-Bug Senario!

[MrHinsh]

For that we need to detect the scenario when building the revisions in the TfsRevisionManager and add the extra one.

[Slawisch]

I made some changes locally just to make it work already. Sure it could be done better, but I just needed it to be done somehow, so here what I did so far:

Created new function to copy last revision and push it with some minor changes:

public SortedDictionary<int, RevisionItem> PushLastRevisionCopy(SortedDictionary<int, RevisionItem> sourceRevisions)
        {
            if (sourceRevisions.Any())
            {
                var revisionToPushJson = JsonConvert.SerializeObject(sourceRevisions.Last().Value);
                var revisionToPush = JsonConvert.DeserializeObject<RevisionItem>(revisionToPushJson);

                revisionToPush.Number += 1;
                revisionToPush.Index += 1;
                revisionToPush.Fields["System.Rev"].Value = revisionToPush.Number;

                // Date updates
                var dateFieldsToUpdate = new List<string>
                {
                    "Microsoft.VSTS.Common.StateChangeDate",
                    "Microsoft.VSTS.Common.ClosedDate",
                    "System.ChangedDate"
                };

                foreach (var fieldName in dateFieldsToUpdate)
                {
                    if (revisionToPush.Fields.TryGetValue(fieldName, out var fieldValue) && fieldValue.Value is DateTime dateTime)
                    {
                        revisionToPush.Fields[fieldName].Value = dateTime.AddMinutes(1);
                    }
                }

                // Clear history field if it exists
                if (revisionToPush.Fields.ContainsKey("System.History"))
                {
                    revisionToPush.Fields["System.History"].Value = string.Empty;
                }

                sourceRevisions.Add(revisionToPush.Number, revisionToPush);
            }

            return sourceRevisions;
        }

And calling it in WorkItemMigrationContext.cs on line 219 just in the beginning of foreach

sourceWorkItemData.Revisions = _revisionManager.PushLastRevisionCopy(sourceWorkItemData.Revisions);

[Slawisch]

I also have some offtopic question, @MrHinsh, we may get The work item type cannot be found. It may have been renamed or destroyed. error in case some type on source has been destroyed but still remains in revisions. So, is there any way to successfully migrate WI with deleted WI Type on Source (and migrate WI's history as well along with WI)?

[MrHinsh]

The error is comming from ADO and not from our code. We can only migrate work items that we can open. 🤷‍♂️

[Slawisch]

Hi again, is there any news on a fix?

[MrHinsh]

There is no fix on my end for this. It's an Azure DevOps OM issue...

[Slawisch]

Sorry, I mean ClosedDate fix, with pushing one extra revision

[Slawisch]

Hey👋 Just wonder, if you meant ClosedDate bug or "work item cannot be found" error in your latest comment?