Fault bucket 736400234, type 5
Event Name: VMM20
Response: None
Cab Id: 0

Problem signature:
P1: vmmservice
P2: 2.0.4271.0
P3: Engine.TaskRepository
P4: 2.0.4271.0
P5: M.V.D.T.Task.GetAllTasks
P6: S.C.Generic.KeyNotFoundException
P7: cdb2
P8:
P9:
P10: 

Attached files:
C:\Windows\Temp\SCVMM.f5c52a6e-8f74-450c-bc57-d0ede6234a3b\report.txt

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportArchive\Report069118df

The logs were empty, and a quick search based on this error didn’t yield any useful results. Eventually, I tried running a DBCC CHECKDB in SQL Server and found some corrupted page links like this:

Table error: Object ID x index ID x, partition ID x alloc unit ID type In-row data). Page x is missing a reference from previous page Possible chain linkage problem.

No wonder I got no hits by searching, it’s quite unlikely that somebody else’s SQL Server suffered the same corruption. I had to repair with data loss, but was able to recover practically everything (my automated backup hadn’t been running for some time).

ALTER DATABASE VirtualManagerDB SET SINGLE_USER WITH NO_WAIT
DBCC CHECKDB (VirtualManagerDB, REPAIR_ALLOW_DATA_LOSS)
ALTER DATABASE VirtualManagerDB SET MULTI_USER WITH NO_WAIT

Good old DBCC CHECKDB did the job.

Repair: The Nonclustered index successfully rebuilt for the object "dbo.tbl_TR_SubtaskTrail, PK_tbl_TR_SubtaskTrail" in database "VirtualManagerDB".
Msg 8945, Level 16, State 1, Line 3
Table error: Object ID 2137058649, index ID 1 will be rebuilt.
        The error has been repaired.
Msg 8978, Level 16, State 1, Line 3
Table error: Object ID 2137058649, index ID 1, partition ID 72057594039173120, alloc unit ID 72057594040418304 (type In-row data). Page (1:12338) is missing a reference from previous page (1:50295). Possible chain linkage problem.
        The error has been repaired.
Msg 8981, Level 16, State 1, Line 3
Table error: Object ID 2137058649, index ID 1, partition ID 72057594039173120, alloc unit ID 72057594040418304 (type In-row data). The next pointer of (1:50295) refers to page (1:34253). Neither (1:34253) nor its parent were encountered. Possible bad chain linkage.
        The error has been repaired.
Msg 8945, Level 16, State 1, Line 3
Table error: Object ID 2137058649, index ID 2 will be rebuilt.
        The error has been repaired.

The moral of this tale: make sure your automated backups are working properly.

So, I thought, clearly what I need is another application server and client. I cloned the VMs I had, then turned them into templates. This is where things went a bit wrong. When I p2v’d the originals, SCVMM cloned the MAC addresses of the physical machines. If I were replacing physical machines in an enterprise, this is probably what I would have wanted. With just one of each machine (and each MAC), the network works fine.

However, when you create a new machine from a template taken from a p2v’d VM, the template also inherits the physical machines MAC, as does the instance of the template. At no point in any of the template/new machine wizards was there a mention of a MAC address. I had previously read an article about how MACs are assigned dynamically, so I assumed this was happening seamlessly. It wasn’t.

In the beginning, I only used one of the two clients at a time, and didn’t notice the intermittent network problems. Then suddenly it seemed like only one of the clients could connect to the network at a time. I could only see file shares from one of them on other machines: I’d connect to one client, and the share on the other would close. Prior to this, I had intermittent problems accessing the app server.

After a considerable amount of time spent measuring the network speed at each client (which worked fine, since I only accessed one of them at a time), I went back to basics with my old friend ipconfig. I’m still quite surprised that I noticed that the MAC addresses on the clients were the same, it’s not something I would normally look at.

The moral of this story: be careful when assigning MAC addresses.