tree e0be0f4f6be27f73378b0e61f25edaf7fb0d05c9
parent 5d17f489b91e07c541dfe1cae2501a75b60b6cf3
author Barret Rhoden <brho@cs.berkeley.edu> 1484321317 -0500
committer Barret Rhoden <brho@cs.berkeley.edu> 1484751602 -0500

x86: vmm: Flush the VMCS when changing owning_proc

The GPCs unload when we finalize the context, which happens before the
process leaves the core at all.  The VMCS lingers, and my intent was to
remove it from the core when the process leaves the core.

However, abandon_core() was the wrong spot for that.  abandon_core() is
more of a "make sure we get out of whatever process context was running on
this core".  There's no guarantee that a process is even loaded at that
time.  What I really wanted was to drop the VMCS (if it was there at all)
when the core is no longer designated as belonging to the proc.

The distinction between these two is related to the difference between
cur_proc and owning_proc.  owning_proc is the process who should be running
on that core, specifically running owning_vcoreid with cur_ctx.  In
contrast, cur_proc (a.k.a. 'current') is the process whose address space is
loaded, for whatever reason.  If the process is running in userspace,
cur_proc == owning_proc.  However, kthreads that work on syscalls for a
process can run on cores that aren't owned by the proc.  Likewise, the
kernel can temporarily enter a process's address space, which usually
involves setting cur_proc.  e.g. send_event(), switch_to(), etc.

Given all this, here was the bug that caused this.  VMs would occasionally
die with an Invalid Opcode trap in the kernel.  I noticed this when they
were killed from ssh (ctrl-c), especially when the VM/VMM was busy.  I
could deterministically recreate it by having the guest spin and sending a
ctrl-c (kill from bash also worked, but kill -9 did not).

The invalid opcode happened during __invept().  From looking at the hexdump
of the arguments, the eptp was 0.  That only gets cleared in __proc_free(),
which made me suspect a refcnt problem.  The ref was indeed 0, so I thought
there was a problem with someone dropping a ref or not upping enough.

It turns out that all refs were accounted for, but the problem was
triggered by kthreads restarting and decreffing the proc before we called
abandon_core().  Specifically:
- kill sends a POSIX signal, the process calls sys_proc_destroy()
- proc_destroy() wakes the parent and sends itself __death, all via KMSGs
  to core_id().
- By the time we get to __death, we're down to about two references:
  owning_proc and cur_proc.  (I'm ignoring the FDs here - there were a
bunch of alarm FDs and syscalls that had refs).
- __death clears_owning_proc, dropping the owning_proc ref.
- Normally, you'd think we'd call abandon_core soon, but first we have the
  __launch_kthread() KMSG, which is restarting our parent's wait syscall
- When restarting that kthread, we switch cur_proc from the VMM (the one
  that is dying, and has only one ref) to the parent.  In doing so, we drop
the final ref for the VMM, which triggers __proc_free() and clears the
eptp.
- Once the parent's syscall is done, we prepare to idle and abandon_core().
  This abandon call isn't clearing the VMM's context, it's clearing the
parents.  abandon() doesn't really care what is running there.
- At this point, __abandon_core() tries to clear the VMCS.  It had never
  been cleared, but it's process (including the GPC!) had been freed.
Yikes!
- The reason the GPC->proc refcnt was zero wasn't because someone messed up
  the refcnts, it's because we didn't clear the GPC before dropping all the
refs.  That GPC->proc ref is internal (aka weak, uncounted).  The ref that
keeps the GPC alive is owning_proc (at least, now it is, after the fix).

Signed-off-by: Barret Rhoden <brho@cs.berkeley.edu>