Name Date Size #Lines LOC

..03-Dec-2018-

linux/H27-Sep-2017-

sys/H27-Sep-2017-

tools/H27-Sep-2017-

COPYING.linuxH A D27-Sep-201718.3 KiB358293

MakefileH A D12-Feb-20196.8 KiB342290

README.mdH A D27-Sep-201711.5 KiB289224

hyperv.hH A D27-Sep-20177.5 KiB20787

kvm.cH A D22-Aug-201863.1 KiB2,7911,902

kvm.confH A D27-Sep-201728 21

kvm.hH A D27-Sep-201713.7 KiB506350

kvm_apicdef.hH A D27-Sep-201711.7 KiB437340

kvm_bitops.hH A D27-Sep-20177.4 KiB297133

kvm_cache_regs.cH A D27-Sep-20172.4 KiB10062

kvm_cache_regs.hH A D27-Sep-20171.6 KiB4719

kvm_coalesced_mmio.cH A D27-Sep-20174.4 KiB197132

kvm_coalesced_mmio.hH A D27-Sep-20171.6 KiB5619

kvm_cpuid.hH A D27-Sep-20178.8 KiB161112

kvm_emulate.cH A D27-Sep-201778.2 KiB3,0602,564

kvm_emulate.hH A D27-Sep-20176.6 KiB20686

kvm_host.hH A D22-Aug-201816.6 KiB516412

kvm_i8254.cH A D27-Sep-201718.2 KiB800591

kvm_i8254.hH A D27-Sep-20172.3 KiB8252

kvm_i8259.cH A D27-Sep-201712.3 KiB607455

kvm_impl.hH A D27-Sep-20174.8 KiB12479

kvm_ioapic.cH A D27-Sep-201710.7 KiB468331

kvm_ioapic.hH A D27-Sep-20172.6 KiB8649

kvm_iodev.cH A D27-Sep-20171.4 KiB5427

kvm_iodev.hH A D27-Sep-20171.7 KiB5521

kvm_irq.cH A D27-Sep-20172.3 KiB11062

kvm_irq.hH A D27-Sep-20173 KiB9660

kvm_irq_comm.cH A D27-Sep-201711.6 KiB486373

kvm_lapic.cH A D27-Sep-201728.3 KiB1,3351,011

kvm_lapic.hH A D27-Sep-20173 KiB8648

kvm_link.cH A D27-Sep-20171.8 KiB6021

kvm_mdb.cH A D27-Sep-201710.6 KiB425313

kvm_mmu.cH A D18-Oct-201770.4 KiB3,0802,404

kvm_mmu.hH A D18-Oct-20172.6 KiB9154

kvm_msidef.hH A D27-Sep-20172.5 KiB8132

kvm_msr.hH A D27-Sep-20173.4 KiB12162

kvm_paging_tmpl.hH A D27-Sep-201715.8 KiB634458

kvm_para.hH A D27-Sep-20173.2 KiB11659

kvm_timer.hH A D27-Sep-20171.4 KiB5123

kvm_types.hH A D27-Sep-20171.8 KiB7535

kvm_vmx.cH A D21-Nov-2018111.4 KiB4,5493,427

kvm_vmx.hH A D27-Sep-201713.6 KiB409320

kvm_x86.cH A D28-Aug-2018123.1 KiB5,3264,090

kvm_x86.hH A D22-Aug-20187.1 KiB296218

kvm_x86host.hH A D24-Aug-201825.3 KiB893641

kvm_x86impl.hH A D27-Sep-20172.4 KiB7340

manifestH A D27-Sep-2017494 1514

msr-index.hH A D22-Aug-201814.7 KiB419316

processor-flags.hH A D27-Sep-20174.3 KiB12569

README.md

1illumos-kvm: KVM for illumos
2============================
3
4KVM is the kernel virtual machine, a framework for the in-kernel acceleration
5of QEMU.  illumos-kvm is a port of KVM to illumos, taking advantage of
6illumos-specific constructs like DTrace, cyclics, mdb, kstat, OS
7virtualization, network virtualization, ZFS, etc.  It is derived from the KVM
8source for Linux 2.6.34, the longterm source for which may be found here:
9
10    git://git.kernel.org/pub/scm/linux/kernel/git/longterm/linux-2.6.34.y.git
11
12To date, this implementation has been verified with a wide range of guest
13operating systems including illumos itself (both SmartOS and OpenIndiana
14distributions), FreeBSD, Plan 9, QNX, ChromeOS, HaikuOS, Microsoft Windows
15and Linux.
16
17The design center for this work is to use the virtualization features made
18available in the microprocessor -- and in particular, Intel's VMX.  As such,
19behavior on microprocessors that do not support VMX -- and more specifically,
20the extended page tables (EPT) found in second generation VMX support --
21should be graceful failure, not degraded operation.
22
23Divergences from KVM
24--------------------
25
26Divergences from KVM fall into several broad categories:  some functionality
27has been removed or not implemented because it is obviated by features of
28illumos (e.g., the custom tracing facility built into KVM); some functionality
29has been removed because it is only relevant to hardware that lacks
30virtualization support (e.g., older x86 hardware) or on hardware for which
31illumos lacks support (e.g., PPC, s390); and some functionality has been
32removed because the implementation complexity was simply too great relative
33to its value.
34
35Of this latter category, three areas of divergence merit special note.  First,
36there is no support for pageable guest memory (that is, guest memory is locked
37down).  While this is an opinionated decision at some level (in our
38experience, memory oversell leads to unacceptable pathologies in all but the
39idlest of workloads), we would welcome the work to integrate the KVM MMU
40notifier support into illumos-kvm.
41
42Second (and relatedly), illumos itself has no support for kernel same-page
43mapping (KSM) as found in Linux.  While illumos could in principle add such
44support, it is our experience that the memory that accrues from this is not
45sufficiently significant to pay for the increase in implementation and
46operator complexity.
47
48Finally, there is no support currently for AMD SVM.  This is not a value
49judgement of AMD's technology, but rather a reflection of limited engineering
50and testing resources.  (In the spirit of full disclosure, it should be said
51that the sponsor of illumos-kvm, Joyent, is an Intel-funded company -- but the
52lack of AMD support reflects only engineering prioritization and lack of
53testing infrastructure; AMD SVM support would be most welcome should someone
54in the community be so motivated as to port and test it.)
55
56Building illumos-kvm
57--------------------
58
59### Preparation
60
61Edit the Makefile and appropriately set the path for the `KERNEL_SOURCE`
62directory to point to the root of a checked out and built illumos directory.
63Building illumos KVM requires several recent additions to illumos,
64so be sure your illumos is up to date.
65
66Verify that you have gcc 4.4.4 installed that is used to build illumos.
67If you are not building this with SmartOS, you may need to modify the
68Makefile such that CC is pointing to the correct gcc.
69
70Verify that you either have SUNWmake or GNU make installed.
71
72### Building
73
74To build, simply use the default make target:
75
76    $ make
77
78To check style, header files, and other various nits:
79
80    $ make check
81
82Installing illumos-kvm
83----------------------
84
85### System requirements
86
87To run illumos-kvm, you will need an illumos that has the fix for issue
881347 (integrated on 2011-08-11). Further, your machine will need to
89support VMX.  To see if your machine supports VMX, run `isainfo -v` and
90look for `vmx`, e.g.:
91
92      % isainfo -v
93      64-bit amd64 applications
94            vmx sse4.2 sse4.1 ssse3 popcnt tscp cx16 sse3 sse2 sse fxsr mmx
95            cmov amd_sysc cx8 tsc fpu
96      32-bit i386 applications
97            vmx sse4.2 sse4.1 ssse3 popcnt tscp ahf cx16 sse3 sse2 sse fxsr mmx
98            cmov sep cx8 tsc fpu
99
100If you do not see `vmx` in this output, the `kvm` driver will be unable to
101attach.
102
103### Required binaries
104
105There are two mandatory artifacts to install, and two optional component:
106
107* `kvm` is the driver itself
108* `kvm.conf` is the driver configuration file
109* `kvm.so` is the mdb module
110* `JOY_kvm_link.so` is the devfsadm plugin
111
112On the target machine, place `kvm` in `/kernel/drv/amd64` and `kvm.conf`
113in `/kernel/drv`. Place `JOY_kvm_link.so` in `/usr/lib/devfsadm/linkmod` then:
114
115    # add_drv kvm
116
117You can verify that the driver installed and attached properly by checking for
118its presence in /dev.
119
120    # ls -l /dev/kvm
121
122Running illumos-kvm
123-------------------
124
125To run KVM, you will need the build product of the illumos-kvm-cmd repo:
126`qemu-system-x86_64`; please follow the instructions in the illumos-kvm-cmd
127repo to execute QEMU such that KVM is enabled.
128
129Monitoring illumos-kvm
130----------------------
131
132Once one or more VMs are running, there is a variety of tooling to help
133understand the operating characteristics of the system.
134
135### kvmstat
136
137The `kvmstat` command, found in the illumos repository, can be used to monitor
138VMs. For example, here is one second of `kvmstat` output from a machine
139running two VMs (one 2 VCPU instance running Linux; another 4 VCPU instance
140running the illumos-derived SmartOS):
141
142       pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
143      4668    0 |     23 :      6      0      0      1      0 |      6     16      0
144      4668    1 |     25 :      6      1      0      1      0 |      6     16      0
145      5026    0 |  17833 :    223   2946    707    106      0 |   3379  13315      0
146      5026    1 |  18687 :    244   2761    512      0      0 |   3085  14803      0
147      5026    2 |  15696 :    194   3452    542      0      0 |   3568  11230      0
148      5026    3 |  16822 :    244   2817    487      0      0 |   3100  12963      0
149
150As for the meaning of the columns, they are explained with `kvmstat -h`:
151
152      # kvmstat -h
153      Usage: kvmstat [interval [count]]
154
155        Displays statistics for running kernel virtual machines, with one line
156        per virtual CPU.  All statistics are reported as per-second rates.
157
158        The columns are as follows:
159
160          pid    =>  identifier of process controlling the virtual CPU
161          vcpu   =>  virtual CPU identifier relative to its virtual machine
162          exits  =>  virtual machine exits for the virtual CPU
163          haltx  =>  virtual machine exits due to the HLT instruction
164          irqx   =>  virtual machine exits due to a pending external interrupt
165          irqwx  =>  virtual machine exits due to an open interrupt window
166          iox    =>  virtual machine exits due to an I/O instruction
167          mmiox  =>  virtual machine exits due to memory mapped I/O
168          irqs   =>  interrupts injected into the virtual CPU
169          emul   =>  instructions emulated in the kernel
170          eptv   =>  extended page table violations
171
172### kstat
173
174As one might expect, `kvmstat` is implemented in terms of kstat.  You
175can use `kstat(1)` to browse the kstats from the `kvm` module:
176
177       # kstat -m kvm
178       ...
179       module: kvm                      instance: 0
180       name:   vcpu-4                   class:    misc
181        crtime                          4407.142410068
182        exits                           5367443
183        fpu-reload                      57302
184        halt-exits                      317275
185        halt-wakeup                     8991
186        host-state-reload               503920
187        hypercalls                      0
188        insn-emulation                  3043881
189        inst-emulation-fail             0
190        invlpg                          0
191        io-exits                        237191
192        irq-exits                       1668
193        irq-injections                  320339
194        irq-window-exits                1635
195        mmio-exits                      617
196        nmi-injections                  0
197        nmi-window-exits                0
198        pf-fixed                        163629
199        pf-guest                        0
200        pid                             3949
201        request-irq-exits               0
202        signal-exits                    460
203        snaptime                        43219.723435123
204        zonename                        global
205
206       module: kvm                      instance: 4
207       name:   vm                       class:    misc
208        crtime                          4407.1241134
209        lpages                          0
210        mmu-cache-miss                  950
211        mmu-flooded                     0
212        mmu-pte-updated                 0
213        mmu-pte-write                   56360
214        mmu-pte-zapped                  0
215        mmu-recycled                    0
216        mmu-unsync-page                 0
217        pid                             3949
218        remote-tlb-flush                1511
219        snaptime                        43219.723875091
220        zonename                        global
221
222
223### DTrace
224
225While there is not currently a stable KVM provider, there are many SDT probes
226in KVM; `dtrace -l -m sdt:kvm` to list these.
227
228Of these, of particular note are the `kvm-guest-entry` and `kvm-guest-exit`
229probes, which fire upon entry to and exit from a guest virtual machine.  To
230determine context, one can use the `vmregs` variable present in illumos.
231
232For example, here's a simple script that shows histograms of time spent in VM
233guests on a per-PID and per-VCPU basis:
234
235        #pragma D option quiet
236
237        kvm-guest-entry
238        {
239                self->entry = timestamp;
240        }
241
242        kvm-guest-exit
243        /self->entry/
244        {
245                @[pid, vmregs[VMX_VIRTUAL_PROCESSOR_ID]] =
246                    quantize(timestamp - self->entry);
247        }
248
249        END
250        {
251                printa("pid %d, vcpu %d: %@d\n", @);
252        }
253
254Here's what the output of running the above might look like:
255
256      pid 3949, vcpu 1:
257           value  ------------- Distribution ------------- count
258             512 |                                         0
259            1024 |@@@@@@@@@@@@@                            26805
260            2048 |@@@@@                                    11641
261            4096 |@@@@@@@                                  14187
262            8192 |@                                        1559
263           16384 |@                                        2931
264           32768 |@@@                                      5653
265           65536 |@@@@                                     8385
266          131072 |@@@                                      6926
267          262144 |@@@                                      6639
268          524288 |                                         785
269         1048576 |                                         0
270
271There are many other ways in which DTrace can be used to understand either
272host or guest behavior; see the `tools` subdirectory from some sample D
273scripts.
274
275### mdb
276
277The `kvm.so` build product is an mdb module that contains several useful
278commands, including a `kvm` walker to iterate over all `struct kvm`
279structures.
280
281Contributing to illumos-kvm
282---------------------------
283
284Unless and until its volume dictate that it be elsewhere, illumos KVM
285discussion should be on the `illumos-developer` mailing list.
286Contributions are happily accepted; please send patches to
287`illumos-developer`.
288
289