Install the F8/F9 keys.

[util-vserver.git] / doc / intro.txt

   1. [1]Introduction [new.gif]
   1.1 [2]Who needs that [new.gif]
   2. [3]Principles [new.gif]
   2.1 [4]Non reversible isolation [new.gif]
   2.2 [5]Isolation areas [new.gif]
   2.3 [6]New system calls [new.gif]
   2.4 [7]Limiting super-user: The capabilities system [new.gif]
   2.5 [8]Enhancing the capability system [new.gif]
   2.6 [9]Playing with the new system calls [new.gif]
   2.6.1 [10]Playing with /usr/sbin/chcontext [new.gif]
   2.6.2 [11]Playing with /usr/sbin/chcontext as root [new.gif]
   2.6.3 [12]Playing with /usr/sbin/chbind [new.gif]
   2.6.4 [13]Playing with /usr/sbin/reducecap [new.gif]
   2.7 [14]Unification [new.gif]
   3. [15]Applications [new.gif]
   3.1 [16]Virtual server [new.gif]
   3.2 [17]Per user fire-wall [new.gif]
   3.3 [18]Secure server/Intrusion detection [new.gif]
   3.4 [19]Fail over servers [new.gif]
   4. [20]Installation [new.gif]
   4.1 [21]The packages [new.gif]
   4.2 [22]Setting a virtual server [new.gif]
   4.3 [23]Basic configuration of the virtual server [new.gif]
   4.4 [24]Entering the virtual server [new.gif]
   4.5 [25]Configuring the services [new.gif]
   4.6 [26]Starting/Stopping the virtual server [new.gif]
   4.7 [27]Starting/Stopping all the virtual servers [new.gif]
   4.8 [28]Restarting a virtual server from inside [new.gif]
   4.9 [29]Executing tasks at vserver start/stop time [new.gif]
   4.10 [30]Issues [new.gif]
   4.11 [31]How real is it ? [new.gif]
   5. [32]Features [new.gif]
   6. [33]Future directions [new.gif]
   6.1 [34]User controlled security box [new.gif]
   6.2 [35]Kernel enhancements [new.gif]
   6.2.1 [36]Per context disk quota [new.gif]
   6.2.2 [37]Global limits [new.gif]
   6.2.3 [38]Scheduler [new.gif]
   6.2.4 [39]Security issues [new.gif]
   6.2.4.1 [40]/dev/random [new.gif]
   6.2.4.2 [41]/dev/pts [new.gif]
   6.2.4.3 [42]Network devices [new.gif]
   7. [43]Alternative technologies [new.gif]
   7.1 [44]Virtual machines [new.gif]
   7.2 [45]Partitioning [new.gif]
   7.3 [46]Limitation of those technologies [new.gif]
   8. [47]Conclusion [new.gif]
   9. [48]Download [new.gif]
   10. [49]References [new.gif]

               Virtual private servers and security contexts

   Running independent Linux servers inside a single PC is now possible.
   They offer many advantages, including higher security, flexibility and
   cost reduction.

   NEW

Introduction

   Linux computers are getting faster every day. So we should probably
   end up with less, more powerful servers. Instead we are seeing more
   and more servers. While there are many reasons for this trend (more
   services offered), the major issue is more related to security and
   administrative concerns.

   Is it possible to split a Linux server into virtual ones with as much
   isolation as possible between each one, looking like real servers, yet
   sharing some common tasks (monitoring, backup, ups, hardware
   configuration, ...) ?

   We think so ... NEW

Who needs that

   The short answer is everybody, or everybody managing a server. Here
   are some applications:

     * Hosting: Complete general purpose hosting (Running many
       independent servers in one box).
     * Experimentation: You are toying with a new services and do not
       want to impact the production services on the same machine.
     * Education: Each student has its own server with root password.
     * Personal security box: Run un-trusted applications with complete
       control over their interaction with the rest of the computer and
       the network.
     * Managing several "versions" of the same server/project and turning
       on/off each version independantly.

   Just think about all the viruses and worms out there, you end up with
   a big everybody using a computer needs this. :-) NEW

Principles

   NEW

Non reversible isolation

   Unix and Linux have always had the chroot() system call. This call was
   used to trap a process into a sub-directory. After the system-call,
   the process is led to believe that the sub-directory is now the root
   directory. This system call can't be reversed. In fact, the only thing
   a process can do is trap itself further and further in the file-system
   (calling chroot() again).

   The strategy is to introduce new system calls trapping the processes
   in other areas within the server. NEW

Isolation areas

   A virtual server is isolated from the rest of the server in 5 areas:

     * File system
       The vserver is trapped into a sub-directory of the main server and
       can't escape. This is done by the standard chroot() system call
       found on all Unix and Linux boxes.
     * Processes
       The vserver can only see the processes in the same security
       context. Even the root server can't see the processes in vservers,
       making the root server less "dangerous" to use. A special
       mechanism (context number 1) exists to view all processes though
       (Limited to root in the root server).
     * Networking
       The vserver is assigned a host name and an IP number. The server
       can only use this IP number to establish services and client
       connection. Further, this restriction is transparent.
     * Super user capabilities
       The super user running in a vserver has less privileges than the
       normal Linux root user. For example, it can't reconfigure the
       networking and many aspect of the system. It can't mount devices,
       can't access block devices and so on.
       Roughly. the vserver super-user has full control over all files
       and processes in the vserver and that's pretty much it.
     * System V inter process communications
       Sysv IPC resources are private to each vserver. The security
       context is used as an extra key to select and assign resources.

   Those facilities are used together to create a runtime environment for
   virtual servers. But they can be used independently to achieve various
   goals. NEW

New system calls

   The new system calls, as well as the existing chroot() system call are
   sharing one common feature: Their effect can't be reversed. Once you
   have executed one of those system call (chroot, new_s_context,
   set_ipv4root), you can't get back. This affects the current process
   and all the child processes. The parent process is not influenced.

     * new_s_context (int ctx)
       This system call sets a new security context for the current
       process. It will be inherited by all child processes. The security
       context is just an id, but the system call makes sure a new unused
       one is allocated.
       A process can only see other processes sharing the same security
       context. When the system boot, the original security context is 0.
       But this one is not privileged in anyway. Processes member of the
       security context 0 can only interact (and see) processes member of
       context 0.
       This system call isolates the processes space.
     * Setting the capabilities ceiling
       This is handle by the new_s_context system call as well. This
       reduces the ceiling capabilities of the current process. Even
       setuid sub-process can't grab more capabilities. The capability
       system found since Linux 2.2 is explained later in this document.
     * set_ipv4root(unsigned long ip)
       This system call locks the process (and children) into using a
       single IP when they communicate and when they installs a service.
       This system call is a one shot. Once a process have set its IPV4
       (Internet Protocol Version 4) address to something different from
       0.0.0.0, it can't change it anymore. Children can't change it
       either.
       If a process tries to bind a specific IP number, it will succeed
       only if this corresponds to the ipv4root (if different from
       0.0.0.0). If the process bind to any address, it will get the
       ipv4root.
       Basically, once a process is locked to a given ipv4root it is
       forced to use this IP address to establish a service and
       communicate. The restriction on services is handy: Most service
       (Web servers, SQL servers) are binding to address 0.0.0.0. With
       the ipv4root sets to a given IP you can have two virtual servers
       using the exact same general/vanilla configuration for a given
       services and running without any conflict.
       This system calls isolate the IP network space.

   Those system calls are not privileged. Any user may issue them. NEW

Limiting super-user: The capabilities system

   Once you have created a virtual environment where processes have a
   limited view of the file-system, can't see processes outside of their
   world and can only use a single IP number, you still must limit the
   damages those processes can do. The goal is to run virtual
   environments and provide some root privileges.

   How do you limit those root processes from taking over the system, or
   even just re-booting it. Enter the capability system. This is not new,
   but we suspect many people have never heard of it.

   In the old Unix/Linux days, user root (user ID 0) could do things
   other user ID could not. All over the place in the kernel, system
   calls were denying access to some resources unless the user ID of the
   process (effective ID in fact) was 0. Plain zero.

   The only way a process with user ID 0 could loose some privileges was
   by changing to another ID. Unfortunately this was an all or nothing
   deal. Enter the capabilities.

   Today, the difference between root and the other users is the
   capability set. User root has all capabilities and the other users
   have none. The user ID 0 does not mean anything special anymore. There
   are around 30 capabilities defined currently. A process may request to
   loose a capability forever. It won't be able to get it back.

   Capabilities allows a root process to diminish its power. This is
   exactly what we need to create custom super-user. A super-user process
   in a virtual server would have some privileges such as binding port
   below 1024, but would not be able to reconfigure the network or reboot
   the machine. Check the file /usr/include/linux/capability.h to learn
   which one are available.

   Note that the new system calls (new_s_context and set_ipv4root) are
   not controlled by capabilities. They are by nature irreversible. Once
   a virtual server is trapped in a chroot/s_context/ipv4root box, it
   can't escape from the parameters of this trap.

   NEW

Enhancing the capability system

   The Linux capability system, is still a work in progress. At some
   point, we expect to see capabilities attached to programs,
   generalizing the setuid concept. A setuid program would become a
   program with all capability granted.

   For now, this is not available. As explained above a process may
   request to loose capabilities and its child process will be trapped
   with a smaller capability set.

   Well, ..., it does not work that way. Unfortunately, until
   capabilities could be assigned to program, we still need a way to get
   back capabilities even in a child process. So the irreversible logic
   of the capabilities is kind of short circuited in the kernel.

   To solve this, we have introduced a new per-process capability ceiling
   (cap_bset). This one represents the capability set inherited by child
   process, including setuid root child process. Lowering this ceiling is
   irreversible for a process and all its child.

   This ceiling is handled by the new_s_context system call and the
   reducecap and chcontext utilities (part of the vserver package).

   Using this, we can setup a virtual server environment where root has
   less capabilities, so can't reconfigure the main server.

   NEW

Playing with the new system calls

   The vserver package provides 3 utilities to make use of the new system
   calls. We will describe shortly how they work and provide few example.
   We invite the reader to try those example, so it has a better feel and
   trust.

   After re-booting with a kernel implementing the new system calls, and
   installing the vserver package, one is ready to do experiment. You do
   not need to be root to test those new utilities. None of them is
   setuid either. NEW

Playing with /usr/sbin/chcontext

   The /usr/sbin/chcontext utility is used to enter into a new security
   context. The utility switch the security context and execute a
   program, specified on the command line. This program is now isolated
   and can't see the other processes running on the server.

   The experiment with this, start two command windows (xterm), as the
   same user ID. In each window execute the following commands:

                                   xterm

   Using chcontext: first window

/usr/sbin/chcontext /bin/sh
pstree
killall xterm
exit

   Using chcontext: second window
   In  the  first window, you start the xterm command (or any command you
   like).  In  the second window you execute chcontext. This starts a new
   shell. You execute pstree and see very little. You attempt to kill the
   xterm  and  you  fail. You exit this shell and you are back seeing all
   processes.

   Here  is  another example. You switch context and you get a new shell.
   In  this  shell  you start an xterm. Then you switch context again and
   start another sub-shell. Now the sub-shell is again isolated.

/usr/sbin/chcontext /bin/sh
pstree
xterm &
pstree
# Ok now you see your xterm
/usr/sbin/chcontext /bin/sh
pstree
# the xterm is not there, killall will fail
killall xterm
# Now we exit the shell
exit
pstree
# the xterm is there
killall xterm
# Ok, it is gone
exit
# We are back to the initial security context

   Using chcontext several times
   Processes isolated using chcontext are doubly isolated: They can't see
   the  other  processes on the server, but the other processes can't see
   them  either.  The  original  security context (0) when you boot is no
   better than the other: It sees only process in security context 0.

   While  playing  with  chcontext,  you  will  notice  an exception. The
   process  1  is  visible  from every security context. It is visible to
   please  utilities  like  pstree.  But  only root processes in security
   context 0 are allowed to interact with it. NEW

Playing with /usr/sbin/chcontext as root

   The  new_s_context  system  call  has  a  special  semantic  for  root
   processes  running  in security context 0 and having the CAP_SYS_ADMIN
   capability: They can switch to any context they want.

   Normally,  new_s_context allocates a new security context by selecting
   an  unused one. It walks all processes and find an ID (an integer) not
   currently in use.

   But  root  in  security  context 0 is allowed to select the context it
   wants.  This  allow the main server to control the virtual server. The
   chcontext  utility  has the --ctx option to specify the context ID you
   want.

   To help manage several virtual server, given that the security context
   0  can't  see  processes in other security context, it is a good thing
   root  in  the  main server (security context 0) is allowed to select a
   specific  context.  Cool.  But  we  also  need  a way to have a global
   picture  showing  all  processes in all security context. The security
   context  1  is reserved for this. Security context 1 is allowed to see
   all  processes  on the server but is not allowed to interact with them
   (kill them).

   This  special  feature  was  allocated to security context 1 and not 0
   (the  default when you boot) to isolate virtual servers from the main.
   This  way,  while  maintaining  services on the main server, you won't
   kill service in vservers accidentally.

   Here is an example showing those concepts:

# You must be root, running X
# We start an xterm in another security context
/usr/sbin/chcontext xterm &
# We check, there is no xterm running, yet we can
# see it.
ps ax | grep xterm
# Are we running in security context 0
# We check the s_context line in /proc/self/status
cat /proc/self/status
# Ok we in security context 0
# Try the security context 1
/usr/sbin/chcontext --ctx 1 ps ax | grep xterm
# Ok, we see the xterm, we try to kill it
/usr/sbin/chcontext --ctx 1 killall xterm
# No, security context 1 can see, but can't kill
# let's find out in which security context this
# xterm is running
/usr/sbin/chcontext --ctx 1 ps ax | grep xterm
# Ok, this is PID XX. We need the security context
/usr/sbin/chcontext --ctx 1 cat /proc/XX/status
# We see the s_context, this is SS.
# We want to kill this process
/usr/sbin/chcontext --ctx SS killall xterm

   chcontext as root
   The  /usr/sbin/vpstree  and /usr/sbin/vps commands are supplied by the
   vserver package. They simply runs ps and pstree in security context 1.
   NEW

Playing with /usr/sbin/chbind

   The  chbind  utility  is  used to lock a process and its children into
   using  a  specific  IP  number.  This  applies  to services and client
   connection as well. Here are few examples. Execute them as root:

# httpd is running
netstat -atn | grep ":80 "
# We see a line like this
# tcp    0   0 0.0.0.0:80    0.0.0.0:*      LISTEN
# Now we restart httpd, but we lock it so it
# uses only the main IP of eth0
/usr/sbin/chbind --ip eth0 /etc/rc.d/init.d/httpd restart
netstat -atn | grep ":80 "
# Now we see a line like
# tcp    0   192.168.1.1:80    0.0.0.0:*      LISTEN
# httpd.conf was not changed.
# Now, restart it normally
/etc/rc.d/init.d/httpd restart
# Now we experiment with client socket
# Log using telnet
telnet localhost
# Once logged in
netstat -atn | grep ":23 "
# You should see a line showing a connection from
# 127.0.0.1 to 127.0.0.1 like this
# tcp  0  0 127.0.0.1:23   127.0.0.1:32881     ESTABLISHED
exit
# Now we do the telnet again, bug forcing it to select a specific IP
/usr/sbin/chbind --ip eth0 telnet localhost
# Log in and then execute
netstat -atn | grep ":23 "
# You will get something like
# tcp  0  0 127.0.0.1:23   192.168.3.9:32883   ESTABLISHED

   Using /usr/sbin/chbind
   NEW

Playing with /usr/sbin/reducecap

   The  reducecap  utility  is  used to lower the capability ceiling of a
   process  and  child process. Even setuid program won't be able to grab
   more capabilities.

# You are not root now
# What is the current capability ceiling
cat /proc/self/status
# The capBset line presents mostly 1s.
/usr/sbin/reducecap --secure /bin/sh
cat /proc/self/status
# The capBset now shows many more 0s.
# The capEff shows all 0s, you have no privilege now
# We su to root
su
cat /proc/self/status
# capEff is much better now, but there are still many 0s
# Now we try to see if we are really root
tail /var/log/messages
# So far so good, we see the content
/sbin/ifconfig eth0
/sbin/ifconfig eth0 down
# No way, we can't configure the interface. In fact
# we have lost most privilege normally assigned to root
exit

   Using /usr/sbin/reducecap
   NEW

Unification

   Installing a virtual private server copies a linux installation inside
   a  sub-directory.  It  is  a  linux inside linux. If you intend to run
   several  vservers  on the same box (which you will certainly do :-) ),
   you  will end up using a lot of disk space needlessly: Each vserver is
   made  up  hundreds of megabytes of the same stuff. This is a big waste
   of disk space.

   A  solution  is  to  use  hard links to connect together common files.
   Using  the  package information, we can tell which packages are shared
   between  various  vservers,  which  files  are configuration files and
   which   are  not  (binaries,  libraries,  resource  files,  ...).  Non
   configuration  files  may  be  linked together saving a huge amount of
   disk space: A 2 GIG rh7.2 installation shrinks to 38megs.

   Using  hard  links  is  cool,  but  may  be  a problem. If one vserver
   overwrite  one file, say /bin/ls, then every vserver will inherit that
   change.  Not  fun!  The  solution is to set the immutable bit on every
   linked  file.  A  file  with  such a bit on can't be modified, even by
   root.  The  only  way  to  modify it is to turn off the bit first. But
   within a vserver environment, even root is not allowed to perform this
   task.  So  linked  file,  turned  immutable, are now safe: They can be
   shared between vservers without side effects: Cool!

   Well,  there is still another side effect. All vservers are now locked
   with  the  same files. We are saving a lot of disk space, but we pay a
   very heavy price for that: Vservers can't evolve independantly.

   A  solution  was  found.  A  new bit call immutable-linkage-invert was
   added.  Combined  with  the immutable bit, a file may not be modified,
   but   may   be   unlinked.   Unlinking  a  file  in  Unix/Linux  means
   disconnecting  the  name  from the data and removing the name from the
   directory. If the data is still referenced by one or more vservers, it
   continue to exist on disk. So doing "rm /bin/ls" on a vserver, removes
   the  file  /bin/ls  for  that  vserver and that's all. If all vservers
   perform this task, then /bin/ls (the data) will be forgotten completly
   and the disk space will be recovered.

   Using  this  trick,  a  vserver gets back its independance. It becomes
   possible  to  update  packages  by  using  the unlink/update sequence:
   Unlink  /bin/ls  first  and  then  put  a  new copy in place. Luckily,
   package manager works this way.

   To keep this story short (probably too late :-) ), a unified vserver:
     * Uses little disk space
     * Can't interfere with other vservers by changing one of its file.
     * Can   perform  package  update/deletion  normally  using  standard
       practice.

   NEW

Applications

   NEW

Virtual server

   The  first  goal  of this project is to create virtual servers sharing
   the same machine. A virtual server operate like a normal Linux server.
   It runs normal services such as telnet, mail servers, web servers, SQL
   servers. In most cases, the services run using standard configuration:
   The services are running unaware of the virtual server concept.

   Normal  system  administration  is performed with ordinary admin tool.
   Virtual servers have users account and a root account.

   Packages are installed using standard packages (RPMs for example).

   There  are  few  exceptions. Some configuration can't be done inside a
   virtual  server. Notably the network configuration and the file-system
   (mount/umount) can't be performed from a virtual server.

   NEW

Per user fire-wall

   The  set_ipv4root()  system  call  may  be  used  to differentiate the
   various users running on an application server. If you want to setup a
   fire-wall  limiting what each user is doing, you have to assign one IP
   per user, even if they are running application on the same server. The
   chbind utility may be used to achieve that. NEW

Secure server/Intrusion detection

   While it can be interesting to run several virtual servers in one box,
   there  is  one  concept  potentially  more generally useful. Imagine a
   physical  server  running a single virtual server. The goal is isolate
   the  main  environment  from any service, any network. You boot in the
   main  environment,  start  very  few services and then continue in the
   virtual server. The service in the main environment could be

     * Un-reachable from the network.
     * The  system  log  daemon. While virtual server could log messages,
       they  would  be unable to change/erase the logs. So even a cracked
       virtual server would not be able the edit the log.
     * Some  intrusion detection facilities, potentially spying the state
       of the virtual server. For example tripwire could run there and it
       would be impossible to circumvent its operation or trick it.

   NEW

Fail over servers

   One  key  feature  of  a  virtual  server is the independence from the
   actual  hardware.  Most hardware issues are irrelevant for the virtual
   server installation. For example:

     * Disk  layout,  partitions  and  the  /etc/fstab configuration. The
       virtual server has a dummy /etc/fstab.
     * Network devices.
     * Processor type, number of processor (kernel selection).

   The  main  server  acts as a host and takes care of all those details.
   The  virtual  server  is just a client and ignores all the details. As
   such, the client can be moved to another physical server will very few
   manipulations.  For  example,  to  move the virtual server v1 from one
   physical one computer to another, you do

     * Turn it off
       /usr/sbin/vserver v1 stop
     * Copy the file /etc/vservers/v1.conf to the new server.
     * Copy all files in /vservers/v1 to the new server
     * On the new server, start the vserver v1
       /usr/sbin/vserver v1 start

   As you see, there is no adjustment to do:

     * No user account merging.
     * No hardware configuration to fix.

   This  opens  the  door  to  fail over servers. Imagine a backup server
   having  a  copy  of many virtual servers. It can take over their tasks
   with a single command. Various options exists for managing this backup
   server:

     * rsync to synchronize the data.
     * Network block devices to synchronize the data in real time.
     * Sharing the installation over a SAN (storage area network).
     * Heartbeat for automatic monitoring/fail over.

   NEW

Installation

   NEW

The packages

     * The kernel
       We  are  supplying a patched 2.4.20 kernel. You will find [50]here
       the kernel, the .config and the patch.
       To  install  the  kernel,  just  untar it. This will create a file
       /boot/kernel-2.4.20ctx-17  (ctx stands for security context) and a
       directory /lib/modules/2.4.20ctx-17.
       Then,  you  need  to  update your boot loader. For lilo, you add a
       section like this at the end of the file /etc/lilo.conf


image=/boot/vmlinuz-2.4.20ctx-17
        label=linux2.4.20ctx-17
        read-only
        root=current

       lilo.conf section to add
       Change   the  /dev/XXXX  to  your  root  partition.  Then  execute
       /sbin/lilo.
       Reboot  and  select  the  proper  kernel.  This  kernel  is  fully
       compatible  with  a normal 2.4.20 and will perform without any new
       configuration.  Note  that  the supplied kernel does not carry all
       the features and modules found on the various distributions.
     * The vserver package
       This  package  provides the various utilities to make use of those
       new system calls. The package also provides a complete solution to
       implement virtual servers. We describe the major components here.
          + /usr/sbin/chcontext
            This is the utility to request a new security context. It can
            be  used to lower the capability ceiling. Execute it to learn
            more.
          + /usr/sbin/chbind
            This  is the utility to select one IP number and assign it to
            a process and its children.
          + /usr/sbin/newvserver (in vserver-admin)
            Front-end to help create new virtual servers.
          + /usr/sbin/reducecap
            This  utility  is  used  to  lower  the capability ceiling of
            children processes.
          + /usr/sbin/vdu
            A  trimmed  down  "du" command reporting space usage of files
            with  a  single link. Useful to tell how much space a unified
            vserver is using.
          + /usr/sbin/vkill
            Locate  the security context associated with a process, enter
            it  and  kill  the  process.  Generally  used  after you have
            located a process with vtop, vpstree or vps.
          + /usr/sbin/vps
            Execute the ps command in security context 1 so all processes
            in  all  vservers are shown. The security context and vserver
            name are mapped inside the report.
          + /usr/sbin/vpstree
            Execute  the  pstree  command  in  security  context 1 so all
            processes in all vservers are shown.
          + /usr/sbin/vrpm
            Apply  an  rpm  command  in several (or all) vservers. Useful
            when you wish to update many vservers with the same package.
        /usr/sbin/vrpm server1 server2 -- -Uvh /tmp/*.rpm
        /usr/sbin/vrpm ALL -- -Uvh /tmp/*.rpm
            After  updating  many  packages in different vservers you may
            want  to  re-unify  them  to recover disk space (and increase
            cache effectivity). You can do this using the vunify command,
            or  simply  by  using the --unify option to the vrpm command.
            After  performing  the  rpm  updates,  vrpm  will trigger the
            vunify utility on the vservers for the updated packages.
        /usr/sbin/vrpm --unify server1 server2 -- -Uvh /tmp/*.rpm
          + /usr/sbin/vserver
            This  is  the  wrapper  to start, stop and administer virtual
            servers.
          + /usr/sbin/vserver-stat
            Produce  a  small  report  showing  the  activity  in  active
            security  context.  The report presents the number of process
            in  each  active  security context as well as the name of the
            vserver associated with this context.
          + /usr/sbin/vtop
            Execute  the  top  command  in  security  context  1  so  all
            processes in all vservers are shown.
          + /etc/rc.d/init.d/vservers
            This  is  an init script used to start all virtual servers at
            boot  time  and  stop  them  at  shutdown  time. Only virtual
            servers  with  ONBOOT=yes  are  started  at  boot  time.  All
            vservers are stopped at shutdown time.
          + /etc/rc.d/init.d/rebootmgr
            This  is a daemon listening to requests from virtual servers.
            It   can  either  restart  or  stop  a  virtual  server.  The
            /sbin/vreboot  and  /sbin/vhalt  utilities  are  used to send
            request to the reboot manager.
          + /sbin/vreboot and /sbin/vhalt
            Those  utilities  are  copied  in  each  virtual server. They
            connect  to  the  reboot manager (rebootmgr) server using the
            /dev/reboot  Unix  domain socket and request either a restart
            or  a  stop  of  the virtual server. The reboot manager issue
            either    a    "/usr/sbin/vserver    vserver    restart"   or
            "/usr/sbin/vserver  vserver  stop"  command.  This allows the
            virtual server administrator to test if all automatic service
            are properly restarted at boot time.
          + /usr/lib/vserver/vdu
            This is a limited clone of the du command. It skips file with
            more  than one link. It is used to evaluate the disk usage of
            an  unified  vserver.  Using  the  normal du for this task is
            misleading since it will count all unified files.

   NEW

Setting a virtual server

   To  set  a virtual server, you need to copy in a sub-directory a Linux
   installation. One way to achieve that is to copy some parts of the the
   current  server  by  issuing the command vserver XX build, where XX is
   the  name of the virtual server (pick one). This basically does (Well,
   it does a little more than that, but this give you an idea):

mkdir /vservers/XX
cd /vservers/XX
cp -ax /sbin /bin /etc /usr /var /dev /lib .
mkdir proc tmp home
chmod 1777 tmp

   Building a virtual server

   This  is normally done using the command /usr/sbin/newvserver. This is
   a  text mode/graphical front-end allowing to setup the vserver runtime
   and configure it. NEW

Basic configuration of the virtual server

   A  virtual  private server has a few settings. They are defined in the
   file /etc/vservers/XX.conf where XX is the name of the virtual server.
   This  is  a  simple  script  like  configuration. Here are the various
   parameters:

     * IPROOT
       In  general,  each  virtual server is tied to one IP using the new
       set_ipv4root  system  call.  This  way several servers may run the
       same  services  without  conflict.  Enter the IP number (a name is
       also valid if defined in the DNS).
       Since  kernel  ctx-12, you can assign more than one IP number to a
       vserver.  Enter  them  separated  by  spaces  and  don't forget to
       enclose them with quotes.
       Bu default, all IPs are configured as an IP alias on the IPROOTDEV
       device  (if defined). If you want to attach the various IP numbers
       to  different  devices,  specify  the device as a prefix to the IP
       number like this:

IPROOT="eth0:192.168.1.2 eth1:192.168.3.1 192.168.1.4"

       IPROOT using multiple devices
       In  the  example above, the IP 192.168.1.4 will be installed as an
       IP alias on the device IPROOTDEV.
       Use  "IPROOT=0.0.0.0"  or  "IPROOT=" if you do not want to tie the
       vserver  at all. It will be allowed to use any IP available on the
       server.
     * IPROOTDEV
       This  is  the  network device use to setup the IP alias defined by
       IPROOT.  This  is generally eth0. If you define this variable, the
       IP  alias will be configured when you start the virtual server and
       un-configure when you stop it.
     * IPROOTMASK
       Netmask  used  to  setup  the  IP  alias.  Uses the netmask of the
       IPROOTDEV device by default. Seldom used.
       If  you have several IPs on one vserver and want to have different
       netmask for each, use the following syntax:

IPROOT="eth0:192.168.1.2 eth1:192.168.3.1/255.255.255.192"

       IPROOT using different netmask
       192.168.1.2  will  use the netmask of eth0, while 192.168.3.1 will
       use the specified netmask: 255.255.255.192.
     * IPROOTBCAST
       Broadcast  address  used to setup the IP alias. Uses the broadcast
       of the IPROOTDEV device by default. Seldom used.
     * ONBOOT
       The vserver package supplies the vservers service. This service is
       installed  in  the  main  server. It is used to start and stop the
       virtual server at boot and shutdown time.
       Virtual  server  with  ONBOOT=yes  will  be  started  and  stopped
       properly like any other services of the main server.
       Once a virtual server is properly configured, it is a good idea to
       set this parameter to yes.
     * S_CAPS
       You  enter  here  the  various capability you want to grant to the
       vserver. By default, a vserver is left with much less capabilities
       than the root server. For example, a vserver is not allowed to use
       raw  socket. This explains why the ping command fails. S_CAPS lets
       you  enumerate  the  capability  you  want to keep in the vserver.
       CAP_NET_RAW will give back ping ability for example.
     * S_CONTEXT
       This  is  optional. In general the security context ID is selected
       by  the kernel. An unused one is selected. If you select an ID (an
       integer  greater than 1), make sure you select a different one for
       each  server. Again, in most cases, you do not need to select one.
       Leave the line commented.
     * S_DOMAINNAME
       A virtual server may have a different NIS domainname than the main
       server. You set it here. If you leave the field empty, the vserver
       will  inherit the same NIS domain as the root server. Enter "none"
       to reset the NIS domain name for this vserver.
     * S_HOSTNAME
       Many  services  (Apache  for  one) use the host name to setup some
       defaults. A virtual server may have a different host name than the
       main server. It is a good idea to fill this line.
     * S_NICE
       The  is  an  optional  priority  level.  It  is an integer ranging
       between  from -20 to 19. Well it is the nice level in fact, so -20
       means  the  highest  priority  and 19 the lowest (the highest nice
       level). All processes in the virtual server will be locked to this
       level (or higher niceness).
       Event root is locked and can't get more priority.
       Note  that  this  has  limited  usefulness.  The  kernel  does not
       differentiate  processes running in different security context for
       scheduling  (for  now  :-)  ).  This  means that a virtual servers
       running many low priority processes may nevertheless claim a large
       share of CPU.
     * S_FLAGS
       This is used to set various flags. Here are the supported flags:
          + lock
            This  flags  prevents  the  vserver from setting new security
            contexts.
          + sched
            It  kind  of  unifies  the  processes  in  a  vserver  from a
            scheduler  view  point.  All processes are weighted as single
            one when compared to other processes in the real server. This
            prevents a vserver from taking to much CPU resources.
          + nproc
            Make the ulimit maximum user process global to the vserver.
          + private
            Once  set on a vserver security context, no other process can
            enter  it.  Even  the  root  server  is  unable  to enter the
            context.  It  can see the process list using security context
            1, but can't signal or trace the process.
          + fakeinit
            This  assigned  the  current  process  so  it  works like the
            process  number  1. Using this trick, a normal /sbin/init may
            be  run  in a vserver. The /usr/sbin/vserver command will use
            /sbin/init to start and stop a vserver. A properly configured
            /etc/inittab is needed though.
               o Processes  loosing  their  parent  are  reparent to this
                 process.
               o getppid()  done by child process of this process returns
                 1.
               o getpid() done by this process returns 1.
               o This  process is not shown in /proc since process number
                 1 is always shown.
               o An  "initpid"  entry  is  available in /proc/*/status to
                 tell which process is the fake init process.
     * ULIMIT
       This contains the command line option to the ulimit shell command.
       You enter here whatever parameters accepted by ulimit. Here is the
       default when you create a new vserver:

ULIMIT="-H -u 1000"

       Default vserver ulimit
       Normally  ulimit  settings  only  affects  users independantly. So
       limiting   a   vserver   this   way,  limit  each  user  processes
       independantly  in  the vserver. Using special flags in the S_FLAGS
       option,  you can make those ulimit settings global to the vserver.
       The  example  above used with the nproc parameter make the maximum
       number  of  process  global.  In  this  case,  a  maximum  of 1000
       processes is available to all users in the vserver.

   NEW

Entering the virtual server

   It  is possible to enter a virtual server context from the main server
   just  by executing /usr/sbin/vserver XX enter (where XX is the virtual
   server name).

   This   creates   a   shell.   From  there  you  can  execute  anything
   administrative you normally do on a Linux server.

   NEW

Configuring the services

   The  virtual  server  can  run  pretty  much any services. Many pseudo
   services,  such  as  network  configuration are useless (the server is
   already configured). After building the environment, enter it (without
   starting  the  virtual  server)  using the vserver name enter command.
   Then  using a tool like Linuxconf (control/control service activity) ,
   or ntsysv, browse all service and keep only the needed ones.

   So  after building the server, you enter it and you select the service
   you  need  in that server. Many services such as network, and apmd are
   either  useless  or  won't run at all in the virtual server. They will
   fail to start completely. NEW

Starting/Stopping the virtual server

   Virtual  server  with  ONBOOT=yes will be started and stopped like any
   other  services  of  the  main  server.  But  you can stop and start a
   virtual  server  at  any  time.  Starting  a  server  means  that  all
   configured  service  will  be  started.  Stopping  it  means  that all
   configured  services  will  be  stopped and then all remaining process
   will be killed.

   Oddly,  starting  a  virtual  server  does  not mean much. There is no
   overhead.  No  monitoring  process  or  proxy  or emulator. Starting a
   virtual server with 4 services is the same as running those 4 services
   in  the  main  server, at least performance wise (the service inside a
   virtual server are locked inside the security context).

   The following commands may be used to control a virtual server:

     * /usr/sbin/vserver server start
     * /usr/sbin/vserver server stop
     * /usr/sbin/vserver server restart
     * /usr/sbin/vserver server running
     * /usr/sbin/vserver server enter
     * /usr/sbin/vserver server exec some commands ...
     * /usr/sbin/vserver server suexec user some commands ...
     * /usr/sbin/vserver        server        service        service-name
       start/stop/restart/status
     * /usr/sbin/vserver server status

   The  running  command prints if there are any processes running in the
   virtual server context.

   Please note

   The  processes running in a virtual server are invisible from the main
   server.  The  opposite  is  true. This is very important. Managing the
   main  server  must  not cause problems to the various virtual servers.
   For  example, doing killall httpd will kill all the httpd processes in
   the current context ( the main server or a virtual one).

   NEW

Starting/Stopping all the virtual servers

   The sysv script /etc/rc.d/init.d/vserver is used to start and stop the
   virtual  server  at boot and shutdown time. It may be used at any time
   to operate all virtual servers. The following commands are supported:

     * /etc/rc.d/init.d/vservers start
     * /etc/rc.d/init.d/vservers stop
     * /etc/rc.d/init.d/vservers restart
     * /etc/rc.d/init.d/vservers status

   The status command reports the running status of every virtual server.
   NEW

Restarting a virtual server from inside

   A  virtual  server  administrator is not allowed to reboot the machine
   (the  kernel).  But  it  is  useful to restart his virtual server from
   scratch.  This  allow  him  to make sure all the services are properly
   configured to start at boot time.

   The  /sbin/vreboot  and  /sbin/vhalt  utilities  are installed in each
   virtual server so they can request a restart or stop.

   The rebootmgr service must be enabled in the main server.

   NEW

Executing tasks at vserver start/stop time

   You can setup a script called /etc/vservers/XX.sh where XX is the name
   of the virtual server. This script will be called four time:

     * Before starting the vserver
     * After starting it.
     * Before stopping it.
     * After stopping it.

   You generally perform tasks such as mounting file system (mapping some
   directory in the vserver root using "mount --bind").

   Here  is  an  example where you map the /home directory as the vserver
   /home directory.

#!/bin/sh
case $1 in
pre-start)
        mount --bind /home /vservers/$2/home
        ;;
post-start)
        ;;
pre-stop)
        ;;
post-stop)
        umount /vservers/$2/home
        ;;
esac

   /etc/vservers/XX.sh
   NEW

Issues

   There are some common problem you may encounter. Here they are.

     * The  main  server  is not tied (by default) to any ipv4root. So if
       the main server has already some service running they are probably
       binding  some  UDP  or TCP ports using the address 0.0.0.0. Once a
       process  has  bound  a  service  with the address 0.0.0.0 (see the
       LISTEN  lines  when  executing the "netstat -a" command), no other
       process can bind the same port, even with a specific address.
       The solution is to start the services of the main server using the
       chbind utility to trap them in one ipv4root. For example

/sbin/chbind --ip eth0 /etc/rc.d/init.d/httpd start

       Assigning on IP to a service
       will limit Apache to the IP address of the eth0 interface. without
       configuration  changes (in httpd.conf). It is probably a good idea
       to  start  the  following  services  in  the main server this way,
       because they will be run by virtual servers as well.

     * httpd
     * sshd
     * xinetd

   To  ease  this,  the  vserver package supplies the following services:
   v_httpd,   v_sshd,  v_smb  and  v_xinetd.  Disable  the  corresponding
   services  and  enable the v_ services and you will lock those services
   on a single IP.

     Cleanup rc.local. This is probably not doing anything useful.

   NEW

How real is it ?

   The  project  is  new.  So  far,  experiments  have  shown very little
   restrictions.  Service  works  the  same in a virtual server. Further,
   performance  is  the  same.  And  there  is  a high level of isolation
   between the various virtual servers and the main server. NEW

Features

   There  are various tricks one can use to make the virtual servers more
   secure.

     * Putting  a  fire-wall  on the box and limiting access to a virtual
       services from another one.
     * Using  port  redirection  to allow one virtual server to logically
       bind several IPs. One virtual server could run several web virtual
       host this way.

   NEW

Future directions

   NEW

User controlled security box

   By  combining  the  capabilities,  the s_context, the ipv4root and the
   AclFS  (component of the [51]virtualfs package), we can produce a user
   level  tool  allowing controlled access to the user own resources. For
   example  the  user  may download any program he wants and execute them
   under  control.  Whenever  the  program  tries to access something not
   specified by the user, a popup is presented and the user may choose to
   terminate the program or allow the access.

   NEW

Kernel enhancements

   We  expect  to  see  some wider usage of the virtual servers. As usage
   grow, we expect to see needs for more control. Here are some ideas.

   NEW

Per context disk quota

   If  one  installs  virtual  servers  and  grant access to less trusted
   users,  he  may  want  to  limit  the disk space used. Since a virtual
   server may create new user accounts and run processes with any user ID
   it wants, the current kernel disk quota is not powerful enough. First,
   it  can't  differentiate between user ID 100 in one virtual server and
   user ID 100 in another one.

   Further,  the  main  administrator  may  want  to  control  disk space
   allocated  to  the  virtual  server  on  a  server  per  server basis,
   unrelated to the various user ID in use in those virtual servers.

   The  kernel  has  already  user  and group disk quota. Adding security
   context disk quota should be easily done.

   To differentiate between user IDs in virtual servers, the kernel could
   coin  together the security context and the user id to create a unique
   ID.  The  kernel  2.4  now  supports 32 user ID, so combining security
   context and user ID in a single 32 bits number should be acceptable.

   NEW

Global limits

   The  kernel  has  supports  for  user  limit  (memory,  processes file
   handles). With virtual server, we may want to limit the resources used
   by  all processes in the virtual server. The security context would be
   used  as  the  key here. The following resources could be limited on a
   security context basis (as opposed to user or process basis)

     * Memory used
     * Processes number
       (Done:  This  is  now  supported with the nproc flag in the kernel
       2.4.16ctx-4. By default a new vserver is limited to 1000 processes
       maximum, configurable).
     * File handles

   NEW

Scheduler

   The  scheduler may become security context aware. It could potentially
   use  this  to  provide  some  fairness  and  control priority based on
   context.  Currently  the  scheduler  is  process oriented and does not
   group  process  together  to  qualify their priorities. For example, a
   user  running  10  compilations  will  get  more CPU than another user
   running a single compilation.

   Currently,  it  is possible to raise the nice (lower priority) for all
   processes  in  a  virtual  server.  This can't be reversed, so you are
   setting  an  upper  limit on priority (Just set the S_NICE variable in
   the  vserver configuration file). Note that a virtual server may still
   start  many low priority processes and this can grab significant share
   of  the  CPU.  A global per security context might be needed to really
   provide more control and fairness between the various virtual servers.

   Done:  The  sched security context flag group all process in a vserver
   so their priority is kind of unified. If you have 50 processes running
   full  speed in one vserver, they will take as much CPU resource than a
   single process in the root server. A vserver can't starve the other...
   NEW

Security issues

   The  current kernel + patch provides a fair level of isolation between
   the  virtual  servers.  User  root can't take over the system: He sees
   only  his  processes,  has  only access to his area of the file system
   (chroot)  and  can't  reconfigure  the  kernel.  Yet  there  are  some
   potential  problems. They are fixable. As usage grows, we will know if
   they are real problems. Comments are welcome:

   NEW

/dev/random

   Writing to /dev/random is not limited by any capability. Any root user
   (virtual included) is allowed to write there. Is this a problem ?

   (kernel expert think it is ok) NEW

/dev/pts

   /dev/pts  is  a  virtual  file-system  used to allocate pseudo-tty. It
   presents  all  the  pseudo-tty  in  use  on  the server (including all
   virtual  server).  User  root  is  allowed  to  read  and write to any
   pseudo-tty, potentially causing problems on other vservers.

   Starting  with  the ctx-6 patch, /dev/pts is virtualised. Although the
   file  numbers are allocated from a single pool, a vserver only see the
   pseudo-tty it owns. NEW

Network devices

   Anyone  can list the network devices configurations. This may inform a
   virtual  user  that another vserver is on the same physical server. By
   using  as  much resources as possible in his own vservers, a malicious
   user  could  slow down the other server. Modification to the scheduler
   explained above could stop this.

   Starting  with  the  ctx-6  patch,  a  vserver  only  see  the  device
   corresponding to its IP number. NEW

Alternative technologies

   Using  virtual  servers may be a cost effective alternative to several
   independent  real  servers. You get the administrative independence of
   independent servers, but share some costs including operation costs.

   Other  technologies  exist  offering  some of the advantages talked in
   this  document  as  well  as  other. Two technologies are available on
   various hardware platform: Virtual machines and partitioning, NEW

Virtual machines

   This  has  been available for mainframes for a while now. You can boot
   several  different  OS at once on the same server. This is mainly used
   to  isolate environments. For example, you can install the new version
   of  an  OS  on  the  same server, even while the server is running the
   current version. This allows you to test and do a roll-out gracefully.

   The advantages of virtual machines are:

     * Total  flexibility.  You  can  run many different OS and different
       version of the same OS, all at once.
     * Robustness.  You  have  total  isolation. One OS may crash without
       affecting the other.
     * Resource management. You can effectively limit the resources taken
       by one OS.
     * Hardware  Independence.  The  client  OS  is  using  virtual disks
       provided by the host OS.

   This  technology  is  not  directly  available  on  PCs. The Intel x86
   architecture  does  not  support visualization natively. Some products
   nevertheless  have appeared and provide this. You can run Linux inside
   Linux,  or this other OS (Which BTW has a logo showing a window flying
   in pieces, which quite frankly tells everything about it).

   The  solutions  available  on  PCs carry most of the advantages of the
   virtual machines found on mainframe, except for performance. You can't
   run  that  many virtual Linux's using this technology and expect it to
   fly.  One  example  of  this  technology is [52]vmware, which is quite
   useful, especially if you must run this other OS... vmware may be used
   to  run Linux inside Linux, even test Linux installation while running
   linux... NEW

Partitioning

   Partitioning  (domains  ?)  is a way to split the resources of a large
   server  so  you  end up with independent servers. For example, you can
   take  a  20  CPUs server and create 3 servers, two with 4 CPUs and one
   with  12.  You  can  very easily re-assign CPUs to servers in case you
   need more for a given tasks.

   This technology provides full Independence, but much less flexibility.
   If  your  12  CPUs  server  is  not working much, the 4 CPUs one can't
   borrow some CPUs for 5 minutes. NEW

Limitation of those technologies

   Oddly,  one  disadvantage  of  those  technologies is a side effect of
   their  major  advantage:  Total  Independence.  Each virtual server is
   running  its  own  kernel.  Cool.  This makes the following tasks more
   difficult or impossible:

     * Sharing  administrative  tasks such as backup. The virtual servers
       are using volumes in the host server. The host server can't handle
       the  files  in those volumes directly without interfering with the
       client  OS. It has to use some services of the client OS to access
       the file.
       The  vserver  solution  does  not  have  this limitation since the
       virtual  servers  are  living in the same file-system, sharing the
       same kernel.
     * Task  monitoring.  The  virtual  servers  run their own kernel. As
       such,  the  host OS can't spy on the tasks and check for intrusion
       for example.
     * Disk  space.  Virtual  servers  are  using  either volumes or full
       devices  in  the  host  server. This space is pre-allocated to the
       maximum needed by the server. You end up with a lot of wasted disk
       space. Imagine running 100 virtual servers this way and allocating
       say  10 gigs to each. You get the picture. The vserver solution is
       sharing  a  common file-system so the free disk space is available
       to all.
       Further,  if  you  are  running the same Linux distribution in the
       virtual  servers, you can unify the disk space using hard link and
       immutable  attributes.  The /usr/lib/vserver/vunify was created to
       test  that.  Using information found in the rpm package the script
       links the files, except configuration ones.
       Testing   vunify   on  a  vserver  installed  with  a  RedHat  6.2
       distribution,  unifying  the  packages  glibc, binutils, perl, and
       bash  saved  60  megs. Quite a few packages are not changing often
       and could be unified.
       Vservers  do  not  need kernel packages and hardware configuration
       tools. This also contribute to save disk space.
     * File system sharing
       A  little  the  same  as above. You can't share file system easily
       between  vservers  unless you use network services (often slower).
       Using  "mount  --bind",  it is very easy to "map" any directory of
       the  root  server  in several vservers, providing raw speed access
       (and even sharing the disk cache).

   NEW

Conclusion

   Virtual  servers  are  interesting  because  they can provide a higher
   level  of security while potentially reducing the administration task.
   Common  operation  such  as  backup,  are  shared between all servers.
   Services such as monitoring may be configured once.

   A  Linux  server  can  run  many services at once with a high level of
   reliability.  As  servers  are  evolving,  more  and more services are
   added,  often  unrelated. Unfortunately there are few details here and
   there,  making the server more complex than it is in reality. When one
   wants  to  move  one  service to another server, it is always a little
   pain:  Some  user  accounts  have  to  be moved and some configuration
   files. A lot of hand tweaking.

   By  installing  services  in separate virtual servers, it becomes much
   easier  to move services around (just by moving a directory although a
   big one).

   Virtual  servers  may  become  a preferred way to install common Linux
   servers. NEW

Download

   The ftp site for this project is
   [53]ftp://ftp.solucorp.qc.ca/pub/vserver  .  You  will  find there the
   following components.

     * [54]kernel-2.4.20ctx-17.tar.gz
       [55]kernel-2.4.20ctxsmp-17.tar.gz
       A  pre-compiled  kernel  for  Pentium class machine and up. An SMP
       kernel is also supplied.
     * [56]vserver-0.22-1.src.rpm
       The source RPM for the vserver utilities
     * [57]vserver-0.22-1.i386.rpm
       A  compiled  rpm  for RedHat 7.x and up. Should work on any recent
       distribution  (glibc  2.2).  You  need  a  recent  distribution to
       operate a kernel 2.4 anyway.
     * [58]vserver-admin-0.22-1.i386.rpm
       Contains  the  command /usr/sbin/newvserver. It is a GUI to create
       vservers.   It  requires  the  linuxconf-utils  and  linuxconf-lib
       packages.  You can get them from [59]here. linuxconf itself is not
       needed though.
     * [60]vserver-0.22.src.tar.gz
       The vserver utilities source
     * [61]patch-2.4.20ctx-17.gz
       The patch against Linux 2.4.20
     * [62]patches
       The various relative patches (ctxN-ctxN+1)

   NEW

References

   This project is maintained by Jacques Gelinas [63]jack@solucorp.qc.ca

   The vserver package is licensed under the GNU PUBLIC LICENSE.

   A FAQ can be found at
   [64]http://www.solucorp.qc.ca/howto.hc?projet=vserver 

   A  mailing list has been created to exchange about this project. It is
   [65]vserver@solucorp.qc.ca .You can subscribe [66]here 

   The mailing list is archived [67]here.

   The change logs for the vserver package are [68]here .

   The    official    copy    of    this    document    is    found    at
   [69]http://www.solucorp.qc.ca/miscprj/s_context.hc 

   This document was produced using the [70]TLMP documentation system 

   [71]Top
   [72]Back to project page
   [73]About tlmpdoc and cookies
   Document maintained by Jacques Gélinas ([74]jack@solucorp.qc.ca)
   Last update: Wed Apr 16 11:22:22 2003

Références

   1. http://remtk/solucor/miscprj/s_context.hc?s1=1&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
   2. http://remtk/solucor/miscprj/s_context.hc?s1=1&s2=1&s3=0&s4=0&full=0&prjstate=1&nodoc=0
   3. http://remtk/solucor/miscprj/s_context.hc?s1=2&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
   4. http://remtk/solucor/miscprj/s_context.hc?s1=2&s2=1&s3=0&s4=0&full=0&prjstate=1&nodoc=0
   5. http://remtk/solucor/miscprj/s_context.hc?s1=2&s2=2&s3=0&s4=0&full=0&prjstate=1&nodoc=0
   6. http://remtk/solucor/miscprj/s_context.hc?s1=2&s2=3&s3=0&s4=0&full=0&prjstate=1&nodoc=0
   7. http://remtk/solucor/miscprj/s_context.hc?s1=2&s2=4&s3=0&s4=0&full=0&prjstate=1&nodoc=0
   8. http://remtk/solucor/miscprj/s_context.hc?s1=2&s2=5&s3=0&s4=0&full=0&prjstate=1&nodoc=0
   9. http://remtk/solucor/miscprj/s_context.hc?s1=2&s2=6&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  10. http://remtk/solucor/miscprj/s_context.hc?s1=2&s2=6&s3=1&s4=0&full=0&prjstate=1&nodoc=0
  11. http://remtk/solucor/miscprj/s_context.hc?s1=2&s2=6&s3=2&s4=0&full=0&prjstate=1&nodoc=0
  12. http://remtk/solucor/miscprj/s_context.hc?s1=2&s2=6&s3=3&s4=0&full=0&prjstate=1&nodoc=0
  13. http://remtk/solucor/miscprj/s_context.hc?s1=2&s2=6&s3=4&s4=0&full=0&prjstate=1&nodoc=0
  14. http://remtk/solucor/miscprj/s_context.hc?s1=2&s2=7&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  15. http://remtk/solucor/miscprj/s_context.hc?s1=3&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  16. http://remtk/solucor/miscprj/s_context.hc?s1=3&s2=1&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  17. http://remtk/solucor/miscprj/s_context.hc?s1=3&s2=2&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  18. http://remtk/solucor/miscprj/s_context.hc?s1=3&s2=3&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  19. http://remtk/solucor/miscprj/s_context.hc?s1=3&s2=4&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  20. http://remtk/solucor/miscprj/s_context.hc?s1=4&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  21. http://remtk/solucor/miscprj/s_context.hc?s1=4&s2=1&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  22. http://remtk/solucor/miscprj/s_context.hc?s1=4&s2=2&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  23. http://remtk/solucor/miscprj/s_context.hc?s1=4&s2=3&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  24. http://remtk/solucor/miscprj/s_context.hc?s1=4&s2=4&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  25. http://remtk/solucor/miscprj/s_context.hc?s1=4&s2=5&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  26. http://remtk/solucor/miscprj/s_context.hc?s1=4&s2=6&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  27. http://remtk/solucor/miscprj/s_context.hc?s1=4&s2=7&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  28. http://remtk/solucor/miscprj/s_context.hc?s1=4&s2=8&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  29. http://remtk/solucor/miscprj/s_context.hc?s1=4&s2=9&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  30. http://remtk/solucor/miscprj/s_context.hc?s1=4&s2=10&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  31. http://remtk/solucor/miscprj/s_context.hc?s1=4&s2=11&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  32. http://remtk/solucor/miscprj/s_context.hc?s1=5&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  33. http://remtk/solucor/miscprj/s_context.hc?s1=6&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  34. http://remtk/solucor/miscprj/s_context.hc?s1=6&s2=1&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  35. http://remtk/solucor/miscprj/s_context.hc?s1=6&s2=2&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  36. http://remtk/solucor/miscprj/s_context.hc?s1=6&s2=2&s3=1&s4=0&full=0&prjstate=1&nodoc=0
  37. http://remtk/solucor/miscprj/s_context.hc?s1=6&s2=2&s3=2&s4=0&full=0&prjstate=1&nodoc=0
  38. http://remtk/solucor/miscprj/s_context.hc?s1=6&s2=2&s3=3&s4=0&full=0&prjstate=1&nodoc=0
  39. http://remtk/solucor/miscprj/s_context.hc?s1=6&s2=2&s3=4&s4=0&full=0&prjstate=1&nodoc=0
  40. http://remtk/solucor/miscprj/s_context.hc?s1=6&s2=2&s3=4&s4=1&full=0&prjstate=1&nodoc=0
  41. http://remtk/solucor/miscprj/s_context.hc?s1=6&s2=2&s3=4&s4=2&full=0&prjstate=1&nodoc=0
  42. http://remtk/solucor/miscprj/s_context.hc?s1=6&s2=2&s3=4&s4=3&full=0&prjstate=1&nodoc=0
  43. http://remtk/solucor/miscprj/s_context.hc?s1=7&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  44. http://remtk/solucor/miscprj/s_context.hc?s1=7&s2=1&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  45. http://remtk/solucor/miscprj/s_context.hc?s1=7&s2=2&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  46. http://remtk/solucor/miscprj/s_context.hc?s1=7&s2=3&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  47. http://remtk/solucor/miscprj/s_context.hc?s1=8&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  48. http://remtk/solucor/miscprj/s_context.hc?s1=9&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  49. http://remtk/solucor/miscprj/s_context.hc?s1=10&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  50. ftp://ftp.solucorp.qc.ca/pub/vserver
  51. http://www.solucorp.qc.ca/virtualfs
  52. http://www.vmware.com/
  53. ftp://ftp.solucorp.qc.ca/pub/vserver
  54. ftp://ftp.solucorp.qc.ca/pub/vserver/kernel-2.4.20ctx-17.tar.gz
  55. ftp://ftp.solucorp.qc.ca/pub/vserver/kernel-2.4.20ctxsmp-17.tar.gz
  56. ftp://ftp.solucorp.qc.ca/pub/vserver/vserver-0.22-1.src.rpm
  57. ftp://ftp.solucorp.qc.ca/pub/vserver/vserver-0.22-1.i386.rpm
  58. ftp://ftp.solucorp.qc.ca/pub/vserver/vserver-admin-0.22-1.i386.rpm
  59. http://www.solucorp.qc.ca/linuxconf/download.hc
  60. ftp://ftp.solucorp.qc.ca/pub/vserver/vserver-0.22.src.tar.gz
  61. ftp://ftp.solucorp.qc.ca/pub/vserver/patch-2.4.20ctx-17.gz
  62. ftp://ftp.solucorp.qc.ca/pub/vserver/patches
  63. mailto:jack@solucorp.qc.ca
  64. http://www.solucorp.qc.ca/howto.hc?projet=vserver
  65. mailto:vserver@solucorp.qc.ca
  66. http://www.solucorp.qc.ca/mlist/index.hc?list=vserver
  67. http://www.paul.sladen.org/vserver/archives/
  68. http://www.solucorp.qc.ca/changes.hc?projet=vserver
  69. http://www.solucorp.qc.ca/miscprj/s_context.hc
  70. http://www.solucorp.qc.ca/tlmp
  71. http://remtk/solucor/miscprj/s_context.hc?s1=0&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
  72. http://remtk/solucor/miscprj/s_context.hc
  73. http://www.solucorp.qc.ca/tlmp/tlmpdoc.hc
  74. mailto:jack@solucorp.qc.ca