Table of Contents
Introduction
The OpenDedup Deduplication NAS Virtual Appliance is designed to is designed to provide Deduplication backed NFS Storage as a guest, within virtual environments. OpenDedup Deduplication NAS Virtual Appliance includes capabilities to create, mount,delete, and export SDFS volumes via NFS. It also includes VMWare storage api integration that allow the quick cloning of Virtual machines located on SDFS Volumes and creation of data stores.
The SDFS file system, which is used by OpenDedup Deduplication NAS Appliance, is designed to support the unique needs of virtual environments and supports enhanced functionality for VMWare, Xen, and KVM. It can deduplicate a petabyte or more of data. It supports over 5TB per gigabyte of memory at a 128k chunk size. It can perform deduplication/reduplication at a line speed of 800 MB/s or more. It supports VMware environments and can deduplicate at 4k block sizes. This is required to deduplicate Virtual Machines effectively. Deduplicated data can be stored locally, on the network across multiple nodes, or in the cloud. The filesystem can deduplicate inline or periodically based on needs. This can be changed on the fly. There is support for volume,file, or folder level snapshots and replication.
This document will guide you through setup and basic management of the Virtual Alliance. The instructions are written from the perstective of using VMware tools but are applicable to all Virtual Environments.
Virtual Appliance Build Information
The Virtual appliance is built on Ubuntu 11.04 and includes some customization to create a more streamlined NAS Appliance.
OS Build Information:
*Ubuntu 11.04 Server (Virtual Edition)
Software
* Base Ubuntu Server Install
* NFS Kernel Server
* OpenSSH Server
* SDFS Manager Version 1.1.1
* VMWare Tools
Virtual Hardware Setup
* 8 GB / (root ext4) 1.2 GB Used
* 256 GB /opt/sdfs (to store SDFS Data ext4) 0 GB Used
* 4 GB of RAM
* 4 CPUs
The guest is setup by deploying the ova file. An OVA file is a openstandard for packaging Virtual Guests for deployment. The link for this OVA File Can be found on the download page. Refer to your HyperVisor (vmware) documentation on how to deploy a ovf file.
Default Shell Login
The shell login and the web based login are the same and syncronized. The default password is set to:
Username : root
Password: admin
The password should be changed at first login. To do this type passwd after logging in. The password can also be changed within the web console.
Networking
By default networking for the virtual appliance is set to use DHCP. The network configuration can be hard set by doing the following from the command prompt. These instructions were provided by howtogeek.com.
Let’s open up the /etc/network/interfaces file. I’m going to use vi, but you can choose a different editor
sudo vi /etc/network/interfaces
For the primary interface, which is usually eth0, you will see these lines:
auto eth0
iface eth0 inet dhcp
As you can see, it’s using DHCP right now. We are going to change dhcp to static, and then there are a number of options that should be added below it. Obviously you’d customize this to your network.
auto eth0
iface eth0 inet static
address 192.168.1.100
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1
Now we’ll need to add in the DNS settings by editing the resolv.conf file:
sudo vi /etc/resolv.conf
On the line ‘name server xxx.xxx.xxx.xxx’ replace the x with the IP of your name server. (You can do ifconfig /all to find out what they are)
You need to also remove the dhcp client for this to stick (thanks to Peter for noticing). You might need to remove dhcp-client3 instead.
sudo apt-get remove dhcp-client
Now we’ll just need to restart the networking components:
sudo /etc/init.d/networking restart
Remote Shell Access
Remote access to the appliance is provided via ssh. Most volume level actions should be performed fromt he web based console.
Web Based Virtual Storage Management
The OpenDedup NAS Virtual Appliance provides a web based management interface that is used to manage the lifecycle of data stored within the appliance.The NAS appliance URL is https://<IP/HostName>. The Appliance IP address is presented on the banner before console login or can be found by logging into the console and typing ifconfig.
Capabilities of the interface include:
Volume Management:
* Creating Volumes
* Mounting Volumes
* Expanding Volumes
* Exporting Volumes via NFS
* Reporting on volume usage
File/Folder Management
* Creating folders
* Deleting files/folders
* Fast Cloning/Snapshotting Files and Folders
* Reporting on file deduplication statistics in real time
Login
To access the management console go to https://. When you first navigate to the page you will be asked for login credetials. The default username and password are the same as the shell login and, by default, set to:
Username : root
Password: admin
The password should be changed at first login. To do this navigate to Administration > Administrator. In this window you can change the root password.The password can also be changed by modifying the root password from the console.
Managing Volumes
Creating a Volume
The SDFS Volume Manager provides the capability to create multiple volumes on the appliance. The Volume manager allow you to create two types of volumes, Local Storage or Amazon Web Service (S3). The Local Storage volume will store all unique blocks of data on the local appliance disk. The Amazon Web Service option will store all unique blocks in the cloud at Amazon using the S3 service.
By default, no volumes are created. To create a volume right click on the SDFS Volumes folder and select "Create A Volume". Once selected, a Volume Creation wizard will appear and provide options for creating a volume.
Creating a Volume that Stores Data Locally
By default the SDFS Volume Manager will create a volume that stores data locally. This option will store all data in /opt/sdfs/volumes/<volume-name> . Local Storage is a good option for data that is used in production since writing and retrieving unique blocks will be done locally
The options for creating a local volume are as follows:
* Volume Name : This is the unique name of the volume. It must be unique and not include spaces
* Default Mount Point : The mount point that the volume will be mounted on the appliance. By default it will mount to /media/<volume-name>
* Mount on Startup : The volume will be mounted when the system starts if this is selected. Otherwise it can be mounted manually.
* Enable Authentication : The volume management api requires authentication. This is an option that allows you to control who can control the volume throught the command line shell sdfscli.
* Volume Password : The pasword for authentication. This defaults to admin
* Volume Block Size: This option defines the block size at which sdfs will dedup. Block size determines the dedupe rate at a trade off of memory. Smaller block sizes dedup better than larger but take up more memory. Each block stored takes up 25 bytes of memory. At 128 KB block size you can store 32x as much data as 4k block size but will not dedup well for data like VMDKs.
* Volume Size : This is the size the volume will be presented as to the NAS Appliance and the NFS Clients. This can be any size and has no determination on the amount of data that can actully be stored within the unique data store. It is safe to make this 4x the unique store size.
* Unique Store Size : This is the store size for unique blocks of data. This can not be changed after the volume is created and cannot exceed the size of the mount point /opt/sdfs or the amount of memory on the system. The memory requirements for a volume are determined as (Unique store size in bytes/block size in bytes)* 25. An addition amount will be added on to this calculation for data caching. This additional amount will be at most 1 GB.
* Backend Storage Type : The storage type for backend data. To store data locally select "Local File System".
After you enter data into the fields on this wizard page click next. It will provide verification of the volume. Once you are comfortable with the select select create volume. If mount on startup was selected, the volume will be mounted at this point. Otherwise, the volume can be mounted by right clicking on the volume and selecting "mount volume".
Creating a Cloud Based Storage Volume (Amazon S3)
To create a S3 backed volume select "Amazon Web Service (S3) from the back end storage type option within the "Create Volume Wizard". For a volume that is being stored at Amazon, a volume block size of 128k should be chosen. Additional Options for this volume typ will appear below the storage type selection. These must be filled out accurately to correctly create a volume and include:
* AWS Access Key : The Access key Amazon provides when a S3 Account is created
* AWS Secret Key : The Secret key Amazon provides when a S3 Account is created
* Unique Bucket Name : The Unique name the bucket is give.
More detail on S3 storage for SDFS can be found
here.Managing Volumes
Once a volume is created it can be manage by clicking on the selected options in the tree menu.
The Options Available are:
* Volume Information tab provides volume statistics and performance information. This page will only be available if the volume is mounted.
* Volume Configuration tab provides some basic options that can be modified. Additional configuration parameters are also shown these cannot be changed.
* Browse Volume File System provides the capabilites to manage the SDFS File System. Actions can be activated by right clicking on a file or folder that you would like to perform the action on. These actions include browse, delete, or take snapshots of a file or folder.
* NFS Exports tab is used to manage NFS Exports for a sdfs volume. Exports can be created, modified, deleted, exported, and unexported for this screen. When and NFS Export is created the target folder of the export will be created if it does not already exist. The nfs folder is created and exported by default when a volume is created and mounted. Other exports can be created manually by clicking the add button. If changes are made to the NFS volume while it is mounted it must be re-exported for the changes to apply.
Snapshots
To take a snapshot of a file or folder right click on the file or folder you would like to take a snapshot of and select take a snapshot. Once selected, a snashot wizard will appear. In the bottom text field enter the path that you would like to target the snapshot to.
VMWare Integration
The SDFS Volume manager provides capability create data stores and rapidly clone Virtual Machines through the VCenter API. To setup VMWare integration enter your VCenter credentials under Administration > VMWare Integration option in the tree.
Creating a Data Store
To export a NFS share as a data store on a VMWare server you can right click on the SDFS Volumes folder in the tree and select "Create VMWare Data Store from NFS Mount". A wizard will then guide you throught the process of creating the data store and VMWare ESX/ESXi target client. You can also create the data store using the VCenter Management Console.
Rapid Cloning VMware Guests
Once the SDFS Volume is populated with guest Virtual Machines they can be rapidly cloned. To do this, navigate to the guest "vmx" file in the "Browse Volume File System" tab for the containing volume.
Once the source vmx file a Clone VM wizard will appear and walk you throught the options for cloning the Virtual Machine. Options for creating the clone are:
* Destination VM Name : The name of the system as it will appear in the VCenter Console. The chosen must be unique and adhere to VCenter naming conventions.
* ESX Host : The target ESX/ESXi host that the new clone(s) will be presented and published on.
* Power On after Clone : If checked the clone(s) will be powered on after created
* Multiple Clones : If selected, allows multiple clones will be created and the following options will appear.
* Number of Clones : The number of clones that will be created
* Start Increment for Clone : The start number to to be appended to the clone name
Once the clones are successfully created they will show up in your vcenter console.
