http://freshmeat.net/projects/zebedee/
This is kind of old…
Archives
All posts for the month March, 2011
(Taken from blog.makezine.com)
Linux Tip: super-fast network file copy
If you’ve ever had to move a huge directory containing many files from one server to another, you may have encountered a situation where the copy rate was significantly less that what you’d expect your network could support. Rsync does a fantastic job of quickly syncing two relatively similar directory structures, but the initial clone can take quite a while, especially as the file count increases.
The problem is that there is a certain amount of per-file overhead when using scp or rsync to copy files from one machine to the other. This is not a problem under most circumstances, but if you are attempting to duplicate tens of thousands of files (think, server or database backup), this per-file overhead can really add up. The solution is to copy the files over in a single stream, which normally means tarring them up on one server, copying the tarball, then untarring on the destination. Unless you are under 50% disk utilization on the source server, this could cause you to run out of space.
Brett Jones has an alternative solution, which uses the handy netcat utility:
After clearing up 10 GBs of log files, we were left with hundreds of thousands of small files that were going to slow us down. We couldn’t tarball the file because of a lack of space on the source server. I started searching around and found this nifty tip that takes our encryption and streams all the files as one large file:
This requires netcat on both servers.
Destination box: nc -l -p 2342 | tar -C /target/dir -xzf –
Source box: tar -cz /source/dir | nc Target_Box 2342
This causes the source machine to tar the files up and send them over the netcat pipe, where they are extracted on the destination machine, all with no per-file negotiation or unnecessary disk space used. It’s also faster than the usual scp or rsync over scp because there is no encryption overhead. If you are on a local protected network, this will perform much better, even for large single-file copies.
If you are on an unprotected network, however, you may still want your data encrypted in transit. You can perform about the same task over ssh:
Run this on the destination machine:
cd /path/to/extract/to/
ssh user@source.server ‘tar -cz -C /source/path/ *’ | tar -zxv
This command will issue the tar command across the network on the source machine, causing tar’s stdout to be sent back over the network. This is then piped to stdin on the destination machine and the files magically appear in the directory you are currently in.
The ssh route is a little slower than using netcat, due to the encryption overhead, but it’s still way faster than scping the files individually. It also has the added advantage of potentially being compatible with Windows servers, provided you have a few of the unix tools like ssh and tar installed on your Windows server (using the cygwin linked binaries that are available).
Fast File Copy – Linux!
Borrowed from UnixCraft..
#!/bin/bash
# A sample shell script to print domain ip address hosting information such as
# Location of server, city, ip address owner, country and network range.
# This is useful to track spammers or research purpose.
# ————————————————————————-
# Copyright (c) 2006 nixCraft project <http://cyberciti.biz/fb/>
# This script is licensed under GNU GPL version 2.0 or above
# ————————————————————————-
# This script is part of nixCraft shell script collection (NSSC)
# Visit http://bash.cyberciti.biz/ for more information.
# ————————————————————————-
# Last updated on Mar/05/2010
# ————————————————————————-
# Get all domains
_dom=$@
# Die if no domains are given
[ $# -eq 0 ] && { echo “Usage: $0 domain1.com domain2.com …”; exit 1; }
for d in $_dom
do
_ip=$(host $d | grep ‘has add’ | head -1 | awk ‘{ print $4}’)
[ “$_ip” == “” ] && { echo “Error: $d is not valid domain or dns error.”; continue; }
echo “Getting information for domain: $d [ $_ip ]…”
whois “$_ip” | egrep -w ‘OrgName:|City:|Country:|OriginAS:|NetRange:’
echo “”
done
Borrowed from Unixcraft…
#!/bin/bash
# Script Name: mbrback   http://igurublog.wordpress.com/downloads/script-mbrback/
# Requires: util-linux
# License: GNU GENERAL PUBLIC LICENSE Version 3 http://www.gnu.org/licenses/gpl-3.0.txt
# do not change these variables!
argsneeded=1
restoretype=””
back=””
devname=””
help ()
{
echo ‘mbrback version 1.0.0’
echo ‘Creates MBR and partition table backups of DEVICE named:’
echo ‘Â Â Â HOST-DEVICE-MBR-back’
echo ‘Â Â Â HOST-DEVICE-partition-back.sf’
echo ‘Restores MBR and partition table from specified backup file’
echo ‘Usage: sudo mbrback DEVICE [BACKUPFOLDER]’
echo ‘Â Â Â Â Â Â (creates backup files of DEVICE)’
echo ‘Usage: sudo mbrback –restoreboot DEVICE [BACKUPFILE]’
echo ‘Â Â Â Â Â Â (restores MBR boot code only)’
echo ‘Usage: sudo mbrback –restorefullmbr DEVICE [BACKUPFILE]’
echo ‘Â Â Â Â Â Â (restores entire MBR)’
echo ‘Usage: sudo mbrback –restorepart DEVICE [BACKUPFILE.sf]’
echo ‘Â Â Â Â Â Â (restores partition table)’
echo ‘Example: sudo mbrback sda’
echo ‘Â Â Â Â Â Â Â Â (creates MBR and partition table backups of’
echo ‘Â Â Â Â Â Â Â Â Â /dev/sda in current folder)’
echo ‘Example: sudo mbrback /dev/sda’
echo ‘Â Â Â Â Â Â Â Â (creates MBR and partition table backups of’
echo ‘Â Â Â Â Â Â Â Â Â /dev/sda in current folder)’
echo ‘Example: sudo mbrback sda /mybackups’
echo ‘Â Â Â Â Â Â Â Â (creates MBR and partition table backups of’
echo ‘Â Â Â Â Â Â Â Â Â /dev/sda in /mybackups)’
echo ‘Example: sudo mbrback –restoreboot sda /mybackups/sys-sda-MBR-back’
echo ‘Â Â Â Â Â Â Â Â (restores MBR boot code of /dev/sda using’
echo ‘Â Â Â Â Â Â Â Â Â /mybackups/sys-sda-MBR-back)’
echo ‘Example: sudo mbrback –restorepart sda /mybackups/sys-sda-partition-back.sf’
echo ‘Â Â Â Â Â Â Â Â (restores partition table of /dev/sda using sfdisk file ‘
echo ‘Â Â Â Â Â Â Â Â Â /mybackups/sys-sda-partition-back.sf)’
echo
echo “When restoring, mbrback will always tell you what it’s going to do”
echo “and allow you to abort before it writes to disk.”
echo
echo “Instructions and updates:”
echo “http://igurublog.wordpress.com/downloads/script-mbrback/”
echo
}
index=0
while [ “$1” != “” ];
do
if [ “${1:0:1}” = “-” ]; then
case “$1” in
–help | -help )
help
exit
;;
–restoreboot )
if [ “$restoretype” = “” ]; then
restoretype=”boot”
else
echo ‘mbrback: can only use one restore option’
exit 1
fi
;;
–restorefullmbr )
if [ “$restoretype” = “” ]; then
restoretype=”fullmbr”
else
echo ‘mbrback: can only use one restore option’
exit 1
fi
;;
–restorepart )
if [ “$restoretype” = “” ]; then
restoretype=”part”
else
echo ‘mbrback: can only use one restore option’
echo
help
exit 1
fi
;;
* )
echo “mbrback: Unknown option $1”
echo
help
exit 1
;;
esac
else
let “index+=1”
case $index in
1 )
devname=basename "$1"
if [ ! -b “/dev/$devname” ]; then
echo “mbrback: /dev/$devname is not a valid device”
exit 1
fi
;;
2 )
back=”$1″
;;
* )
echo “mbrback: Too many arguments”
exit 1
;;
esac
fi
shift
done
if (( index < $argsneeded )) || [ “$devname” = “” ]; then
echo “mbrback: missing arguments”
echo
help
exit 1
fi
if [ whoami != “root” ]; then
echo ‘mbrback: must be run with sudo’
exit 1
fi
sysname=$HOSTNAME
if [ “$restoretype” = “” ]; then
# create MBR and table backups
if [ “$back” = “” ]; then
back=pwd
else
if [ ! -d “$back” ]; then
echo “mbrback: $back is not a valid backup folder”
exit 1
fi
fi
dd if=/dev/$devname of=”$back/$sysname-$devname-MBR-back” bs=512 count=1
sfdisk -d /dev/$devname > “$back/$sysname-$devname-partition-back.sf”
else
# restore
if [ “$back” = “” ]; then
echo “mbrback: you must specify a backup file”
exit 1
elif [ ! -f “$back” ]; then
echo “mbrback: file not found – $back”
exit 1
fi
if [ “$restoretype” = “boot” ] || [ “$restoretype” = “fullmbr” ]; then
sfhead=head --bytes=21 "$back"
if [ “$sfhead” = “# partition table of ” ]; then
echo “mbrback: $back is not an MBR backup file”
exit 1
fi
if [ “$(stat -c%s “$back”)” != “512” ]; then
echo “mbrback: $back is wrong size for an MBR backup file”
exit 1
fi
fi
if [ “$restoretype” = “part” ]; then
sfhead=head --bytes=21 "$back"
if [ “$sfhead” != “# partition table of ” ]; then
echo “mbrback: $back not a valid sfdisk backup file”
exit 1
fi
echo
echo “You are about to overwrite your /dev/$devname partition table with”
echo “the contents of $back”
echo
echo “WARNING!!! Unless the partition table has been damaged or you”
echo “Â Â Â Â Â Â Â Â Â Â have accidentally deleted a partition, you should abort.”
echo
echo “WARNING!!! Restoring the partition table from an out-of-date backup”
echo “Â Â Â Â Â Â Â Â Â Â may render ALL the data on your drive unreadable.”
echo
echo “WARNING!!! Do not proceed if /dev/$devname is mounted.”
echo
elif [ “$restoretype” = “boot” ]; then
echo
echo “You are about to overwrite your /dev/$devname MBR boot code with”
echo “the contents of $back”
echo
echo “WARNING: Restoring your MBR boot code from an out-of-date MBR backup”
echo “Â Â Â Â Â Â Â Â file may render your computer unbootable.”
elif [ “$restoretype” = “fullmbr” ]; then
echo
echo “You are about to overwrite your ENTIRE /dev/$devname MBR with”
echo “the contents of $back”
echo
echo “WARNING!!! The full MBR contains both boot code and the drive’s”
echo “          partition table. Unless the partition table has been”
echo “Â Â Â Â Â Â Â Â Â Â damaged or you have accidentally deleted a partition”
echo “Â Â Â Â Â Â Â Â Â Â you should abort and restore boot code only with”
echo “Â Â Â Â Â Â Â Â Â Â –restoreboot instead.”
echo
echo “WARNING!!! Restoring your full MBR from an out-of-date MBR backup may”
echo “Â Â Â Â Â Â Â Â Â Â render your computer unbootable and ALL the data on your”
echo “Â Â Â Â Â Â Â Â Â Â drive unreadable.”
echo
echo “WARNING!!! Do not proceed if /dev/$devname is mounted.”
fi
echo
echo “Do you want to proceed? (you must type yes to proceed)”
read s1
if [ “$s1” != “yes” ]; then
echo “mbrback: no changes made – aborted at user request”
exit 2
fi
if [ “$restoretype” = “part” ]; then
sfdisk /dev/$devname < “$back”
elif [ “$restoretype” = “boot” ]; then
dd if=”$back” of=/dev/$devname bs=448 count=1
elif [ “$restoretype” = “fullmbr” ]; then
dd if=”$back” of=/dev/$devname bs=512 count=1
fi
echo “/dev/$devname was updated”
fi
exit 0
Start with a freshly installed copy of Windows 2008R2
Open up Server manager
Select Roles –> Add Role Services –> Select Identity Management for UNIX
this should select all 3 boxes..ie make sure all 4 boxes are select..hit next, then finish.
You must reboot. Reboot.
After the reboot and after logging back in…just sit at the desktop the installation will continue. It should pop open server manager again, and finish the installation.
syswtestdss1sd.2008R2.local
2008R2.local
10.139.208.85
installing the DNS for domain 2008R2.local. Since this domain was just a test, I added a local DNS server to the Windows host. Configurate the domain in there..and had the UNIX boxes also use that for now. Make sure PTR records are setup correctly.
To install Identity Management for UNIX components
- Open Server Manager. To open Server Manager, click Start, point to Administrative Tools, and then click Server Manager.
- In the tree pane, expand Roles.
- On the role home page for AD DS, in the Roles section, in the list of common tasks, click Add Role Services.
- On the Select Role Services page of the Add Role Services Wizard, select the Identity Management for UNIX role services that you want to install, and then click Next.
- If the wizard prompts you to install any other role services that are required by Identity Management for UNIX components, click Yes.
- After verifying your selections on the Confirm Installation Selections page, click Install.The computer must be restarted after the installation of Identity Management for UNIX finishes.
Note
You must be a member of the Administrators group on the local computer to install Identity Management for UNIX components.
Server Manager is available only to members of the Administrators group on the local computer. By default, Server Manager opens when an administrator logs on to the computer. You can open Server Manager from the Start menu, the Quick Launch bar, or from Administrative Tools.
Installing Identity Management for UNIX by using Windows PowerShell
You can use the Windows PowerShell set of cmdlets for Server Manager to install Identity Management for UNIX.
To install Identity Management for UNIX by using Windows PowerShell
- Open a Windows PowerShell session with elevated user rights. To do this, click Start, click All Programs, click Accessories, click Windows PowerShell, right-click the Windows PowerShell shortcut, and then click Run as administrator.
- Load the Server Manager module into the Windows PowerShell session before working with Server Manager cmdlets. Type the following, and then press Enter.Import-Module ServermanagerNoteWindows PowerShell cmdlets are not case-sensitive.
- Do one of the following.
- To install all Identity Management for UNIX components, type the following, and then press Enter. Add-WindowsFeature ADDS-Identity-Mgmt -restart
- To install only Password Synchronization, type the following, and then press Enter. Add-WindowsFeature ADDS-Password-Sync -restart
- To install only Server for NIS, type the following, and then press Enter. Add-WindowsFeature ADDS-NIS -restart
A restart of the computer is required when you install Identity Management for UNIX. The -restart parameter restarts the computer automatically after installation is complete.
NoteAdd the -whatIf parameter to your command to instruct Server Manager to show the list of all software that is installed by default as a result of the command. Running the command with the -whatIf parameter does not result in an actual installation; the command results show only what would be installed by an actual installation.
Installing Identity Management for UNIX by using a command line
Server Manager command line tools allow you to install or remove roles, role services, or features in a Windows Command Prompt. Identity Management for UNIX can be installed by using the Server Manager line command because it is a role service of the Active Directory Domain Services role.
You can run the Server Manager line command, ServerManagerCmd.exe, from within any directory on the local computer. You must be a member of the Administrators group on the local computer to run the Server Manager command.
Important
Because of security restrictions imposed by User Account Control in Windows Server 2008 R2, you must run ServerManagerCmd.exe in a Command Prompt window opened with elevated user rights. To do this, right-click the Command Prompt executable, or the Command Prompt object on the Start menu, and then click Run as administrator.
Note
Other parameters are available for this command, such as -restart, which automatically restarts the computer after installation if it is required by the programs you have installed. It is recommended that you read about additional Server Manager command line parameters in the topic “Overview of Server Manager Commands” in the Server Manager Help.
To install Identity Management for UNIX by using a command line
- Open a Command Prompt window with elevated user rights. To do this, click Start, click All Programs, click Accessories, right-click Command Prompt, and then click Run as administrator.
- Type one of the following, and then press ENTER.
- ServerManagerCmd.exe -install ADDS-Identity-Management -restart to install all of Identity Management for UNIX
- ServerManagerCmd.exe – install ADDS-NIS -restart to install Server for NIS
- ServerManagerCmd.exe – install ADDS-Password-Sync -restart to install Password Synchronization
A restart of the computer is required when you install Identity Management for UNIX. The -restart parameter restarts the computer automatically after installation is finished.
NoteAdd the -whatIf parameter to your command to instruct Server Manager to show the list of all software that is installed by default by the command. Running the command together with the -whatIf parameter does not result in an actual installation. The command results show only what would be installed by an actual installation.
Setting up Password Synchronization for use with standalone UNIX-based hosts
Step
Reference
Read about Password Synchronization.
Overview of Password Synchronization
Log on as a member of both the Schema Administrators and Enterprise Administrators groups.
Install Password Synchronization on all Windows-based domain controllers. If the passwords of local accounts on a server are to be synchronized, install Password Synchronization on the server. If Windows domain passwords are to be synchronized, install Password Synchronization on all domain controllers.
Install Identity Management for UNIX Components
Set the password encryption key.
Setting the password encryption key
Change other settings, as needed.
Setting default synchronization; Setting computer-specific synchronization properties
Add UNIX-based computers with which passwords will be synchronized if they are not members of the Network Information Service (NIS) domain. For each computer, on the General tab of the Add Computer dialog box, clear the Synchronize password changes to this computer check box, select the Synchronize password changes from this computer check box, and then click OK. If you want to use non-default values, you can also specify values for the port number, encryption key, or both.
Adding or removing computers for synchronization
Ensure that the Password Synchronization configurations on all domain controllers in the domain are identical.
Configuring UNIX-based standalone hosts to work with Password Synchronization
Step
Reference
Install and configure the Password Synchronization single sign-on daemon (SSOD) on all UNIX-based computers with which passwords will be synchronized. Be sure to change the default encryption key in the sso.conf file to match the Password Synchronization encryption key set in previous steps before copying it to the UNIX-based computers.
Install the Password Synchronization daemon on UNIX-based computers
Specify which users have permissions to synchronize passwords.
Controlling password synchronization for user accounts
Start the Password Synchronization daemon.
Start or stop Identity Management for UNIX components
Install and configure the Password Synchronization PAM on all UNIX-based computers from which password changes are to be synchronized with Windows passwords.
Install the Password Synchronization pluggable authentication module
——————————————————————-
To add a bunch of hosts to the UNIX server manager for UNIX in MMC.
Create a text file called allhosts.txt; add all the hosts you want to add
,
,
hostname1
hostname2
hostname3
hostname4
.
.
Either copy this into a bat file or run it from command line. (carriage returns = 😉 Save it anything .bat, script.bat
for /f %%i in (%1) do (
psadmin add %%i)
To run it:Â open command prompt; cd into your working directory; script.bat allhosts.txt
——————————————————————————-
You will need to make sure you also install the NIS (Server for NIS)Role. At first I didn’t think we needed this as we were not coming from a NIS environment, but this role adds the UNIX attribute tab to the Users Properties. Without this you can’t assigned UNIX uid/gid/ home dirs, or login shell.
To use NIS: open IDMU; select NIS right click and start service. This service adds the NIS domain to the User props: UNIX attrib tab.
You will need to create a UNIX group first. Create a group like normal, right click new, group. come up with a new..I choose UNIXgroup (very creative).
Once that is created, right click on that and click the UNIX atrib tab.
NIS Domain:Â Select your domain, ours test is 2008R2.
GID I choose 5000. click ok
——————————————————-
Using Kerberos for UNIX/Linux directory and for authentication servvices.
Windows:Â 2008r2:Â AD domain controller, DNS server, AD Services IDMU(make sure to include NIS)
Other Network File and Print services. MS services for NFS
Linux:Â krb5-libs, pam_krb5, and krb5-workstation.
—————————————————————————-
Novell SLES11 updates
vi /etc/ldap.conf
host 10.139.208.85
base dc=2008r2,dc=local
uri ldap://syswtssestdss1sd.2008dr2.local/
binddn syncuser@2008r2.local
bindpw P@ssword
scope sub
timelimit 120
bind_timelimit 120
idle_timelimit 3600
nss_initgroups_ignoreusers root,ldap
referrals no
ssl no
nss_base_passwd dc=2008r2,dc=local?sub
nss_base_shadow dc=2008r2,dc=local?sub
nss_base_group dc=2008r2,dc=local?sub?&(objectCategory=group)(gidnumber=*)
nss_map_objectclass posixAccount user
nss_map_objectclass shadowAccount user
nss_map_objectclass posixGroup group
nss_map_attribute uid sAMAccountName
nss_map_attribute homeDirectory unixHomeDirectory
nss_map_attribute gecos cn
nss_map_attribute shadowLastChange pwdLastSet
nss_map_attribute uniqueMember member
pam_login_attribute sAMAccountName
pam_filter objectclass=User
pam_password ad
—————————————————————-
/etc/sso.conf
ENCRYPT_KEY=]R;nuR=|=W}L7pj>@4hE
PORT_NUMBER=6677
SYNC_HOSTS=(10.139.208.85,6677,]R;nuR=|=W}L7pj>@4hE)
USE_NIS=0
———————————————————
copy the binary to /usr/local/bin/ssod
This needs to be running, create startup script that runs this.
_______________________________________
make sure your unix host is populated in IDMU password sync.
_______________________________
For testing since we are in a different domain, make sure /etc/hosts is updated with the syswtestdss1sd.2008r2.local and it’s IP.
————————————————–
update /etc/resolv.conf …this maybe just in test since they are different domains.
