Tuesday, June 18, 2013

Automate Adding Certificates to Datapower from Clearcase

I created a script that will copy the certificates that have been added in the last 2 days to a Clearcase Vob to a local folder.  Then from the local folder the script will verify that these are certificate files using openssl.  If everything verifies then it will build the connection script to Datapower and have DP pull the certificates into the shared certificate folder on the device.   JUST WHAT EVERY DATAPOWER ADMIN NEEDS!

My datapower devices were previously setup for password less login so you don't see the password as Datapower pulls the certificates on the device.  It has the server key for logging in to grab them in the user agent.
Because Datapower has to pull files I could go to Clearcase directly.  Clearcase has to be initialized for each Cron job and Datapower has no ability to access an existing session that has access to Clearcase.  All of this  was easy to work around but that explains my file copy out of clearcase with the certs.  It works nicely as a logging mechanism if something fails.  You know which certs didn't get loaded.


Use the script at your own risk.  It may contain errors so test it first in your environment.

#!/usr/bin/sh
#
# The program will copy certs from repository into the sharedcert directory
# First the new certificates have to be copied to a local directory
# then the commands are sent to Datapower to pull only the files moved to the new directory
# the files are saved into a time stamp folder to know when they were added to the device

PATH=/usr/bin:/bin:/usr/sbin:/sbin:/opt/ssh/bin
HOME=/home/k/kcastell/datapower-mgt
cc_vob="/vobs/webservices/DataPower/Certs/SI"
si_folder=$HOME/SICerts
si_files="$si_folder/*.cer"
si_loc=/datapower-mgt/SICerts
#find $cc_vob/*.cer -mtime -2 -print
dptime=`date  +%m%d%y_%H%M%S`
#email=kevin.castellow@workintel.com
email="soamgmt@workintel.com"

#Device list for datapowers
devices=/home/datapower-mgt/scripts/Properties/SIcertDeviceList.txt


#echo|find $cc_vob/*.cer -mtime -2 -print

#Generate list of new certs
certlist=`find $cc_vob/*.cer -mtime -2 -print`

#Test to see if there is any new certs found before building the command file
if [ -n "$certlist" ]; then

#We have a new cert so email the team that it will be uploaded
alert_mess="A new certificate was detected for the SI environment.  A follow up email should confirm or alert to any failures for the certificates.  Prepare for loading..."
echo $alert_mess|mailx -s "NEW CERTIFICATE BEING PROCESSED" $email

#Move the new certs to a local folder
for certificate in $certlist
do
   cp $certificate $si_folder
 
   #Verify the certificate is in the right format for the scripts to work nicely
   #Datapower can accept a certificate file in almost any format and read it correctly.
   #Openssl likes certs to be in base64 encoding so we will make the Base64 encoding the standard
   cert_data=echo | openssl x509 -noout -in ${certificate} -issuer -subject -dates -email
   cert_valid=`echo $?`
   echo $cert_valid
   echo $cert_data
   if [ "$cert_valid" -ne "$zero" ]; then
     error_mess="The certificate is not the right format for everything to validate.  Update the $certificate and save it as Base64 binary encoding.  This will make everything consistent.  This script and the certificate will not be uploaded until it is corrected. No devices were updated by this script."
     echo $error_mess|mailx -s "CERTIFICATE FORMAT INCORRECT" $email
   exit 15
   else
   succs_mess="The certificates were validated to be in the Base64 Binary format.  The upload can proceed"
   echo $succs_mess
   fi
done


#Get Datapower to pull the certs to the device

cert_commands_file="dp_cert_commands.txt"
cert_commands_execution_output="dp_cert_commands_execution_output.txt"

# remove old files to make sure we create new files, instead
# of appending to old ones
rm -f $cert_commands_file
rm -f $cert_commands_execution_output


# create list of commands, in a file. These commands
# will be sent to the DP ssh server
echo bkupadmin >> $cert_commands_file
echo work1234 >> $cert_commands_file

echo default >> $cert_commands_file
echo configure terminal >> $cert_commands_file
mkdir $si_folder/$dptime

#Tell Datapower where to find files on local unix box
for certfile in $si_files
do
#echo $certfile
        cert=`basename $certfile`
     
echo copy -f scp://kcastellow@ddweb22$si_loc/${cert} sharedcert:///${cert} >> $cert_commands_file
done


echo exit >> $cert_commands_file
echo exit >> $cert_commands_file
chmod 777 $cert_commands_file

#Repeat the cert file upload using the same command file for each device in this environment

while IFS=\| read ipaddress servername dp_domain
do
# redirect the output of the ssh session to a file, so we
ssh $ipaddress < $cert_commands_file > $cert_commands_execution_output
echo $servername
echo $dp_domain
cp $cert_commands_file $si_folder/$dptime/$servername'_'cert_commands.txt
cp $cert_commands_execution_output $si_folder/$dptime/$servername'_'comm_executions.txt

#This will verify there was no error in the upload based on logs created
#Add another word to the or condition to include it as an error
err_count=`egrep -c 'failed' $si_folder/$dptime/$servername'_'comm_executions.txt`
echo $err_count
zero=0
if [ "$err_count" -ne "$zero" ]; then
echo ERROR has occurred in uploading certificate to $ipaddress $servername at $dptime !
error_mess="An error has occurred in the uploading of a certificate to the $ipaddress device.  Check that the device is operating properly.  The script will shut down and not perform any more uploads until this is corrected to prevent further damage."
echo $error_mess|mailx -s "CERTIFICATE UPLOAD FAILURE" $email
exit 15

else
succs_mess="Certificates appear to be loaded successfully to the non-prod device at $ipaddress. Certs loaded are: $certlist"
echo $succs_mess
echo $succs_mess|mailx -s "CERTIFICATE UPLOAD SUCCESS" $email
fi

done < "$devices"

#Move the files that were uploaded
mv $si_files $si_folder/$dptime

else
echo No New Certs Found.
fi

Thursday, May 9, 2013

Datapower V6.0

Here are some links that were provided during the last IBM IMPACT Conference on the upcoming changes in Datapower.




What's New in DataPower. http://www.slideshare.net/secret/q1Pio89scmiUkq

Here is another useful presentation on Common Use Cases
http://www.slideshare.net/secret/I9YB5dIydzbIHW

Saturday, May 4, 2013

OpenSSL Printout of All Certificates and the Expiration Dates

Below is a script that will create two emails. One email will be the expired certificates along with certificate details.  The second email is a list of certificates that are about to expire.
I labeled the yellow to show the use of openssl to get the data off the certs.

The red section is interesting because I spent a decent amount of time working on the math to determine when a certificate is close to expiration.  I completed the script and then learned about the use of openssl and it will print out the days to expiration.  I will write about that in another post.  For now you can use this script as the target script from the CRONTAB.  Everything should work in the ksh.

#!/bin/ksh
# Written by Kevin Castellow to produce a report that includes information on the certificate
# within a designated time frame usually 30 days.
# This report has more functionality than Datapower cert monitoring

#Set environment for CRONTAB
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/opt/ssh/bin
HOME=/home/k/kcastell/datapower-mgt/
#files="$HOME/SICerts/*.cer"
files="/vobs/webservices/DataPower/Certs/SI/*.cer"
expired_output="$HOME/Reports/cert_expired_report.txt"
warning_output="$HOME/Reports/cert_warning_report.txt"
manifest_output="$HOME/Reports/certificate_manifest_report.txt"
#email="kevin.castellow@workintel.com"
email="soapolicymgmt@workintel.com"
rm $expired_output
rm $warning_output

#Set variables for Script
openssl_command="openssl x509 -noout -in"
cert_attributes="-issuer -subject -dates -email"
month=$(date +%m)
day=$(date +%d)
year=$(date +%Y)
x=0
y=0
z=0

print "\nToday's Date is" $month $day $year|tee -a $expired_output $warning_output $manifest_output
for certificate in $files
do
    #creates variable of end_date from each certificate in format notAfter=Jan 21 18:51:22 2028 GMT
     #print "Analyzing certificate file" $certificate
     end_date=$(echo | openssl x509 -noout -in ${certificate} -enddate)
     cert_data=$(echo | openssl x509 -noout -in ${certificate} -issuer -subject -dates -email)
     cert_exp=${end_date#notAfter=}

     cmonth=$(echo ${cert_exp} | cut -f1 -d' ')
     cdate=$(echo ${cert_exp} | cut -f2 -d' ')
     cyear=$(echo ${cert_exp} | cut -f4 -d' ')
     #print "Current certificate expiring on this day," $cmonth $cdate $cyear
   case $cmonth in
             Jan)
               certmonth=1;;
             Feb)
               certmonth=2;;
             Mar)
               certmonth=3;;
             Apr)
               certmonth=4;;
             May)
               certmonth=5;;
             Jun)
               certmonth=6;;
             Jul)
               certmonth=7;;
             Aug)
               certmonth=8;;
             Sep)
               certmonth=9;;
             Oct)
               certmonth=10;;
             Nov)
               certmonth=11;;
             Dec)
               certmonth=12;;
             *)
               certmonth=Error:;
esac
let monthplus=month+2
let monthdiff=month-certmonth
let yearplus=year+1
total_files=$(ls -l $files | wc -l)


if (( (certmonthcyear)));
 then
 print "####################################################\nAnalyzing certificate file $certificate\nEXPIRED EXPIRED EXPIRED EXPIRED\n$cert_data\n">>$expired_output
 let z=z+1
else
if (( (certmonth<= monthplus && year==cyear) || (monthdiff==10 && cyear==yearplus) || (monthdiff==11 && cyear==yearplus) ));
then
print "*****************************************************\nAnalyzing certificate file $certificate\nCurrent certificate expiring on this day," $cmonth $cdate $cyear "EXPIRING SOON! EXPIRING SOON!\n$cert_data \n">>$warning_output
let x=x+1
else #print "Not Expiring.\n"
let y=y+1
fi
fi
done
print "Number of files scanned=$total_files\nNumber of valid certificates=$y"|tee -a $expired_output $warning_output $manifest_output
print "Number of expiring certificates in report=$x"|tee -a $warning_output $manifest_output
print "Number of expired certificates in report=$z"|tee -a $expired_output $manifest_output

cat $expired_output|mailx -s "EXPIRED CERT REPORT FOR SI" $email
cat $warning_output|mailx -s "CERTIFICATE WARNING REPORT FOR SI" $email

Thursday, May 2, 2013

CRON Example with setting Clearcase Environment

I am not an avid user of Clearcase but I recently had to use it in a CRON job.  First write your normal script.  Then I had to use this approach to get the script to work by setting the clearcase environment first.
This script sets the environment then calls the main scripts.

#!/bin/ksh
export PATH=/usr/atria/bin:$PATH
HOME=/home/k/kcastell/dp-mgt
s_vobs=/vobs/webservices/DP

cleartool setview -exec "/home/k/kcastell/dp-mgt/scripts/SIcertreport.sh" kevin_unix
cleartool setview -exec "/home/k/kcastell/dp-mgt/scripts/Prodcertreport.sh" kevin_unix

The "kevin_unix" is the name of my view in clearcase.

It is all based on the use of "exec" in each line.

Wednesday, May 19, 2010

Do You Repeatedly Create Certificates using Openssl?

If you are like me and maintain a decent number web servers you probably also have more than one SSL certificate on each server. It doesn't take very long for the maintenance to get out of control. I created a standard unix script that walks me through the process using openssl command line parameters. Here is my first go at this script. I am using a Solaris Sparc server and the standard installation of Openssl on Solaris.



bash-3.00$ vi create_key_csr.sh
"create_key_csr.sh" 47 lines, 1561 characters
#/usr/bin/ksh -vx
#This script will create an apache keystore and csr file to be submitted to a CA
OPENSSL_DIR=/usr/sfw/bin #Verify this is correct on your system
SCRIPT_HOME=/opt/app/apache2/scripts #Could be any location you choose
KEYPASSWORD=xxxx
KEYPASSWORD2=yyy


#First create the keystore for the user
echo "Please provide the name of the keystore you want to create."
echo "The file should end with .key extension. "
read KEYSTORE
echo "Second enter the location where you want the keystore created: "
echo "Do not put a trailing slash and the location must exist. "
read DIRECTORY
echo "This is the location and name where the keystore will be created."
echo $DIRECTORY/$KEYSTORE


#echo "Choose a password for this private key store. This should be something y
ou never forget!"
while [ "$KEYPASSWORD" != "$KEYPASSWORD2" ]
do
echo "Choose a password for this private key store. This should be something yo
u never forget!"
stty -echo
read KEYPASSWORD
stty echo
echo "Verify Password:"
stty -echo
read KEYPASSWORD2
stty echo
done
echo "Key Password accepted."

$OPENSSL_DIR/openssl genrsa -des3 -passout pass:$KEYPASSWORD -out $DIRECTORY/$KEYSTORE 1024

#Encrypt output private key using 128 bit AES and the passphrase ``hello'':
# openssl genpkey -algoritm RSA -out key.pem -aes-128-cbc -pass pass:hello

############################################################################
#GENERATE CSR

#The openssl.cnf file contains your repeat values. I have certificates that have the same values #for most fields. Put the default values in your openssl.cnf file to store the defaults.
echo "What is the name of the Certificate Signing Request file?:"
read CSR
$OPENSSL_DIR/openssl req -new -config openssl.cnf -key $DIRECTORY/$KEYSTORE -out
$DIRECTORY/$CSR



exit

#There is no warranty on this script. If you see a problem let me know. It is an initial pass at #building a repeatable process for creating certificates.

When the script is completed you should have a key file, and a CSR file ready to give to any Certificate Authority.

Tuesday, April 27, 2010

Apache 2.3 in Beta

There is good news and bad news. The good news is there is a new version of Apache already in Beta release with some pretty good features. The bad news is there is a new version of Apache and if you are like me you hate migrating or updating.

I didn't make this up and you can view the entire list of updates at this link:
apache updates in 2.3

I am going to list a few of the changes that I will look to test and seem useful to me. Remember to check out my most recent blog performing a complete build of the LDAP client using SSL to authenticate a web user.


* mod_authnz_ldap: Add AuthLDAPBindAuthoritative to allow Authentication to try other providers in the case of an LDAP bind failure.PR 46608 [Justin Erenkrantz, Joe Schaefer, Tony Stevenson]

*) mod_ssl: Add support for OCSP Stapling. PR 43822.
[Dr Stephen Henson
]

* mod_ldap: If LDAPSharedCacheSize is too small, try harder to purge
some cache entries and log a warning. Also increase the default
LDAPSharedCacheSize to 500000. This is a more realistic size suitable
for the default values of 1024 for LdapCacheEntries/LdapOpCacheEntries.
PR 46749. [Stefan Fritsch]

*) mod_ratelimit: New module to do bandwidth rate limiting. [Paul Querna]

*) mod_heartbeat: New module to generate multicast heartbeats to know if a
server is online. [Paul Querna]

*) mod_privileges: new module to make httpd on Solaris privileges-aware
and to enable different virtualhosts to run with different
privileges and Unix user/group IDs [Nick Kew]

*) New module mod_sed: filter Request/Response bodies through sed
[Basant Kumar Kukreja
]

*) mod_auth_form: Add a module capable of allowing end users to log
in using an HTML form, storing the credentials within mod_session.
[Graham Leggett]

*) mod_ldap: Correctly return all requested attribute values
when some attributes have a null value.
PR 44560 [Anders Kaseorg
]

*) mod_ldap: Add support (taking advantage of the new APR capability)
for ldap rebind callback while chasing referrals. This allows direct
searches on LDAP servers (in particular MS Active Directory 2003+)
using referrals without the use of the global catalog.
PRs 26538, 40268, and 42557 [Paul J. Reder]

*) mod_ldap, mod_authnzldap: Add support for nested groups (i.e. the ability to authorize an authenticated user via a "require ldap-group X" directive where the user is not in group X, but is in a subgroup contained in X. PR 42891 [Paul J. Reder]

*) mod_authn_dbd: Export any additional columns queried in the SQL select into the environment with the name AUTHENTICATE_ This brings mod_authn_dbd behaviour in line with mod_authnz_ldap. [Graham Leggett]


These are just a few of the main ones that I think will be very useful. Obviously adding additional functionality with LDAP lookups is great, but the additional ability to use a relational database to retrieve parameters makes the server super flexible. I will have to revisit the build of the mod_dbd. I have not had great success with Oracle connectivity in the past from Apache.

I am also curious about a rate limiter, and a built in login functionality. This sounds like a encroachment into the SSO world by Apache, but I am all for it.











Monday, March 15, 2010

Part 2- Configuring Apache to use LDAP Authentication

After trial and error I found a configuration that worked for me. I have an IIS CA set up on my network. I figured it would be the trickiest to get working.

In this version(2.2.14) of Apache it is important to know the trusted global cert is a fairly new setting. Some blogs still referenced an older setting that is not valid with this version.
I took the default httpd.conf and added a url that I wanted protected with an LDAP authentication.


SSLRandomSeed startup builtin
SSLRandomSeed connect builtin


####Here are the pieces I added.

LDAPTrustedGlobalCert CA_BASE64 /opt/app/apache2/conf/certs/LDAP/iis-root.cer

LDAPVerifyServerCert On #If you use this set to OFF it is not valid for production environments.


AuthType basic
AuthName "LDAP Protected"
AuthBasicProvider ldap
AuthzLDAPAuthoritative on
AuthLDAPBindDN "cn=directory manager" #I also don't suggest you use this account!!
AuthLDAPbindPassword admin123
#AuthLDAPURL "ldap://dev04.company.com:1640/o=abc,c=us?uid?sub?(objectClass=*)
AuthLDAPURL "ldaps://dev04.company.com:4001/o=abc,c=us?uid?sub?(objectClass=*)
Require valid-user


###End of File

My notes are this. Make sure you use the LDAPVerifyServerCert set to On. Using this set to Off allows any certificate to be used and pretty much defeats the purpose of SSL. Although you may have encryption you won't know who you are talking with that set incorrectly.

I did not have any luck when I wanted to trust a single individual certificate. I tried pulling the LDAP server's certificate from a Sun LDAP Directory but Apache would never seem to trust an individual certificate no matter what setting I used.

I then changed the Sun LDAP Directory Server Certificate to use a certificate from a CA. That worked. It was even a Microsoft CA, Sun LDAP, and Apache using the OpenSSL libraries for connectivity.