05.16.07
Posted in Technology, Linux at 11:59 am by Stoner
Redhat Cluster Services is very touchy. It expects everything to be set up properly or it’ll snap like a rubber band stretched too far. One of the things it expects is LSB-compliant init scripts.
Some init scripts do not implement these specs properly or completely. MySQL is a noted violator (both the scripts from MySQL and from Redhat - go figure.)
In particular, you need to be aware of two issues:
- does the script implement a ’status’ function?
- does the script return success when the service is already stopped?
Item #1 is critical to RHCS. It calls the init script once a minute, passing it the ’status’ parameter. If the script does not implement a status function, then RHCS will continuously bounce the service, thinking it has failed somehow.
Item #2 is a little more subtle. When a service is already stopped, stopping it again is of no real benefit/use and according to the LSB, it should be considered a successful action. If a stop-after-stop returns an error, then RHCS will assume something happened that cannot be resolved by failing over to another node and flags that service as failed, a situation that would require manual intervention to correct.
Since MySQL init scripts are notoriously broken, here’s what to look for. In the ’stop)’ branch of the script argument handling case statement (or in the ’stop()’ function if it’s used) there is usually an ‘if’ statement that checks for a MySQL pid file. Many times, it looks like this:
MYSQLPID=`cat "$mypidfile" 2>/dev/null `
if [ -n "$MYSQLPID" ]; then
Or even this:
if test -s "$pid_file"
then
Now, we need to check what happens if there is no pid file in the else clause. Here’s one example:
else
ret=0
action $"Stopping $prog: " /bin/true
fi
return $ret
Here’s another example:
else
echo "No mysqld pid file found. Looked for $pid_file."
fi
In the first example, the return code, zero, is stored in the ret variable. The action call prints a message to the screen and the /bin/true tells action to print a success message. This would be the preferred method of coding an init script.
In the second example, a statement is written to the screen. The return code is simply the exit status of the echo command, which is success (unless the echo command, itself, failed to write to standard out - something that isn’t very likely to happen.) While this example achieves the desired result, it also isn’t 100% LSB-compliant (since it doesn’t use logging functions for reporting the status) and should be avoided.
MySQL may not be the only init script to fail LSB-compliance. If you use RHCS for other services, you should check the init scripts to make sure they pass muster.
Permalink
05.07.07
Posted in Technology, Linux, Life, Family, Hobbies at 4:07 pm by Stoner
Well, maybe not Super Colossal but busy, none the less. On Saturday, I drove to the NOVALUG meeting to sit on a panel for a database discussion. I represented MySQL, Peter took Oracle and Angelo was PostgreSQL. We each did a 10 minute run-down of the major features of the database, then into a Q&A session. Or at least, that’s what we planned to do. Had to take a lot of time to define basics of databases, including terminology like “tables”, “columns” and “database.” I guess I assumed too much of the audience.
We had some good discussions around questions like “How do you size a database server for 500 gigs of data? For 1 terabyte of data? How do you detect and repair corruption? Can I have the database fire off an event when a piece of data changes” etc. Afterwards, a buddy of mine and I went to Old Dominion Brewery for lunch. I had my usual, Angus burger (medium rare) with fries and an Oak Barrel Stout (draft, not bottle.) If you can find Oak Barrel Stout, give it a shot - but be warned, they change the recipe slightly for the bottled version. OBS draft is a whole other world of goodness.
After I got home, I played World of Warcraft for a few hours then went to bed.
On Sunday, after dealing with a client melt-down, I grabbed my .45 and 410 shotgun and headed to Blue Ridge Arsenal to squeeze off a few hundred rounds with my friend. He had his .38 revolver and rented a 9mm pistol and we spent a good hour blowing holes in paper targets. I did manage to shoot myself in the finger….with a staple gun. I reached around the cardboard to put pressure on it so I could staple the target to it and CLICK. Damn staple went into the tip of my trigger finger. Afterwards, we went to Starbucks for some coffee and lengthly discussions on Brazilian strip clubs, Bruce Campbell’s appearances in the Spiderman trilogy, rednecks and other topics of interest.
When I got home, I rinsed the dirt off my car (you have to driver a half mile on the dirt road to get to my house.) Then I picked up a lot of branches and crap in my front lawn. I never raked the leaves from the fall and the wind storms of this spring blew a lot of dead branches down. Yard work is satisfying work…especially when it’s your own lawn. I have a huge pile of leaves to mulch up…and nothing that needs mulching so I’ll have to dump it in the woods out back. Oh well.
Permalink
04.26.07
Posted in Technology, Linux at 8:51 pm by Stoner
I’m giving Ubuntu 7.04 a spin as my primary desktop. My Gentoo is sitting under the desk with only a power cord and network cable plugged in. I enabled X11 forwarding for ssh and set up an authorized key so I can pull files or run any programs I need.
I just thought of something….the Gentoo box is 32-bit x86 and the Ubuntu is 64-bit AMD - will that make a difference with forwarded X connections?? Gonna hafta give that a real go. I already ran MyPasswordSafe remotely. I did see some cryptic error messages but it didn’t seem to impact functionality of the program, maybe I just got lucky on that one. This’ll be an interesting experiment.
As for running Ubuntu for a full day - it was pleasurable. Some things I ran into:
- I had to install a few audio/video codecs for Totem (so I can watch/hear Windows Media and Mpeg4.) Nice smooth playback.
- Firefox is 64-bit, which means binary plugins like Flash, Java and Acrobat Reader won’t work. I found a page that details how to install the 32-bit Firefox along-side the 64-bit, but really, I don’t want to have to switch between the two. I’ll hold out with the 64-bit Firefox.
- No OTR plugin for Gaim/Pidgin in the default Ubuntu repositories.
Permalink
04.20.07
Posted in Technology, Linux at 11:12 pm by Stoner
I just loaded Ubuntu 7.04 onto my AMD64 desktop. Very impressive. Kudos to the Ubuntu team for putting together a solid release. Now, I’m very tempted to switch from Gentoo to Ubuntu because it seems so much faster and smoother….but then, I’m comparing apples to oranges. The Ubuntu box is a AMD64 1.4 GHz processor with 512 megs of ram whereas the Gentoo box is Intel Pentium III 800MHz with 256 megs of ram. That’s a major difference.
I tried running Beryl but the video card I slapped in this isn’t up to the challenge (an old Matrox card.) I tried running the Beryl window manager and the screen simply went grey…nuttin’ but nuttin’. I had to Ctrl-Alt-Backspace to kill the X session and login with Gnome. Oh well. Video cards aren’t hard to come by these days.
I do like how the Ubuntu community is more “on top” of releases. For instance, Gaim 2.0.0beta6 is available in Ubuntu 7.04 whereas with Gentoo still has Gaim 1.5 marked as stable and 2.0.0beta6 as unstable. Actually, I masked it so I could install it…and it crashed. Not sure if it was Gaim or the Jabber plugin (I need it to use the Jabber server at work.) Also, I’ve been struggling to get Amarok installed on Gentoo. Some bug somewhere in kde-base/kdelibs (don’t know if it’s a code bug or a bug with the Makefile) is preventing it from installing. Without kdelibs, Amarok just aint gonna happen. With Ubuntu, I selected it in the Synaptic Package Manager and it happily downloaded it and installed it.
Which brings me to another item that’s been on my TODO list: check out Amarok. I’m listening to some selections from the integrated Magnatune.com store because I haven’t transferred any of my music library to this machine (I just built it, damnit.) Very sweet. I like the interface, I like how easy things are. Things “just work” the way I think they should. It just made my list of “must-have” apps for a pleasant computer life.
I’m gonna keep giving Ubuntu and Amarok a spin around the block. There’s so much more I want to check out on it. I need to throw in a CD burner, a new video card, a bigger hard drive (I can easily fill up the tiny 10 gig drive that’s in it now) - hell, I’m gonna try throwing it on my IBM ThinkPad 560 to see how it performs there (this laptop needs a new battery…it won’t even hold the date/time after power off!)
If you run Ubuntu or use Amarok, what do you think of them?
Permalink
04.02.07
Posted in Technology, Linux, Hobbies at 11:01 pm by Stoner
Well, I made some headway on my three projects:
DS-330 Voice Recorder: apparently, people smarter than me have tried to get this working under Linux but had no luck. The encoded files use a proprietry format (DSS) and the SDK is a Windows-only library. So nerts to Olympus. Looks like I’ll have to sell it. Shame really, it’s a nice recorder and worked very well. If yo’re interested in buying it, let me know.
Amarok (kdelibs): still haven’t built this damn library. I changed to serial compiles (instead of the default parallel compiles) but that didn’t fly. I filed an addendum to bug 171397. Hopefully they can figure it out (I have no desire to sift through the tons of code for kdelibs and Qt 3. Ick.)
Gumstix RTC: I tried soldering a 4-wire cable (hacked off an old IDE cable) to the RTC breakout. No luck. The tip on my solder iron is too damn big! I couldn’t get good heat transfer. I’ll have to grind the tip down to a finer point and try again.
I did write a script to save the date and time when I shudown my gumstix (posted it to the mailing list.) That way, when I boot it up, the date doesn’t revert back to January 1, 1970. This screws with programs that use a file’s timestamps (last access, last modification and last change.) My script prevents time from going backwards so those types of programs are happy. A battery-backed RTC and ntpd would be best…and I’m getting there.
Transferred all my music from the Dell/XP to the Gateway/Gentoo. Bonus. Now I can work with my tunes…gotta have my tunes. Music doth soothe the savage beast.
Tomorrow, we’re grilling steaks! A friend sent us some porterhouse steaks from Omaha Steaks! Damn, they’re good! I bought some hickory chips to throw on the coals. Tomorrow’s gonna be some good eatin’!
Permalink
03.28.07
Posted in Technology, Linux, Hobbies at 9:09 pm by Stoner
Got three “projects” going on right now. Need to get my RTC module from Sparkfun soldered to my Gumstix, need to get my Linux system talking to my Olympus DS-330 voice recorder and get kdelibs built so I can try out Amarok.
I’ve had the RTC module for a while now…just never got around to using it. Well, I cut up a standard IDE cable into a 4-wire strip…gonna use that as my power/data bus. The pads on the boards are clearly marked so it’s just a matter of cranking up the ol’ soldering iron and making it happen. It talks to the kernel using I2C so I need to build the kernel modules and install them. After that, my gumstix should be able to maintain time after power is off. [FYI: the gumstix modules don’t have a RTC so when they loose power, they loose the date/time.]
I was rummaging through a box and found my Olympus DS-330 voice recorder. This was an excellent recorder and would be nice to use it again…this time with my Linux box. For kicks, I plugged the USB cable in to my computer but the device wasn’t automatically detected. This means I need to do some research on how to connect it/configure it appropriately. What I’m not sure about is how it should function with the computer. Should it show up as a mass storage device or a USB microphone or both? With Windows, it acts as a USB mic and the recordings are access through the special software they give you. My guess…it won’t work but I’ve been wrong before.
Finally…Wil Wheaton has been harping on Amarok. Well, since I’m moving my Winders computer to the upstairs office…that’ll mean greater dependence on my Linux box in my office. And I can’t go without tunes! I want to give Amarok a spin but, damnit, wouldn’t you know it - kdelibs won’t build (I’m a Gnome-man, myself.) I already checked the Gentoo bugzilla and someone reported it. I need to track that bug because I can’t build Amarok without it. How annoying.
I’ll post updates as my projects progress.
Permalink
03.26.07
Posted in Technology, Linux, Life, Hobbies at 8:01 pm by Stoner
On Sunday, I made a giant pot of chili on Sunday, complete with big chunks of onions, red and yellow bell peppers, deer meat and hot deer sausage, mushrooms and 3 types of beans. I spent over an hour in the kitchen chopping, browning, opening cans, straining, etc. I put in 2 whole fresh tomatoes Denise and I picked up at a local road-side vendor, a can of stewed tomatoes, a big can peeled tomatoes, 2 small cans of tomato sauce and a fair dash of spices. It came out really good so now I have lunch & snack food for the next few days.
Where did I get all my culinary skills? From standing in front of the stove and tossing things into a pot - literally. I don’t have recipes or secret formulas (well, I do but nothing to write about.) I would encourage you to do the same. Don’t plan anything other than a few key ingredients. Let everything come spontaneously.
Today, I finally finished updating my Gentoo system. It hadn’t been updated since I packed it away for the big move in November. That’s 4 months of updates that needed processing. If you’re a Gentoo user, you know that you can’t just “emerge –update world” after 4 months. You need to take it in stages. So, I processed all update up to the kernel itself, recompiled a new kernel, then processed more updates, pausing at key packages. I usually pause at things like udev, glibc, gcc…something very fundamental to the system. I also pause with large packages that take forever to update, like Firefox, X, Gnome, etc. They’re just take an obnoxiously long time to build.
My usual pattern is to update to a certain point, then test the system to make sure things aren’t broken. This time, due to the 4 months between updates, that wasn’t possible because some packages aren’t compatible with others unless they are all updated. Case in point, Gnome. Many Gnome packages are supposed to be updated at the same time. During my update-pause procedure, some Gnome packages were updated while others remained at the lower revision - which means testing is pointless since the different versions will no doubt break things.
In the end, everything updated just fine. When I tested things, everything was snappier and speedier - Gnome menus snap when opened, Firefox and Evolution windows open in mere seconds when launched. I’m very happy. Now I can move this Winders box upstairs and put my Gentoo system on my desk and use it as my primary machine in my office. Life is good.
Permalink
03.13.07
Posted in Technology, Linux at 8:32 pm by Stoner
One of my biggest and most successful clients is using Redhat Cluster Services to make their MySQL database highly available (using a 2-node Active/Passive configuration.) The idea behind a RHCS cluster is to remove all Single Points of Failure, thus, making a service or services highly available. Well, my client’s cluster had a melt down last summer because of one overlooked item.
A quick background blurb on clustering with Redhat. The idea, like I said, is to remove all SPoF by using as much redundancy as possible: multiple systems (nodes in cluster lingo), multiple power sources, multiple network connections, multiple SAN connections, etc. etc. etc. Even if a node completely dies, the cluster software will failover the service(s) to another node in the cluster. That’s the theory and it works rather well in practice.
So, how did my cluster melt down? Well, there’s this subsystem in Linux called the file system. It is responsible for managing the files and directories on some storage medium, like a hard drive. Well, with a cluster, you can’t put your data on a local drive because if that node fails, the other nodes can’t access the data - so you put the data on either a NAS (Network Attached Storage) or a SAN (Storage Area Network.) In this case, we used a SAN because you don’t want to run MySQL over NAS (well, you can but that’s a story for another day.)
Long story short, the file system on the SAN holding the MySQL data files got corrupted. Please note, this was not a fault with Redhat’s Cluster Services. It did what it was supposed to do: fail completely. Failing over the service is pointless because the data on the SAN will still be corrupted regardless of which node is active. Exactly how it got corrupted, I don’t know. I do know that it was an ext2 file system (I didn’t build this cluster, I inherited it) and it blew up in spectacular fashion. I had to rebuild the file system on the SAN (using ext3 this time) and restore the database from a backup that was about 10 hours old. Obviously, my client wasn’t very happy.
To protect against this type of failure in the future, we set up MySQL replication to another MySQL server, one that was not a part of the cluster nor attached to SAN. If we should have a total cluster failure, it would only take minutes to point their application to the replication server and get their sites back up and running. We could then fix the cluster without doing it hastily. In order to put the clustered database back into full production, we would have to halt the application, dump the data and import it back in to the production database, then fire the application back up - a process that would take approximately 20-30 minutes to complete (the dump+restore time is dependent on the database size, obviously.)
A second solution we considered would be to use a clustered file system (like GFS, Lustre or Veritas.) This does add another layer of complexity to the original cluster itself. The second cluster would require full redundancy of all parts, which means more money spent and physical resources to manage. Additionally, you have a cluster dependent upon a cluster, which makes maintenance not very fun at all.
We’ve been running with the cluster+replication scheme for over 6 months and it’s been very solid. We actually had to fail over to the replication database because we had to perform some lengthy maintenance on the SAN. Thanks to this arrangement, my client experienced less than 30 minutes of total downtime during a 4 hour maintenance window. They were happy, which makes me happy.
Permalink
03.09.07
Posted in Technology, Political, Linux at 2:40 am by Stoner
“What are RSA warnings about man in the middle attacks?”
This was posed in a chat room and is a most excellent question. One that has a simple answer but very deep and far-reaching implications.
“What is a Man-in-the-Middle (MitM) attack?”
Quite simply, it’s when an attacker sits between two communicating parties (web browser and web server, ftp client and ftp server, Skype phone and Skype phone, IM client and IM server, etc.), intercepting their network traffic, usually without their knowledge or consent. While in this position, the attacker may choose to simply read the conversation, make a copy of it or change it while it’s in transit. Without proper safe guards, neither the sender nor receiver will know the attacker is there or whether or not their communication was changed in any way.
A common example of this is a telephone wiretap that law enforcement agencies use to record conversations of suspects. The police act as the man-in-the-middle (well, the phone company does all the leg work to tap the line, the police just hit the Record button on the tape deck.)
One important thing to remember is you never have a direct, unbroken line of communication to another entity (think: two tin cans connected by a string), whether it’s a telephone call, an IM chat, an HTTPS session or an email. Your data gets lumped in with a lot of other folks data and pushed along shared paths. Usually, it gets broken down into smaller packets, sent out on a wire, then reassembled at the destination. Another quirk of the Internet is that those packets may not all travel along the same route before they get to the destination.
“So where is this all going?”
Well, first off, you need to know, proof-positive, that the entity you’re connecting to is who they claim to be. If you can’t establish that when you connected to Paypal that you 100%-for-certain are talking to Paypal, then everything else is moot. You also need to know that the data that gets transmitted back and forth has not been altered in any way. Once you connect to your online broker, how do you know that the account number didn’t have two of the numbers transposed on your sell order (possibly depositing the funds into someone else’s account?) And finally, how do you know that someone isn’t reading your social security number off your tax forms when you submit it to the IRS through eFile?
“You sure are long-winded. Can you cut to the chase?”
Encryption. Yes, the strange, fuzzy world of really, really big numbers, obscure phrases like “initialization vectors,” “message hashing” and “cipher block chaining”, complex mathematics and more acronyms than should be allowed by law. This is the technology that can help with all this. Properly implemented and used*, an encryption system can provide the following features:
- identification - verifying with certainty that entities are who they claim to be
- message integrity - verifying that a message was not altered in any way between source and destination
- message protection - transforming data such that only those authorized to read it may do so
(*Note, I said ‘Properly implemented and used.’ It is very easy to screw up encryption systems, whether a means of sharing keys of not handled securely or an algorithm has a programming flaw in it or people simply don’t care enough to follow all the steps necessary to ensure the system is working properly.)
For the rest of this, I’m going to simplify things for brevity, believe it or not, and even fudge a thing or two for sake of clarity. If you know about encryption, you’ll know when I do.
You’re already very familiar with all these features. Let’s examine HTTPS (HTTP over Secure Socket Layer or SSL.) With HTTPS, when your browser connects to an HTTPS web server, that server presents its certificate, identifying it to you. This certificate has been digitally “signed” by a 3rd party system, known as a Certificate Authority (CA.) Your browser contains the certificates of many CA systems (Verisign, Thawte, AT&T, America Online, Wells Fargo…the list goes on) so it is able to verify the 3rd party’s signature on that certificate, even talk to the 3rd party CA to verify that the web server’s certificate genuinely belongs to the server that presented it to you (thus, satisfying feature #1.) Once identity is established, it uses the certificates as part of the encryption process to provide features #2 and #3, thus, protecting you from an attacker.
Now, suppose someone attempted a MitM attack on an HTTPS web site. Part of the web server certificate’s details includes the server name, like www.paypal.com or login.yahoo.com. Your browser checks that the hostname in the certificate is the same as the site it’s connecting to. If they’re different, it will pop up a warning box, alerting you to this inconsistency and allowing you to halt communications. In order for a MitM attack to succeed, the attacker would have to fool you into connecting to a host that looks similar to your intended web destination, like www.paypa1.com (note, the ell is now a one.) This commonly happens in phishing emails because they use HTML to disguise the true URL you connect to when you click on the supposed “update your account” link. If you do click on the link, the MitM attack succeeds because the browser is connecting to a HTTPS site with a valid certificate (www.paypa1.com). The attacker makes a 2nd connection from their site to www.paypal.com, passing the data back and forth - all the while, you think you’re connected to PayPal. At this point, the attacker can simply read the data, recording your username, password, bank account info, etc. or they may even change it, depending on their goals at the time.
“What about the message that started this book you’re writing?”
The message that started all this comes from the SSH protocol. SSH provides all 3 features we want, identification, message integrity and protection. The message is a warning about the identification phase of an SSH connection. When SSH is set up on the server, host keys are generated. These are similar to SSL certificates except there is no 3rd party CA Authority. When you connect to a SSH server with a client (like putty or SecureCRT), you are presented with the host key of the server. At this point, you can accept it or reject it. When you accept it, it gets recorded for later reference. If you reject it, communication stops.
“Without a CA Authority, how do you know you’re connecting to the server you think you are?”
Good question. A host key looks like a lot of random, senseless data. You can get what’s called a “fingerprint” of the host key. This fingerprint is intended to be shared among those who will be connecting to the system and it is much easier to read than a host key itself. When folks want to connect to the server for the first time, their SSH client will show them the fingerprint for the host key presented to it and they can compare that fingerprint with the one they got earlier. If they match, they know the host key is genuine and they can store it for later connections. If they don’t match, they know they aren’t talking to the system they thought they were and can drop the connection without compromising security. The distribution of key fingerprints is left up to the folks managing the systems.
“Anything else I should know?”
Tons. I would encourage you to get more educated about encryption. There’s more to it than math and weird names. There is a pretty serious political atmosphere surrounding it and if you start researching it, you’ll find out just how far reaching encryption is and why it’s so important.
Some links to get you started:
http://www.philzimmermann.com/EN/background/index.html
http://www.schneier.com/
http://en.wikipedia.org/wiki/Whitfield_Diffie
Permalink
Next entries »
