06.26.08
Posted in Technology, Linux, MySQL at 4:07 pm by Stoner
A bit of advice to anyone wanting to write an article on MySQL that includes setting up users: familiarize yourself with the concept of Least Privileges. That is, only grant those privileges absolutely necessary to do a job and nothing more. I just finished reading an article on how to set up RSyslog to log to a MySQL database. Halfway through the article is a listing showing the grant statement. I’ll share just the fun part:
grant ALL ON Syslog.* …
My first reaction when I see a “grant all” is to ask: why? Why does an application need every database privilege? Well, I finished the article, then went to the RSyslog web site and spent all of 4 minutes researching why the app needs so much privilege. As it turns out, it doesn’t. Right there in blank and white:
“It is sufficient to grant it INSERT privileges to the systemevents table, only.”
Whew. Okay. The app doesn’t need all those privileges. When I checked the author’s credentials, he presents himself as someone who manages firewalls, antispam and antivirus systems - someone you would hope would be security-conscious and aware of privileges. Apparently not - or at least, not when it comes to databases. I couldn’t find an email address to write to the author nor does the web site have a form to post feedback on articles so I can’t point this out nor the fact that he completely left out how to configure RSyslog to connect to MySQL.
<sigh>
Permalink
06.24.08
Posted in Technology, Linux at 5:51 pm by Stoner
The Redhat Summit was a success for me. I got a lot of useful information from the presentations I attended, especially the “Performance tuning Jboss Enterprise Application Platform” session given by Andy Miller. Andy talked about tuning many of the pools in Jboss, like the thread pool and database connection pools. The juiciest piece of info was running Jboss with HugeTLB. We’re going to hit that one hard in terms of testing, validating and deploying in our production environments.
The “Optimizing the SOA Enterprise: Using Jboss and Redhat Enterprise Linux virtualization” session, given by Isaac Christofersen of Booz Allen Hamilton, was also very interesting. The built a cluster of Xen VMs, then build a cluster of Jboss instances in it. They used GFS on top of iSCSI LUNs for shared mounts. This got them away from HBA cards and deploying a separate SAN fabric. The trickiest part they encountered was setting up VLANs on the VM bridges. They had to modify a script or two to get it to work correctly. The benefit they got was being able to provision a Jboss app server in minutes on a VM infrastructure. Everything was highly available (a cluster in a cluster) and they actually realized greater performance then Jboss app servers sitting on real hardware.
The “What’s the fuss about fastboot and the new kernel crash dumping?” session, given by Vivek Goyal of Redhat, was another very interesting session. It talked about kdump and kexec, booting another kernel without rebooting the hardware and more. The ability to jump from one kernel into another (kexec) provides some really good benefits. kdump is built on kexec and provides a great framework for kernel developers and dealing with some really tricky kernel crashes. If you have unexplained kernel crashe, look into this as it stands to provide more and better information for resolving issues.
Finally, the “Augeas: A Linux configuration API” session, given by David Lutterkort of Redhat, was good. Augeas will fill a hole in Linux nicely, if they can come up a way to effectively manage “complex” configuration files, like DHCP, Apache and any XML-formatted file. Augeas uses regular language to pick apart and piece back together a config file and those mentioned do not lend themselves to simple regular expressions.
I went to other sessions but they were either a wash (mostly things I already knew but was deceived by a poorly written description) or a dog & pony show trying to sell me products/services.
Other highlights of the Summit:
- IBM sponsored Wednesday night’s dinner at Fenway Park. Fenway is a great baseball stadium. I got some grainy photos on my cell phone and sat on the Green Monster.
- Redhat provided lunch during the conference, which meant I could spend more on dinner.
- Schwag I got: a Redhat backpack, a baseball with Redhat and IBM’s logo on it, a plushTux penguin from Trusted Computer Systems, Inc., a tee shirt from QLogic, a foam Tux penguin rom R1Soft, a 1 GB USB drive from the Fedora Project, a cheesy monitor duster from BakBone Software
- The seafood in Boston is great. I had several varieties from chowder, which were all better than anythig you’ll get from a can. Oyster bars rule! Free wifi rules over $10.95/day hotel wired connections (always use a VPN for any network connections, regardless.)
Permalink
02.26.08
Posted in Technology, Linux, MySQL at 5:49 pm by Stoner
At $WORK we typically use Redhat Cluster to make MySQL highly available. We have a number of 2-node clusters (active/passive) and it works nicely, for the most part. One thing I find very annoying is RHCS has no notification framework to notify you when a service is relocated to another node (because of failure or otherwise.)
In a perfect world, the only reason MySQL would pass between nodes is for server maintenance, under the control of a human. In reality, crap happens. Redhat Cluster manages all this, but I still want to know that a failover happened (is hardware going bad? did a software bug cause a failure? etc.)
Most monitoring tools weren’t designed to report on a clustered service. I can add a check that connects to MySQL using the cluster-managed IP - but that only tells me if MySQL has failed completely. The failover between nodes generally takes under a minute so it’s easy for that type of monitoring check to miss a failover. Parsing syslog output is less than optimal because syslog is unstructured text. Blech. Process and disk checks can’t be used on each node because they’ll only exist on one node, the active node.
My current solution is a bit of a kludge but it works. We configure MySQL to use standard names for binary logs and error logs (they never change, regardless of which node the database is running on.) The only file name that isn’t standardized is the pid file, which defaults to the hostname of the server running MySQL. So, with that bit of info, I wrote a script that connects to MySQL using the cluster-managed IP and issues the query “show variables like ‘pid_file’” and parses the result. It then compares the hostname against a stored value in a file, which contains the hostname of the current active node. If the hostname changed, then MySQL has changed nodes (whether due to failure or manual relocation.) This changing of hostnames is considered an error condition and the monitoring system sends alerts appropriately. The NOC staff consult procedures to determine the level of failure and call people if necessary. Clearing the error requires editing the file and setting the current hostname as the active node.
What would be nice would be some informational functions in MySQL to give a bit more data about the server itself, things like hostname, IP address MySQL is bound to, port, etc. Then I could use “select hostname()” or similar instead of relying on a file name. Before someone says “you connected to the database so you know all that already,” not all of this information is known to a database connection, particularly if your application uses a database connection pool or the connection goes through a proxy, masking the real database.
Permalink
02.25.08
Posted in Technology, Linux at 6:10 pm by Stoner
I always get a chuckle when I see a new product released and within hours/days/weeks, there is a Slashdot article about someone porting the Linux kernel to it. What is this fascination with running Linux on everything?
“Hey John, I just picked up this safety pin at Worst Buy!”
“Cool, Bob. Let’s put Linux on it!”
Personally, I don’t care that you can install Linux on a Roomba or some other device, complete with Asterisk, and control a Roomba to vacuum your house with your cell phone (yes, someone actually did this.) I don’t foresee myself in a hotel room halfway around the world thinking to myself “I wish there were a way I could vacuum my house right now so it’d be clean when I get home.”
The way I feel about this is:
- Just because you can does not mean you should.
- If you do, then don’t complain if hardware doesn’t work right and the vendor doesn’t give you any help.
Permalink
01.28.08
Posted in Technology, Linux at 12:45 pm by Stoner
Open Source means freedom. Freedom to change the product to do what you want it to. Freedom to make your version available to others. Freedom to integrate your changes back into the original. This freedom doesn’t come without a price.
That price is proliferation and decay. I’ll give you a prime example, the GNU/Linux operating system. DistroWatch watches over 350 Operating systems (some are BSD and not GNU/Linux - getting exact numbers isn’t important at this time.) Some are geared toward specific uses, such as use as a firewall, use as a forensic analysis toolset, use as a data center server system or use as a desktop system, to name a few. The problems become evident:
- Which do you use? How do you evaluate and compare distributions?
- How is the distribution governed? Is it one or a few people “throwing darts at a dart board” or is there a formal governing body overseeing the project? How stable is the governing body? What are their motivations?
- What happens if support for a distribution falls off? Do you pick up maintenance? Do you switch? Do you do nothing and hope/pray that someone else will swoop in and pick up the maintenance?
- What happens if someone forks the code to implement a feature or fix you need? Do you switch completely to the new branch? Do you back-port the feature to the branch you’re on?
- What happens if a developer (either for the OS or in the company using the OS) dies or leaves for greener pastures? Who is going to take over their position? Will the replacement follow the road-map or take the project in a new direction?
These are not easy questions to answer. For small and mid-sized businesses, they may be forced to stick with the 800 lb gorillas. Going with a smaller or lesser known distribution may be too much of a business risk. If the 800 lb gorillas don’t offer what you need and a less-known distribution does, do you take that risk? These questions aren’t limited to the operating system, they can be asked of any OSS project, like OpenOffice.org, Audacity, Apache, GNU Cash, etc.
Open Source is great at leveling playing fields, offering choice and spurring innovation but all that comes with a price that some may not be able or willing to pay. Choose wisely.
Permalink
10.24.07
Posted in Technology, Linux, Life at 5:10 pm by Stoner
I’ve been part of the crew interviewing candidates for Linux sys admin positions at the company. I’m surprised and alarmed at the number of applicants that can’t answer Basic Linux 101 questions. Things like
- what is an inode?
- what is a mount point?
- what is the lost+found directory?
These are general *nix questions. They aren’t specific to a particular brand of UNIX or Linux distribution. If you don’t know these very basic things, you shouldn’t be applying for a sys admin job.
Permalink
09.17.07
Posted in Technology, Linux at 5:27 pm by Stoner
Multicast in Linux is a GREAT BIG BLACK HOLE. There is no suitable documentation on it. What I’ve found is many years old, for the 2.0/2.2/2.4 kernel tree or in reference to getting Linux to route multicast packets. The best doc I’ve found is by Juan-Mariano de Goyeneche - from 1998!
Additionally, there are little to no user-space tools for working with multicast. You can use ‘netstat’ to see what multicast groups the host is subscribed to and the number of subscribers…other than that, you are powerless to do anything. What would be nice would be tools to:
- list the subscribers (by process ID and socket number (since an app can subscribe to multiple multicast groups)
- detach a subscriber from a group, something akin to “kill -9 PID:socket_number”
- see the traffic stats for each multicast subscription
Linux is definitely lacking in the multicast department.
Permalink
05.16.07
Posted in Technology, Linux at 11:59 am by Stoner
Redhat Cluster Services is very touchy. It expects everything to be set up properly or it’ll snap like a rubber band stretched too far. One of the things it expects is LSB-compliant init scripts.
Some init scripts do not implement these specs properly or completely. MySQL is a noted violator (both the scripts from MySQL and from Redhat - go figure.)
In particular, you need to be aware of two issues:
- does the script implement a ’status’ function?
- does the script return success when the service is already stopped?
Item #1 is critical to RHCS. It calls the init script once a minute, passing it the ’status’ parameter. If the script does not implement a status function, then RHCS will continuously bounce the service, thinking it has failed somehow.
Item #2 is a little more subtle. When a service is already stopped, stopping it again is of no real benefit/use and according to the LSB, it should be considered a successful action. If a stop-after-stop returns an error, then RHCS will assume something happened that cannot be resolved by failing over to another node and flags that service as failed, a situation that would require manual intervention to correct.
Since MySQL init scripts are notoriously broken, here’s what to look for. In the ’stop)’ branch of the script argument handling case statement (or in the ’stop()’ function if it’s used) there is usually an ‘if’ statement that checks for a MySQL pid file. Many times, it looks like this:
MYSQLPID=`cat "$mypidfile" 2>/dev/null `
if [ -n "$MYSQLPID" ]; then
Or even this:
if test -s "$pid_file"
then
Now, we need to check what happens if there is no pid file in the else clause. Here’s one example:
else
ret=0
action $"Stopping $prog: " /bin/true
fi
return $ret
Here’s another example:
else
echo "No mysqld pid file found. Looked for $pid_file."
fi
In the first example, the return code, zero, is stored in the ret variable. The action call prints a message to the screen and the /bin/true tells action to print a success message. This would be the preferred method of coding an init script.
In the second example, a statement is written to the screen. The return code is simply the exit status of the echo command, which is success (unless the echo command, itself, failed to write to standard out - something that isn’t very likely to happen.) While this example achieves the desired result, it also isn’t 100% LSB-compliant (since it doesn’t use logging functions for reporting the status) and should be avoided.
MySQL may not be the only init script to fail LSB-compliance. If you use RHCS for other services, you should check the init scripts to make sure they pass muster.
Permalink
05.07.07
Posted in Technology, Linux, Life, Family, Hobbies at 4:07 pm by Stoner
Well, maybe not Super Colossal but busy, none the less. On Saturday, I drove to the NOVALUG meeting to sit on a panel for a database discussion. I represented MySQL, Peter took Oracle and Angelo was PostgreSQL. We each did a 10 minute run-down of the major features of the database, then into a Q&A session. Or at least, that’s what we planned to do. Had to take a lot of time to define basics of databases, including terminology like “tables”, “columns” and “database.” I guess I assumed too much of the audience.
We had some good discussions around questions like “How do you size a database server for 500 gigs of data? For 1 terabyte of data? How do you detect and repair corruption? Can I have the database fire off an event when a piece of data changes” etc. Afterwards, a buddy of mine and I went to Old Dominion Brewery for lunch. I had my usual, Angus burger (medium rare) with fries and an Oak Barrel Stout (draft, not bottle.) If you can find Oak Barrel Stout, give it a shot - but be warned, they change the recipe slightly for the bottled version. OBS draft is a whole other world of goodness.
After I got home, I played World of Warcraft for a few hours then went to bed.
On Sunday, after dealing with a client melt-down, I grabbed my .45 and 410 shotgun and headed to Blue Ridge Arsenal to squeeze off a few hundred rounds with my friend. He had his .38 revolver and rented a 9mm pistol and we spent a good hour blowing holes in paper targets. I did manage to shoot myself in the finger….with a staple gun. I reached around the cardboard to put pressure on it so I could staple the target to it and CLICK. Damn staple went into the tip of my trigger finger. Afterwards, we went to Starbucks for some coffee and lengthly discussions on Brazilian strip clubs, Bruce Campbell’s appearances in the Spiderman trilogy, rednecks and other topics of interest.
When I got home, I rinsed the dirt off my car (you have to driver a half mile on the dirt road to get to my house.) Then I picked up a lot of branches and crap in my front lawn. I never raked the leaves from the fall and the wind storms of this spring blew a lot of dead branches down. Yard work is satisfying work…especially when it’s your own lawn. I have a huge pile of leaves to mulch up…and nothing that needs mulching so I’ll have to dump it in the woods out back. Oh well.
Permalink
04.26.07
Posted in Technology, Linux at 8:51 pm by Stoner
I’m giving Ubuntu 7.04 a spin as my primary desktop. My Gentoo is sitting under the desk with only a power cord and network cable plugged in. I enabled X11 forwarding for ssh and set up an authorized key so I can pull files or run any programs I need.
I just thought of something….the Gentoo box is 32-bit x86 and the Ubuntu is 64-bit AMD - will that make a difference with forwarded X connections?? Gonna hafta give that a real go. I already ran MyPasswordSafe remotely. I did see some cryptic error messages but it didn’t seem to impact functionality of the program, maybe I just got lucky on that one. This’ll be an interesting experiment.
As for running Ubuntu for a full day - it was pleasurable. Some things I ran into:
- I had to install a few audio/video codecs for Totem (so I can watch/hear Windows Media and Mpeg4.) Nice smooth playback.
- Firefox is 64-bit, which means binary plugins like Flash, Java and Acrobat Reader won’t work. I found a page that details how to install the 32-bit Firefox along-side the 64-bit, but really, I don’t want to have to switch between the two. I’ll hold out with the 64-bit Firefox.
- No OTR plugin for Gaim/Pidgin in the default Ubuntu repositories.
Permalink
« Previous entries
