Stoner's World

Disclaimer: I work for a company that offers cloud services.

Henry Newman’s October 9th article at EnterpriseStorageForum.com wants to steer you away from cloud storage for reasons I don’t grasp. It’s filled with a lot of numbers and Doom’s Day scenarios but what it doesn’t say is that these numbers and scenarios apply equally, regardless of whether you’re looking at Cloud Storage or a traditional, on-premise storage solution. I have several problems with the article. Let’s start with this statement about cloud storage’s potential:

This is followed by 3 tables about disk error rates with all kinds of numbers. Disk error rates are what they are. Until manufacturers build more reliable devices, the only thing you can do is build in redundancy and error checking to compensate. This affects cloud storage and traditional solutions alike so it’s really a non-factor. Oh yeah, he does point out that commodity (consumer-grade) drives tend to have lower hard error rates and, thus, higher catastrophic loss potential. There’s no arguing this. What I will argue is his “commodity drives typically used in cloud services.” Yes, some providers use commodity (consumer-grade) drives to keep costs low. Other cloud providers use enterprise-grade drives. Some enterprise storage vendors use commodity drives in their enterprise products, so it it fair for me to say “commodity drives typically used in enterprise storage?” Let’s get real here, the devil is in the details. Talk to your storage provider, be it an “enterprise” vendor or a cloud vendor. Make informed decisions based on fact, not conjecture.

Here’s another quote from the article: “one of the biggest issues is simply that hardware is going to break.” How is this going to limit cloud storage’s potential for enterprises? I’m sure he doesn’t mean to suggest that a traditional, on-premise enterprise storage environment has hardware that doesn’t break. Beyond that, I’m not sure why this is even brought up.

Okay, let’s look at these “bandwidth limitations.” I’m still trying to figure this one out. How is bandwidth limiting for Cloud and not for a traditional solution? SAN fabrics have limited bandwidth just like the Internet. His “point” is made using a worst-case scenario – a complete rebuild of a large data set. Yes, bandwidth is going to be a huge bottleneck – for both cloud and traditional solutions. If you’re trying to rebuild a data set in San Francisco using data in New York – it doesn’t matter what technology you’re using. The same data is going over the same wire at the same speeds. It’s going to take 1 day to replicate 1 Petabyte of data using an OC-768 connection for either cloud or traditional solutions, making bandwidth a non-factor when comparing both approaches.

This statement really makes me scratch my head:

So, how is all that data going to get to the recovery site? That’s right, replication over limited bandwidth. The same “reason” suggesting that bandwidth is going to limit cloud storage’s potential is going to limit your traditional enterprise storage environment.

Let’s cut to the chase. “Cloud” is like “cluster” – it means different things to different people. There are many ways to build a “cloud storage” solution. You can use enterprise-grade hardware, RAID, multiple replicas – everything you might use in a traditional storage environment or you can cut costs and go with cheaper hardware, eliminate RAID, keep only one replica of data…it’s all in how it’s engineered. Secondly, cloud storage has it’s limitations, just like a traditional storage environment. It can be used very effectively or it can be used very poorly. If you use something in a way it wasn’t designed and it performs poorly, you really can’t blame the technology, only yourself. Yes, you can drive in a nail using a wrench but I’d rather use a hammer.

One of the key points that makes “cloud” storage what it is is the technology used to access it. For example, Amazon’s S3 storage uses an API, either REST or SOAP calls (technically, you can use regular HTTP requests but you’ll have to parse some XML but it’s really as simple as this.) This is a world of difference from traditional, file/directory-based storage accessed via open()/fopen() system calls. This is what gives cloud storage it’s great flexibility. From an application perspective, the storage is “everywhere and nowhere, all at once.” I don’t have to connect to a particular data center, use weird protocols or hardware connectors – it’s all done using the familiar HTTP protocol or an application library that wraps REST/SOAP calls in convenient functions or object methods. And since cloud storage is built on TCP/IP protocols, I can leverage firewalls, IDS solutions and, in the case where REST/SOAP calls are used, load balancers and caching systems to improve performance and reliability.

In summary, it’s how you engineer a solution. Garbage in, garbage out. If you engineer an enterprise-grade solution, you get enterprise-grade benefits and results. Remember, it wasn’t very long ago that GNU/Linux was considered a toy operating system and not enterprise-capable. Today, GNU/Linux powers 88% of the most powerful compute clusters, as reported by TOP500.

So I’m whipping up a quickie Perl script to dump the bodies of all functions/routines out of MySQL, each to its own file. Yes, I know I can feed a short SQL statement to the mysql client or use mysqldump to do this. I enjoy writing these short-n-sweet scripts to do this. It lets me explore and learn. So anyway, I’m testing against my local  instance using the root account when the Principle of Least Privileges creeps into my forethought. What are the fewest MySQL privileges necessary to ensure my script works properly while ensuring the least amount of damage can be caused?

Read More >>

We lost power on Sunday. No big deal…the UPS kept my desktop, router and a few choice pieces of equipment running until my generator automatically kicked on. Unfortunately, the lights on my Linksys Wireless/VoIP router didn’t come back on (it’s plugged into the surge-protected outlet, not the battery-protected outlet.) Damn. I tried plugging the router into another outlet across the room – still no lights. Double damn. I broke out my voltmeter and checked the voltage coming from the power supply cord. Bingo! Zero volts. Luckily, the power supply sacrificed it’s life for the router, <sarcasm>which is contrary to design, the router is supposed to fail before the power supply.</sarcasm>

I went digging in one of my “junk” boxes and came out with one of those select-a-voltage power supplies with interchangeable tips from Radio Shack. This one just happened to put out 1 amp at one of the user-selectable voltages ranging from 1.5V to 12V. I checked Linksys power supply – Output: 12V @ 1amp. The barrel plug on the Radio Shack model fit the Linksys and was already center-positive. A perfect match! I hooked up all the cables and got back to business with a “new” power supply feeding electric juice to the core of my network.

I can’t believe Radio Shack came to my rescue. I mean, usually I find their stuff to be over-priced & cheaply made and usually functions for the 90 day warranty before dying on the 91st day. Some of you may swear by Radio Shack and have all kinds of victory stories but that’s not been my experience over 20 years. So, needless to say, I’m shocked but happy. I should take this time to find a replacement power supply for my Linksys.

Red Hat Certified Engineer logo

At least, Red Hat thinks so. In August, I finally got around to taking the RHCE exam…and passed. Cert #805009770342158.  Don’t believe me? Click on the logo and plug in my cert number and see for yourself.

On many blogs, pages, books I read about measuring MySQL performance in terms of Queries Per Second (qps) and read/write ratio. Unfortunately, most places don’t explicitly tell you how to measure those. One of the reasons, I suspect, is the slowly changing internals of MySQL for reporting the nitty-gritty statistics to figure these out. Well, I’m here to say I don’t know them either, but…I’m going to take a stab at it and I could use some help.

My first stab is at measuring the read/write ratio. I’m working against MySQL 5.0.x releases and I’m looking at the Com_* counters from SHOW GLOBAL STATUS, grabbing them every $sleep_time seconds. There’s quite a number of them to look at. Many can be ignored, like Com_show_grants or Com_show_plugins. Of the rest, they get dumped into either a “read” or “write” bucket. From there, just do some simple math (calculate deltas, running totals, etc.). There are 2 Com_* counters that I consider as both read and write, Com_insert_select and Com_replace_select, because they read data from a table and write it back to a table. Here’s the list of Com_* counters I’m considering:

Reads: Com_select, Com_insert_select, Com_replace_select
Writes: Com_alter_*, Com_create_*, Com_delete*, Com_drop_*, Com_insert*, Com_load*, Com_rename*, Com_truncate, Com_update*

There are several Com_* counters that can (potentially) generate reads/writes in the database. Things like Com_check, Com_flush, Com_repair, etc. I’m not sure how to consider those. For now, I’m ignoring them. So, community, am I getting the correct counters? Am I interpreting them correctly?

Oh, you want to see the code? Fine. Jump over here.

Overall, I have a problem with counting statements. It doesn’t take into consideration data volume. If you use an extended INSERT, you could load thousands of rows but only register a single INSERT in the Com_insert counter. Similarly, if you do a ‘DELETE FROM tablename’, it only registers as a single statement in the Com_delete counter but it could cause a huge number of disk writes (in proportion to read statements.) So, after getting statement counting nailed down, I’ll be investigating how to measure data volume read versus data volume written. Looking at both statement counters and volume measurements should give admins a better idea how their database is performing.

An email (one of our partner’s marketing newsletters) forwarded to me by my manager had this quote in it:

“SaaS represents a technology evolution that will eventually displace ERP client-server models, much like ERP client-server models displaced mainframe models in the 1990s.”

Boy, someone’s really hitting the Koolaid hard. If you strip away all the buzzwords from the past 20 years, strip away all the crap, you’ll find we’ve only come full circle. Consider this:

mainframe + dumb terminals

• the dumb terminal connected to a serial “network” allowing you to run the app stored on the mainframe
• all processing was done on the mainframe
• the dumb terminal couldn’t do anything else

client-server

• much of the work, like input validation and data display, was pushed from the sever to the desktop/client
• the desktop/client connected to an actual network
• the desktop/client had games, allowing the user to keep occupied when the server or network was down

SaaS

• the desktop/client was replaced with the dumb terminal, called the browser
• all data validation moved back to the server, the browser’s only function is as an eye-candy display that used display technologies (XHTML, DHTML, CSS, Javascript, etc.) generated/stored on the server
• the user still has games to occupy themselves when the server or network is down

The only difference between the mainframe/terminal days and SaaS is: the “mainframe” is managed by someone else and the network is a bit more complex than plain, old serial lines.

I’ve started my first brew. I’ve got the crushed crystal, chocolate malt and black patent grains steeping. The cans of liquid malt are in a pan of hot water (warming it will make it pour easier.) Once all the malt and bittering hops are in, it’s a 45 minute boil, then finishing hops for another 10 minutes. After that, I toss in the aroma hops for 5 after I remove it from the heat. From there, it’ll get chilled, poured into the carboy and I’ll toss the yeast, plug it up with an airlock and let the magic happen. This recipe is for an American style nut brown ale.

Edit: I’ve entered the boil. From here, it’s a 45 minute wait (with the occasional check-in to make sure things aren’t boiling over.)

Lessons learned #1: conduct boil where there’s no wind. The wind is blowing enough to dissipate the heat, which is causing the boil to go from a strong simmer to a rolling boil. The variation is killing me. Next time, get a wind block, check the weather or do it some place with 4 walls and a ceiling.

Edit: just added the finishing hops. 10 more minutes of boil, then a 5 minutes rest after adding the aroma hops.

Edit: the brew is done. It’s sitting in the carboy, wrapped in a blanket, fermenting. Now we play the waiting game.

My autosiphon, tubing and bottle brush arrived on Thursday. My father-in-law brought the propane burner Saturday. I had everything necessary to begin brewing my first homebrew…except good weather. We worked all day Saturday (when the weather was prefect) so I couldn’t fire up the brew pot. On Sunday, the clouds rolled in and it sprinkled all day, which meant I couldn’t boil wort.

It wasn’t a total loss. On Saturday, the family piled into the Blazer and we headed over the Strasburg to the antique dealers. I picked up two nice decanters. I scrubbed them out using the bottle brush and plenty of hot water+soap. I filled one with my homemade amaretto and saw exactly how much sediment was in it. Turns out, not too much. Next time, I’ll double-up the cheese cloth to filter out more of the lemon zest. I’ll also use some brown sugar to give it a darker amber hue (plus the touch of molasses should be nice.)

I’ve been toying with the idea of making moonshine to serve as the alcoholic base for future liquor-making endevors. I need to research the legality of that. Quite frankly, I’m happy using vodka as the base for amaretto but I feel it would be more fun to do it all – distill my own alcohol then turn it into something more, like a Southern Comfort or a Drambuie-style liquor.

Over the past weekend, I went on a hunt for beer-making supplies. In Winchester, I found Murphy Beverage Company. They have beer and wine and equipment for making said beer and wine. It’s a small store with a corresponding small selection of…everything. But, they do carry hops, malts and yeast and since it’s the only store in the area like it, its all good. I bought a funnel with a built-in screen, a bottle capper and a carboy brush.

I still lacked a very, very key piece of equipment – a brew pot. What I wanted was a 6 or 7 gallon stainless steel pot. Catalogs, like Cabela’s, list such a device at approximately $120-$150, not including shipping. I went to every place I could think of that might carry such pots, Gander Mountain, Sears, JC Penny, Lowe’s, Home Depot, etc. The closest I could find was a 5 gallon aluminum pot, which is both too small and the wrong metal.

Finally, I broke my long-standing resolution to never again set foot in a Wal-mart. Let me tell you, I hate Wal-mart. With a passion. Every store I’ve been in is always the same: 8 billion people, employees who don’t know where anything is and 2 cashiers. And they’re always like that, no matter the hour of the day.

I went in because my wife needed to return something, shampoo or some such, I don’t know. I was grumbling with a storm cloud over my head at the prospect of walking those aisles. I meandered around and found myself in the kitchen section. I looked up and was blinded by a bright light shining off..what, I couldn’t tell (I was blinded, duh.)  I shuffled over and reached out a hand to grasp…a handle of a metallic object. I pulled it off the shelf with trepidation. The weight forced me to hug it close to me. When my vision returned I looked at what I was holding. It was a large cylinder made of metal. As I rotated it in my hands, the label came into view (well, the English text of the label, that is.) A 22 quart, stainless steel cooking pot! A quick, mental calculation – 4 quarts in a gallon, 22 divided by 4 equals 5 remainder 2 or 5.5 gallons. Disbelief shone on my face like a fog light searching out lost ships. I was large enough*, the right metal, and the price…an unbelievable $49.99.

I trembled as I made my way to the registers. Something wasn’t right with the scene before me. The nearest 3 registers had only 1 person each, not the line to the back of the store I was accustomed to. I nearly fainted but was able to steady myself by leaning on the elderly lady carrying two 50 pound sacks of water softener salt. I slid into the register aisle closest to me. As my pot inched closer to the magical laser thingy that scans the bar code, I kept looking around, expecting someone to tackle me, declaring that I was in the special “elderly persons faced with hip replacement surgery” line and forced to carry my item to the back of the store to wait in line for a cashier with bad acne and no teeth. With trembling fingers, I swiped my credit card through the card do-hickey, fearful that I would wake up in my bed – all this a cruel dream. When the cashier handed me the receipt, I realized it was real…I wasn’t dreaming. I had purchased an item at Wal-mart without waiting an eternity in line behind a down-trodden mother carrying a baby with an ear-splitting wail. The cashier was courteous, polite and looked to have at least a high school education. I danced my way to the car, jubilant at finding my brew pot!

Oh happy day!

* I was hoping for a 6-7 gallon pot but a 5.5 will do nicely. I plan to freeze 1 gallon of water and brew using 4 gallons. When the boil is complete, I can put the 1 gallon ice block in to a) cool the wort and b) bring the volume closer to 5 gallons.

So far, I have the following equipment and ingredients for my brewing adventure:

• 5 gallon carboy
• hydrometer (with tube)
• 2 air locks
• sanitizer
• liquid malt extract
• powder malt extract
• grain malts
• grain bag
• hops (pellets)
• yeast
• priming sugar
• bottle caps

All of this courteous of my father. The equipment is from his wine making excursions. The ingredients come from a kit he bought me for my birthday. It’s for a brown ale, which suits me just fine.

I still need the following equipment:

• 8 gallon stainless steel pot
• propane burner and stand (optional but preferred)
• food-grade tubing
• siphon
• bottle filler
• capper
• secondary fermenter or a bottling bucket
• thermometer (something more appropriate for brewing then the ones I already have)
• bottle washer

I figure I can easily make a bottle tree for drying the bottles after sanitizing them. Oh, and I need bottles – about 5 dozen or so. I’m excited. This is going to be fun.