Steve's House of Virtualization and Waffles


Content! Sweet, Sweet Content!

I have a bit of an experiment going on over at Chris Wahl's site, which you can see here.

The thought is that maybe having someone harass me periodically for content may lead to more being produced. I'm not sure that follows, but as we've gone almost two years without anything gracing this page, I suppose it's worth a shot.

I'm about 80% pleased with that post. I wish it were a little tighter, a little more focused, and had more of a through-line, but it's in the can and now someone else's problem.  Coming back here to post this little meta-update, I'm struck by how different it is from the Hypervisor Choice post a ways down. It's almost a complete 180. It's as if my opinions have changed based on my experiences.  Growth, I think they call it.

Anyway, read that thing, or don't. I don't know what else to tell you.


Filed under: Uncategorized No Comments

“Game Changer”

Every now and then, I find myself in a conversation that just feels important. The kind you remember in third-person later, the backgound faded away into your own private black box theater production.

Been thinking about a recent one quite a bit lately. A co-worker (the guy responsible for remote access stuff--Citrix, SSLVPNs, etc) was pacing back and forth down our row of cubes, poking away at the "test" iPad, swearing under his breath. I asked him what was up. His reponse was something like this:

"This thing's a piece of shit. I've got Citrix Receiver on here, and it mostly works, but I keep forgetting that hitting the home button closes it, so I can't switch cleanly from published app to published app. I can do published desktop, but then if I want to check the iPad calendar or e-mail, same problem. Real users are going to hate this. We just got back from demoing this to [the guys that run the company], and they declared it a 'Game Changer.' Said it was going to change the way the entire firm worked. I don't get it. How is this kludgy, one-app-at-a-time POS a Game Changer?"

I agreed that the combination of an iPad and Citrix Receiver was somewhat less than ideal. But I could see where this was heading, and did my best to reframe the situation. Users showing up with iPads and saying "make it work" represented a new model for us. We had the rogue Mac user here and there, but the bulk of the userbase was content to passively consume whatever technology was made available. To have people go out on their own and find a piece of technology that seemed interesting to them, and then to ask us to help integrate it into their workflow, well, it would be fair to call that game changing.

Encapsulating the desktop experience into a scaled-down iPad app might be a reasonable place to start--it's quick, easy, and cheap--but I don't think that's really what the users are asking for. They want real integration between the tools they provide and the work they need to do. The iPad has a perfectly good calendar and e-mail app built-in, so why should we ask them to fire up a remote desktop session just to get to Outlook? Why can't they use native tools to produce and review documents, schedule meetings, and whatever else these guys do all day?

I think when the guys that run the company tell you that they want to reshape the way they work around a new piece of technology, there's an implication there that shouldn't be ignored. Help them do it, or they'll find people that will. So in that sense, yes, it's very much game changing. Now it's up to you to decide if it's still a game you can play.

Tagged as: No Comments

Apple Waffle

I've had some complaints about the lack of breakfast coverage. Here's an Apple Waffle from the Original Pancake House in Champaign, IL.

Filed under: Waffles No Comments

Friday Morning Fire Drill – VMware HA and Spanning Tree

When I woke up last Friday, I wasn’t expecting to see 116 up and down alerts from our notification system. Not the most encouraging start to the day. Turns out, some planned but poorly communicated network core work was being done at one of our remote sites, and VMware HA kicked in and restarted everything. The on-site and on-call guys had leapt into action to disable VMware HA for the duration of the planned work, but a spirited discussion over who was to be blamed for the outage had broken out at layer eight. Wagons had been circled, accusations had been thrown, and I hadn't even had my coffee yet.

The ESX hosts in question had a single pair of NICs carrying VMkernel, Service Console, and VM Network traffic. Not ideal, to be sure, but it's the best we could do with the hardware we had. This site has no server distribution layer, so each NIC ran directly into one of two core 6509s. Teaming was configured with both adapters active, balancing based on port ID, failover detection set to Link Status only, and Failback enabled. See below.


Only one core switch was taken offline, so why did an isolation response kick in? Googling and poking around VMware Communities yielded this KB article: My Network Guy counterpart confirmed that the ports in question did not, in fact, have Portfast enabled. This was an oversight in that site's configuration—this was enabled everywhere else.

This seemed to tie a neat little bow around the issue, but were still scratching our heads over why Spanning Tree would cause an issue when a core switch went down. Spanning tree would only kick in when a port state goes up. A shared lightbulb went off above our heads and we went back to check timestamps on the alerts. The down alerts weren't issued when the switch went down, they were issued when the switch came back up.

We had failback enabled. This meant that as soon as ESX detected the original link was available, it would redirect traffic over it. Without Portfast enabled, the link would register as enabled, but be in blocking mode for 30-50 seconds while spanning tree converged. 30-50 seconds being longer than the default 15 second isolation response time, our VMs were powered down. The online documentation confirms this, stating that Link Status only detection "Relies solely on the link status that the network adapter provides. This option detects failures, such as cable pulls and physical switch power failures, but not configuration errors, such as a physical switch port being blocked by spanning tree or misconfigured to the wrong VLAN or cable pulls on the wrong side of a physical switch."

Going forward, we're obviously going to enable Portfast on all ESX-facing ports. We're also going to disable failback for the Service Console portgroup. And we're going to give the Network team a good rap across the knuckles with a ruler, reminding them that in a post-virtualization world, we really need to coordinate these things better. We prefer to treat HA as a safety net, disabling it when we know maintenance is going on.


Bounded Rationality, Satisficing, and Hypervisor Choice

I was a little worried last week. In an internal roadmap meeting, I was reminded of a project queued up for Q3 under the innocuous name of “Virtualization Platform Assessment.”

It’s a good thing. We’re a VMware customer today, our EA is coming to a close next year, and we need to take a good look at other vendors to make sure that VMware’s suite is still the best fit for our needs.

What had me worried, though, is a conversation I had last year, around budget time, when I tried to pitch upgrading hosts in our main datacenters to Enterprise Plus licensing. The comment was something along the lines of “Isn’t what we have now good enough?”

It’s funny how that phrase only really comes up at budget time. Rest of the year, it’s ever-upward, best-in-class, cutting-edge—but come October, Good Enough often carries the day.

While vSphere is easily the platform that gives us the most flexibility and options going forward, I’m worried that the powers that be are going to crack under the pressure our MS reps are applying--“You’re already paying for Hyper-V and SMS-E—why pay twice for virtualization?” Truth of the matter is, Hyper-V (and a few more hosts) probably would be good enough for what we’re doing today, right now. But it’ll really put a damper on what we hope to do tomorrow.

Before I start to sound too petty, I’m not blaming anyone. After some thought, I realized that this is just how people are wired. In an ideal world, we’d take the time and effort to examine all options and select the one that maximized current and future utility. This is optimizing, the basis of the “Economic Man” model of decision making, and is pretty much a pipe dream. More often than not, we don’t have the time and energy for an exhaustive process—particularly with a deadline looming—so we settle. We satisfice, picking the easiest option that is good enough to get by.

This is not a new idea. Herbert Simon caught on to it around 1957, in a paper called “A Behavioral Model of Rational Choice,” packaging it into what’s come to be called “bounded rationality.” Bang your head against the concept too often, and it’s easy to get complacent, and let Good Enough become part of the culture. Complaining is an option—and fun, too—but that’s just tilting at windmills. The only way to get the right things done is to embrace it.

Satisficing choices are made due to lack of time and lack of information. The example above was my own fault—I brought it up too late in the year for it to get the consideration it deserved. That said, there’s an organizational issue to work through, too—proposals for the next year are solicited at a time when it’s nearly impossible for them to get the time and attention necessary to make an informed decision. Hence, satisficing occurs.

I’m taking a different route this year, talking loudly and openly about the projects that I know I want to do next year now, when there’s time for them to be discussed. I'm also trying to actively reach across the aisles, by forming an informal cross-functional working group that meets for lunch once or twice a month to talk through shared pain points and future aspirations. During the formal process in the fall, the good ideas will be old news and should coast through without too much trouble.

This also gets things like SRM and View established on the road map, and means that other vendors will need to be evaluated against those initiatives, properly framing this Fall’s assessment. I'm less worried now.

Tagged as: 5 Comments

VMworld 2009 General Session – Wednesday

About 45 minutes to go until the start of today's General Session. Last year's featured demos of VMware Fault Tolerance, ThinApp, and View, so we should be in for something interesting. As with yesterday, I'll do a half-assed live blog thing during the session. Later in the week I'll post additional thoughts on both days.

And I'm in. I know that picture looks a lot like yesterday's. Just a coincidence.

Rumor is that the list price of all the gear powering the VMworld datacenter is around $35 million. Of that, $3.50 went to wireless access points, and so ends my live blogging experience.

I have notes from the session, but right now I'm a bit fried from taking (and passing) the VCP410 exam, so I'll post a write-up a bit later.

Tagged as: No Comments

VMworld 2009 General Session – Tuesday

7:45 am: I walked in as "Changes" was playing, a song I'll forever associate with Novell's last TV ad campaign. Probably not an association that VMware wants customers making, though it's too late for that. I don't feel quite legitimate enough for the Blogger Table, so I'm camped out by the camera guy.

7:50 am: Camera guy dozing off. I think everyone had a long night.

8:10 am: Hey, this session is only starting ten minutes late. We're making progress.

Timestamping has gotten annoying.

Todd Nielson, COO,  is talking up the company's history. Only 30 of the Fortune 1000 are not using VMware in some capacity. Issues a challenge--if you can talk them into switching to VMware, you'll get a free pass next year.

Todd kind-of-apologized for the lab situation yesterday. Hours will be extended, and he promises everyone that wants to get into them will be able to do so.

Talking about the three pillars of savings--Hard dollars, staff time, and global resources.

Paul Martiz takes the stage. I really love this guy's accent--I could listen to him all day.

Recurring theme--too much time, money, and effort is spent keeping the lights on, not enough on innovation and competitive advantage.

Paul talks about the coming turf war--internal applicaitons developers threaten to outsource application deployment to cloud providers to bypass IT bureaucracy. Advocates encapsulating the "pillars of complexity" through virtualization to minimize that conflict.

Talking up vSphere as a platform, not just a product.

Talking up improvements in storage, networking, performance, and security. Touts reaching 350,000 IOPS.

Next big push: Management. New suite of products that tie into vCenter to extend management options.

Tom Brey from IBM takes the stage. His accent is unremarkable. Demoing power reporting and management functions of IBM servers, and the reporitng integration between the IBM tools and vCenter. Neat stuff--you can report on power consumption per VM.

Back to Paul's dulcet tones. Fleshing out the new management products coming out.  Capacity planning, configurations management, operations managmeent, business continuity. Higher-level offerings around service profiles, service catalogs, self-service, and chargeback. AppSpeed comes in at an even higher-level around applications provisioning and applicaitons scheduling.

Lab Manager Demo.

Talking up SMB offering: VMware vSphere Enterprise Essential--"IT in a Box."  $166 / processor

New service--VMware Go. On paper, a web-based service for automating ESXi installation and configuration. In principle, a way to bring SMBs into the VMware community.

Building the partner ecosystem around the vCloud Initiative. New concept--vDatacenter. VMs->vApps->vDatacenters. Some day, vDatacenters will be able to be hosted by external cloud providers, but managed by the same pane of glass used to administer internal resources. Emphasis placed on the portability of vDatacenters--you can move them back and forth between internal and external cloud. Talks about vCloud Express, a way to rapidly deploy servers with an externally-hosted cloud service.

vCloud Express Demo. Seems cheap--$1ish per server day for reasonable workloads. Being a server guy was fun while it lasted.

vCloud API announced.

Focus switches to VMware View.

Steve DuPree from HP takes the stage to talk about HP's VDI reference architecture. His accent is unobjectionable.

Some dude from TELUS whose name I missed takes the stage to talk about his company and how they've deployed View and PCoIP. Attendance has begun to dwindle.

Paul's wrapping it up. Talking about the SpringSource aquisition.  Talking about a new generation of lightweight frameworks, combined with virtualization, creating radical simplification of applications. Sounds good to me. I'm not a developer, so I'm checking out.

Tagged as: No Comments

The Scene Outside the Self-Paced Lab Room at VMworld 2009

I passed these guys as I left the Moscone Center.

Tagged as: No Comments

Live from VMworld 2009

Got checked in, but they had run out of the fancy blogger buttons. They told me they may have more later, and I'm just self-absorbed enough to go back and ask.

Not too much excitement planned for today. There are four or five self-paced labs I'd like to do, and then it's off to the Solutions Exchange Welcome Reception for schmoozing and then the Tweetup for a drink and continued schmoozing.

Edit: Make that two or three labs. The two I was most interested in--Upgrading to vSphere and vCenter Orchestrator--will not be available today. Squeezing them in later in the week is going to be a challenge.

Tagged as: No Comments

Believe it or not, @vmwarecares

I had planned to hold off on the first post here until I hit the ground at VMworld 2009, but I had something happen this week that deserves special mention.

Without getting too specific, as it's no longer a problem, I had a bit of an issue with VMware earlier in the week. I ran afoul of a particular policy, and I wanted to talk the appropriate party responsible for said policy and ask them to revise it. I talked to eight different people in and around VMware without making any real headway. Everyone seemed very understanding, but told me that they couldn't really help; all they could do was pass me along to someone else.

I totally understand how these sorts of things happen. It's a big company, it was a grey area, and the issue affected only a small handful of people. With everything that must be going on there this week, it's easy to see how people may have wanted to avoid making any extra noise.

But that didn't help me any. I thought about how to best proceed, and I considered posting something semi-inflamatory here and trying to spread it around through various social networking outlets. Maybe I could embarrass someone into action. As much fun as that would have been, I do try hard to only be a jerk when it's the last resort. Also when I think it would be really funny, but that didn't apply here.

In poking around looking for an appropriate outlet, I discovered Rick Blythe at Rick's the Social Media Specialist at VMware, sort of a customer advocate that keeps an eye on Twitter and the VMware Communities looking for issues that may have fallen through the bureaucratic cracks.

I gave Rick a short blurb about my issue, and within two days he had the policy revised and my issue put to bed. I have no idea how he got that done (pet theory: compromising pictures of executives), but I'm exceedingly greatful.

I wasn't aware Rick's role existed, and I'm sure I'm not the only one. Do keep him in mind should you run into a problem going through the official channels.

Filed under: Virtualization 1 Comment