New-RemoteProcess

I have been a bit busy at home and work recently.  Will be starting back up on the white box soon!

I recently wrote about Get-NetworkStatistics.  This basically runs netstat on a remote system and parses the results.  Today, we ran into an issue where we wanted to run ‘set’ on a remote system, and the on-site technician could not get to the command prompt due to certain restrictions.  I already had the framework for running a command remotely, so I made a few adjustments to generalize things.

Getting started

Click here for the function code and ps1.  Once you add the function to your session, you can run commands like this:

image

In this example, I ran ‘set’ against localhost and c-is-ts-91l.  I specified the pattern ‘processors’ to filter the results.  The result is an object for each computer with the computer name, output from the command, errors from the command, the command that ran, and the pattern.

image

In this example, I ran ‘ipconfig /all’ and access the results property.

image

In this example, I ran ‘set’, filtered for ‘computername’, and select only the computer and results.

Do keep in mind the results property is an array.

There are certainly other options for this!  Mark Russinovich’s PsExec is always a solid choice, but collecting and perhaps acting upon the output might require a bit of work.  If you have PowerShell remoting set up in your domain and all systems have PowerShell, that might be a better option.

If you run into any issues or have any suggestions, please let me know!

Parallel PowerShell

*Edit:  Updated code is available in this post

There are times where running parallel tasks in PowerShell would be handy.  Unfortunately, while Workflows can run foreach –parallel, there’s no built in function to handle this in PowerShell.  I came across a few tasks where parallelization would save time, and figured it was time to dive into the subject. I ended up with three functions that heavily borrow from the work of Tome Tanasovski, Boe Prox, and Ryan Witschger.

Before we go into these, I would highly recommend you watch Dr. Tobias Weltner’s webcast on speeding up PowerShell. I found it quite insightful on a variety of topics – certainly well worth the quick registration process. There are other helpful resources on parallelization available as well, including these from Jon Boulineau, Will, Joe Pruitt, and Chris O’Prey.

Do keep in mind that several of the authors above warn that you should use them with caution, and that I am not nearly as familiar with the language as these authors : ) That being said, the modifications I made were minimal; for example, adding a non-functional progress bar, adding AD queries, and other non-runspace changes.

What can Runspaces do for me?

A while back I built a query that runs through AD and queries every single computer account, collecting a variety of information and running certain diagnostics. It is resource light and a helpful backup to existing monitoring solutions. Unfortunately, it takes a very long time, as a good number of systems will not respond; those three seconds add up quickly. Running things in parallel cut the entire query from a day or two to 5 hours.

Running more simple tasks like a single test-connection or a couple WMI queries would see an even greater benefit from parallelization. If you check out Boe’s post on this, you will see that for a simple WMI query upon successful test-connection, a 200 system query takes 22 seconds using runspace pools with a throttle of 10. The next fastest method was the standard synchronous foreach at 158 seconds. If you manage many systems, you can see that using runspace pools will be invaluable.

Here’s another real world example.  I pulled the first 100 computer accounts from AD and simply ran test-connection against them.

8 Minutes 42 seconds

Many of these computers no longer exist, but running queries against computers that might not respond is a likely scenario that will delay your commands.  In this case, the performance difference went from 54 seconds using a runspace pool with 10 threads, to 8 minutes and 42 seconds otherwise.

How do I get started?

Go back and make sure you review the posts above that go into actual technical detail. Then download the code from here.  Add the functions to your profile or a custom module, paste them into your session, or dot source them.  Once the functions are available, run get-help –full against them for more details!  The functions are Foreach-Parallel, Run-Parallel, and Run-ParallelJobs.

The Run-Parallel and Run-ParallelJobs functions each include parameters to query AD.  These query the default domain.  To use the QuerySvrAD and QueryWksAD to query server and workstation computer accounts, you must modify the scripts with the appropriate distinguished names of the OU or containers where these accounts could be found.  Ctrl+F and look for ExampleDC to find the lines where this can be set.

I plan to expand on these; for example, progress tacking could be improved, adding parameters to allow more than one argument passed to the parallelized script(block) by stacking AddArgument(), and adding timeout values or resource constraints on runspaces.

If you run into any trouble or have any suggestions, please let me know!

Other stuff

Google posted a bit on their datacenters and invited someone from Wired to visit and write a story on it.  Wouldn’t it be nice!

Get-NetworkStatistics

Many PowerShell commands offer the ability to run against a remote computer.  If not, PowerShell remoting can help (Helpful resources from Ravikanth Chaganti and Dr. Tobias Weltner / Don Jones).  If you don’t have PowerShell remoting set up across your enterprise, I would recommend looking into it.  More details on executing processes through PowerShell can be found here.

So!  Perhaps you have a command you want to run against a remote system on a domain, and little has been set up apart from standard administrative shares.  One option would be to create the process on the remote system via WMI, save the data on that remote system, and pull the data to your local system for processing.

The other day I was pining for the ability to run netstat remotely.  Shay Levy has a nice script that parses netstat -ano text results into a PSCustomObject that you can actually work with.  He added PID to process name conversions, and filters to return specific processes, addresses, ports, states, and protocols.  This is quite handy as is, but wouldn’t it be great to run against a remote computer?

It turns out someone had done this; The One True Chris posted a script that creates the process on the remote system via WMI and pulls the data back in to your local system.  Unfortunately, this simply returns raw text, limiting your ability to work with the data.

I managed to butcher the code behind each of these functions to create a hybrid.  With this, you can specify a remote system and netstat runs on that system.  The results are retrieved and processed with Shay’s code to produce netstat data in object form.

Getting Started

The function can be found here.  Keep in mind you will need appropriate access to create a process with WMI and connect to the administrative share for remote systems.

Get-NetworkStatistics2

Please go easy on me : )  This is the first code I’m posting publicly, and it certainly isn’t pretty!  Feel free to use and modify this as needed, and let me know if you run into any issues!

Why you need an SSD

Oddly enough, a decent portion of IT professionals still have not experienced SSD. The basic idea is this: with a standard hard drive, there are spinning disks and a mechanical arms that cut down on how many input/output operations per second you get (IOPS) and how quickly you can randomly get a piece of data (latency). With an SSD, there are no moving parts.  No home lab or virtualization solution will be complete without SSD.

Edit:  This was written as an addendum to a post on building a white box for a home lab.  I doubt anyone making storage decisions for their organization would mistake this for advice in a production environment, where you would base your storage solutions on workload, budget, and other factors.  That being said, any production virtualization workload these days should probably have SSD involved, either in a tiering solution or pure (expensive) flash storage for critical workloads with high IOPS requirements.

SSD or Spinning Disks?

So! You have a fancy 15k RPM hard drive or two lying around and don’t need SSD? Your state of the art hard drive likely gets on the order of 200 IOPS. Most consumer level SSDs will give you on the order of thousands or tens of thousands of IOPS. Unless your goal is redundancy, a single SSD will beat out most arrays of high performance spinning disks you might build at home.

Why does this matter? If you want to run multiple virtual machines with their own virtual hard disks and data stored in various locations on spinning disks, you will be severely limited by the mechanical pieces of a standard hard drive. You will find this in the enterprise as well. Ask anyone wearing the virtualization hat what their primary bottleneck is, and you will more than likely hear storage IO (RAM otherwise).  If you plan to run VMs, nothing should be prioritized over an SSD.  RAM would come a close second.

Whither SSD?

Don’t believe the FUD spread about SSD potentially dying early due to limited writes. This is true, in that you can hit a brick wall. When you hit that brick wall is another matter. Your SSD will keep track of which bits it has written and how often, and spread the load appropriately. I won’t go into the math, but it is safe to say your SSD will most likely last far longer than a standard hard drive.

Lastly, do be cautious before buying an SSD. Once you experience SSD performance, you will likely be motivated to upgrade any laptop(s) or desktop(s) you regularly use. There is no going back. My desktop boots in less than 20 seconds and is noticeably faster than any setup I’ve used in the past.

Picking an SSD

So! You decided to buy an SSD. Which do you pick?  Any SSD is an upgrade over a hard drive. The difference between the SSDs you might chose is much smaller. That said, I prefer drives known for a combination of performance, reliability, and nerd-affinity. Go with a Samsung 830 or Crucial M4 and you will be happy. 256 GB seems to be a good price/capacity range and should hold a good number of VMs. I personally went with two. AnandTech and ArsTechnica should provide you with further details if needed. slickdeals (popular or trending deal links) and fatwallet will help you find sales, which are very common.

You can keep your old hard drives

Do keep in mind that you may want to augment your storage with spinning disks.  SSDs come at the cost of capacity and price.  You may want to keep one or more spinning disks to house large files that don’t need high IOPS, and to house your backups.

Building a whitebox Home Lab

Overview

After wrapping up courses for an IT-related degree, I realized I had little real world experience. So little that I was immediately dropped for consideration of even an entry level add-and-remove-users-from-active-directory position when I acknowledged I didn’t have much experience in Active Directory User and Computers.

Regardless of the merits of their reasoning, many employers will not risk hiring employees without specific real world experience. What can you do about this if you are interested in the field of IT but don’t have much experience? You can create this experience yourself!

Plan things out

What you use in your home lab should reflect your interests and goals. You will find many variations in network, storage and server equipment and services or software depending on what experience you are aiming for. This post assumes you want experience in the virtualization and systems side of things.

Be sure to budget out what you can afford and plan accordingly. If you aren’t in the best financial shape, the last thing you want is to fall into the trap of realizing you didn’t plan things out and now need to spend more and more on components in order to get up and running.  Make sure to research your choices to ensure they are compatible with other hardware and software you plan to use.

What do you need?

We’ll start with the basics. You should ideally invest in a UPS device that will cover the system you run your virtual environment on. I knew this, yet made the mistake of starting a number of Microsoft labs while my lab PC was still upstairs for convenience after being built, sans UPS. When the power went out, AD and DNS weren’t the same, even when spread across a few VMs. Next time the UPS will come upstairs at build time : )

If you will be connecting to this computer remotely, transferring files, or building a cluster of servers, you should ideally be using CAT 6 Ethernet to provide gigabit speeds.  I run my computers through a cheap but solid gigabit switch behind my router.

Now you need a computer! If you have access to a fancy rackmount or old server from work, they should do the trick. You could also pick up a whitebox from Dell, HP, or your vendor of choice. I’m a bit picky and like to specify exactly what goes into my computers; if you want an optimal computer, don’t need vendor support, and pick the right components, this is always a good route.

What goes in the computer?

There are two primary bottlenecks in most virtual environments: Storage IO and RAM. You need to prioritize your budget and build around these core components.

Storage

One SSD that will hold VM files at the very least. 256 GB should give you decent breathing room, 2x 256 GB SSD drives will be more than optimal. Check my next post on why SSD is an absolute requirement for any lab where you plan to virtualize one or more computers.

Optional storage

One regular hard drive for temporal / large files, two or more regular hard drives for backups

RAM

RAM is more straightforward, but equally important. Most consumer motherboards and the free ESXi hypervisor will limit you to 32 GB. If you have the budget, go for 32 GB. Cut down on your CPU budget before you cut down on RAM.

CPU

The basic idea here is that more cores are better. Hyperthreading gives you two threads per core, essentially doubling your cores. Make sure you get the features you need; for example, with an Intel option you would want hyperthreading, VT-x and EPT, and VT-d if you want direct passthrough. Keep in mind VT-d requires both CPU and motherboard support. I went with an i7-3770 as this provides 8 threads. Intel Xeon E3 is another option, but keep in mind this will push up your budget with ECC RAM. This site will let you search/filter/compare Intel processors.

AMD is certainly an option and may help you if you are limited in budget. They offer some solid price per core options. That being said, I generally stick to Intel.

Motherboard

Look for the features you need. You probably want to limit your search to boards with SATA 3 and USB 3. I wouldn’t be concerned with onboard LAN; even Intel chipsets might not work in ESXi or have signed Server 2012 drivers yet. If you want VT-d support, good luck finding a motherboard. I found it quite difficult to pair up a well-reviewed board with the features I wanted and VT-d support on top.

NIC

ESXi can be quirky with NIC support. As mentioned above, some built in Intel ethernet adapters won’t work in ESXi, and may not have signed Server 2012 drivers. I ended up with a dual port Intel Pro/1000 adapter. You can generally find these on sale and they have a solid reputation.

Power Supply

Get what you need, nothing more. I personally prefer a high efficiency module PSU. Keep your PSU in mind when selecting a UPS device. For example, if you pick an active PFC power supply, you will need a Pure Sine Wave UPS system or you might run into issues…

Case

Pick something that fits everything you need, offers good airflow, and meets any specific criteria you have (e.g. if you have it in sight, you might want to steer clear of the neon-lit monstrosities).

What did I end up with?

I had the convenience of not needing anything right away. I waited for each component to hit a nice sale before diving in, used credit cards with reward opportunities, and when possible used FatWallet.com cash back vendors. slickdeals (popular or trending deal links) and fatwallet will help you find sales. There was no ‘everything arrives today!’ feeling, but it was worth saving a few hundred dollars : )

  • UPS: CyberPower CP850PFCLCD PFC Compatible 850VA 510W Pure Sine Wave Tower UPS
  • Case: Azza Genesis 9000B Full Tower Case – I love this by the way; it is indeed a monstrosity, but it fits everything I need and was significantly cheaper than comparable cases.  I keep it out of sight in the basement.
  • CPU: Intel Core i7-3770 Ivy Bridge 3.4GHz (3.9GHz Turbo) LGA 1155 77W Quad-Core Desktop Processor
  • Motherboard: MSI Z77A-GD80 LGA 1155 Intel Z77
  • RAM: G.SKILL Ripjaws X Series 32GB (4 x 8GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800) Desktop Memory Model F3-1600C9Q-32GXM
  • Backplane: Thermaltake MAX-1562 Storage Bay Adapter
  • SSD: 3x Samsung 830 series 256 GB
  • HD (Backups): 2x Western Digital WD Green WD20EARX 2TB 64MB Cache SATA 6.0Gb/s 3.5″ Internal Hard Drive
  • NIC: Intel EXPI9402PTBLK PRO/1000 PT Dual Port Server Adapter
  • PSU: Seasonic 560W 80 Plus Gold ATX 12V/EPS 12V Power Supply – X-560 SS-560KM

computer

Where can I get more help?

There are a number of online resources that can assist in your home lab build. Google or Bing will be quite helpful, and the following resources may help:

What’s next?

Hypervisors, software and lab guides. I went with Windows Server 2012 / Hyper-V, and most of the software I am trying can be found through DreamSpark, TechNet or MSDN. After diving into why you need an SSD, I will post a follow up on picking a hypervisor for your lab and where you can acquire software for testing.