5 tactics to beat big data hiring challenges

Hiring for specialist roles in emerging areas of technology can be tough – and expensive. But you don’t need to over-pay when building a new team if you follow the right strategy.

When big data started heating up, I faced the challenge of building a group of qualified engineers in New York City when most of the available talent was located on the West Coast. We could have staffed up by paying more, but that isn’t necessarily the best use of company funds. Instead, we did the following:

1. Identified talent we could “skill up”
2. Helped build a technology community
3. Hired offshore
4. Partnered wisely with recruiters
5. Maximized our networks

These tactics can be modified to help grow any specialist technical team. Consider these five strategies to address your hiring challenges.

1. Teach an engineer to fish

In 2008, when we first saw the need to develop a big data team oriented around the Hadoop ecosystem, there were very few big data engineers in NYC. That essentially left two choices: Recruit and relocate from the West Coast, or mentor and train our own team. 

[ Trying to find and retain cloud talent? See our related resource Hybrid Cloud: The IT Leader's guide. ]

Given the limited availability and high costs of finding, hiring, and relocating engineers, we chose to build and train our own team. We established a hiring and training process oriented around an engineer’s intelligence, ability to learn, and interest in working on big data engineering challenges. This approach was, and continues to be, wildly successful. 

The engineers we hired were ecstatic to be on the forefront of learning and establishing a new engineering discipline. Another plus: While many established West Coast big data engineers would likely be inclined to leave for a higher salary, the engineers we hired thrived in learning and overcoming challenges, and exhibited outstanding loyalty to their technology leaders and the company. In short, they appreciated the opportunity.  

2. Tap the power of community

Big data community participation worked well for us. We founded the NY Hadoop User Group in 2008 to share knowledge and grow the big data community within the metropolitan area, and the group has substantially improved since then. We also sent key members of our team to Hadoop World and other conferences, where they rubbed elbows with leaders in the big data world. Some early influencers ended up writing seminal books around Hadoop, and some of our team members served as book reviewers. Getting your team members and company name out into the community can also be enormously helpful in recruiting.

3. Expand your team offshore

While we didn’t find any experienced big data engineers offshore, Eastern Europe was home to many smart, experienced engineers with analytical math backgrounds who were excited to learn about big data technologies. You can either build your own remote offshore team or partner with a consulting firm to help you build one.

Unless you have deep experience in running offshore teams, I recommend finding a partner. We interviewed a number of companies to find one we liked and trusted, then worked together to attract, hire, and teach the engineers how to solve big data challenges. If you take this approach, don’t contract out the work as a project-based consulting engagement – instead, integrate and treat the offshore engineers like any other member of your core team. 

Mentor offshore team members not only in big data technologies, but also the language and mechanics of your industry and business, as any good engineering leader does with their own core engineering team. Engineers who understand your industry are more intuitive and successful at solving business problems; the same holds true for your offshore team members. Treat them as valued employees and they will reciprocate by forming long-term ties and loyalty to you.

4. Treat recruiters well  

I’ve worked in New York for 15 years, and in that time I’ve established strong relationships with some talented technology recruiters. I spend time teaching them what skills and talents are important to me in candidates, and the type of engineering culture that I like to build. I also give them my attention, return their calls, offer prompt feedback, and make hiring decisions quickly. 

Those recruiters find great candidates for me and establish long-term relationships with my team. Those engineers go on to highly successful careers of their own, and in turn hire their own big data engineering teams. This creates a holistic community of big data technology leaders and specialty recruiters who help to service our big data hiring needs.  

5. Work that network

I’ve also developed strong relationships with technology leaders and engineers within the big data community. Leveraging those contacts provides a bird’s-eye view into which companies are doing well, as well as which ones are struggling and may have some great engineers looking for career opportunities.

When you need to hire quickly, get the word out to your network and work it: Take colleagues to breakfast, lunch, dinner, or drinks to discuss your opportunity. Even if it’s not a good fit for them at the time, they often will give you leads on other potential candidates. Your meeting won’t always lead to a hire, but at least you’ve gotten reacquainted and shared advice and other potential leads. 

Does it work?

Yes. This process worked when I built my first big data team, and it is working now at PebblePost, where we are developing new areas of big data and data science expertise. This year we tripled our team in just four months, and we plan to do it again in 2018.  

I hope these tactics help you build the specialty skillsets your business requires. If you can recommend other approaches that have worked for you, let me know – we can build a community and discuss specialty hiring techniques.