Software Delivery

I managed the team that took part in a deployment of genomics analysis software to a government programme based in Cairo Egypt worth £2m in total, I also acted as a hands-on technical lead.

I managed the resourcing of a consultant from China to attend for 3 weeks and had scheduled to be there on the first week. The deployment was time-sensitive and had to be completed on-premise in this window as we wouldn’t have remote access.

The consultant from China was delayed, so I had to use my technical skills to quickly get the install going having not performed one before. I did alone on the first day – a Sunday – and then brought in my other consultant and a DevOps resource on conference call for the following two days, until the Chinese consultant arrived. All this took place whilst the client was monitoring over my shoulder – so I had to present confidence in what I was doing at the same time. The software was successfully deployed inside of the 3 weeks. However there were some performance issues identifed.

Health Check

After concerns were raised by the client regarding performance – with accusations flying, I took the decision to prepare a mirror environment in our offices and we requested soem of their sample files to process and confirm that the software was capable of processing them – which it was. So the issues pointed to infrastructure.

I agreed to attend site with one of the product managers to perform a health check against the software and hardware. I had some input in what to look for from the DevOps team, but largely had to script the health check myself against Kubernetes, NFS and PostgreSQL. Onsite, I ran multiple tests and perfected the script which used a combination of tools including kubectl, docker, fio, iostat, mtr and netstat. With this I was able to narrow down the issue to packet loss between the application and NFS, which was hosted on a different network split by a switch or router – which was not as specified in the Solution Design we had outlined.

This helped to not only provide the customer with confidence in the solution and our commitment to support, but did prevent them from walking away from the contract as they had previously disputed ownership of the issue.