Data Pipelines (ETL) Archives ~ fitzmoskal https://fitzmoskal.me/category/skills/tech-skills/data-pipelines-etl/ You are _here_ Thu, 29 Feb 2024 18:13:53 +0000 en-GB hourly 1 https://wordpress.org/?v=6.9.5 https://fitzmoskal.me/wp-content/uploads/2024/02/cropped-result_880649-32x32.jpeg Data Pipelines (ETL) Archives ~ fitzmoskal https://fitzmoskal.me/category/skills/tech-skills/data-pipelines-etl/ 32 32 11678478 Canadian Investment Asset Management Company – Ditigal Transformation https://fitzmoskal.me/canadian-investment-asset-management-company-ditigal-transformation/ Thu, 22 Feb 2024 14:37:17 +0000 https://fitzmoskal.me/?p=1426 I was part of a consultancy team to deploy a new ITAM function and I was specifically responsible for discovery

The post Canadian Investment Asset Management Company – Ditigal Transformation appeared first on fitzmoskal.

]]>

I was part of a consultancy team to deploy a new ITAM function and I was specifically responsible for discovery and integrations to the CMDB, a project worth $3m over 3 years.

The particular challenges faced were that all of their IT functionalities were outsourced to different partners and vendors and some of these partners insisted on work orders to help us with data gathering. This was a complex challenge which required some level of obtaining buy-in and negotiating with the client to get work orders processed as quickly as possible to meet our tight deadline.

I held multiple workshops with the client and partners – and we deployed a “waterfall” data pipeline for populating the CMDB.

After deployment was completed, an issue arose with the accuracy of the CMDB and the wording of the contract which was “95%” accuracy, with no caveats. The first milestone payment was held back whilst the team resolved this critical issue.

I was not directly involved with the client by this time, but as part of resolution I suggested to the account management team that they needed to ensure the contract wording was updated in future to specific automated data only. I also assisted with identifying the sources of data, and types of data – such as that which would be manually entered and would ultimately be down to the client to ensure it met “95%” quality.

The final approach with the client was to review all CIs and fields within the CMDB to be populated and adopt a MoSCoW approach, determining which data were neccesary and which data they agreed, for example, would be impossible to obtain. This massively improved the baseline results and showed that CMDB population was already very close to “95%”. After some additional work on the data pipeline the customer agreed to release the first payment milestone.

The post Canadian Investment Asset Management Company – Ditigal Transformation appeared first on fitzmoskal.

]]>
1426
Management Consulting – Global Programme Software Rollout https://fitzmoskal.me/management-consulting-global-programme-software-rollout/ Thu, 22 Feb 2024 13:42:28 +0000 https://fitzmoskal.me/?p=1420 I was hired as a Delivery Manager and Enterprise Architect for the global rollout of Discovery and CMDB for an

The post Management Consulting – Global Programme Software Rollout appeared first on fitzmoskal.

]]>

I was hired as a Delivery Manager and Enterprise Architect for the global rollout of Discovery and CMDB for an international Managment Consulting company. This was a 2 year programme directed at North America and European regions worth €2m.

In the first year I identified and implemented €90k worth of cost savings to the programme by automating level 1 support so that the resources were no longer required. I led a team of 8 offshore engineers in India and Poland and was responsible for their training and managing their progress on the the deployment to individual accounts.

I reported to the Programme Manager and was responsible for not only the software architecture but the security design needed to obtain global CISO sign off and allow the programme to proceed when it hit a major hurdle and was blocked by security compliance. One of the many hurdles was implementing RBAC into the solution which had not been designed for full RBAC control.

I worked with one of my lead engineers to find and document all the group permissions available in the tool, then employed a hack to prevent certain pages from loading without the right permissions. This satisfied the requirements of Global CISO. During the process we even discovered a vulnerability that allowed access without permissions, which was reported back to the vendor. The vendor was so impressed that they asked if they could see our documentation.

The post Management Consulting – Global Programme Software Rollout appeared first on fitzmoskal.

]]>
1420
US Investment Bank – SLM & SAN Storage https://fitzmoskal.me/us-investment-bank-slm-san-storage/ Thu, 22 Feb 2024 13:21:12 +0000 https://fitzmoskal.me/?p=1414 I worked for a partner consultantancy with the Software and Configuration Management team to extend and customise BMC Discovery to

The post US Investment Bank – SLM & SAN Storage appeared first on fitzmoskal.

]]>

I worked for a partner consultantancy with the Software and Configuration Management team to extend and customise BMC Discovery to provide support for their licensing and auditing function. With the software license discovery and modelling that I workend on both indendently and sometimes as part of a team, the bank was able to make substantial savings in the 100’s of thousands of US Dollars on licensing to vendors such as Oracle, IBM and Microsoft.

I worked under and Agile framework to develop TPL code, performed monthly regression analysis on software install data and created and maintained all documentation for customisations.

I was also part of the team to code and develop an extension that modelled their various SAN storage solutions deployed. After the solution was implemented, the bank was able to identify 10,000 US Dollars in monthly savings by being able to turn off a decommission storage arrays that were no longer being used.

The post US Investment Bank – SLM & SAN Storage appeared first on fitzmoskal.

]]>
1414
Digital Health Checks https://fitzmoskal.me/digital-health-checks/ Tue, 20 Feb 2024 19:37:14 +0000 https://fitzmoskal.me/?p=1378 During my time as a consultant I have worked to develop various digital health checks to assist companies with taking

The post Digital Health Checks appeared first on fitzmoskal.

]]>
During my time as a consultant I have worked to develop various digital health checks to assist companies with taking a tempurature test of their IT systems and strategy. The view of health checks is sometimes that they are just a tool to drum up more business – which to an extent is true, but the health checks I developed have been critical in assisting companies to identify risks, avoid costly mistakes and pinpoint the source of complaints.

I have developed comprehensive health checks for BMC Discovery, Kubernetes, NFS and PostgreSQL. I have also developed exception reporting on data pipelines for data quality assessment. The health checks I have developed have saved customers in the following areas:

  • Identified security vulnerabilities around plaintext usernames and passwords storage
  • Identified issues around NFS performance
  • Identified opportunities for license reclaimation
  • Identified usage patterns for insecure practices (i.e. logging in as root user)
  • Identified various misconfigurations in non-alignment with best practices

The post Digital Health Checks appeared first on fitzmoskal.

]]>
1378
Online Betting Platforms – CMDB Data https://fitzmoskal.me/online-betting-platform-cmdb-data/ Tue, 20 Feb 2024 18:56:33 +0000 https://fitzmoskal.me/?p=1372 I was brought in to architect and deliver an data pipeline for updating their CMDBs from multiple data sources. The

The post Online Betting Platforms – CMDB Data appeared first on fitzmoskal.

]]>

I was brought in to architect and deliver an data pipeline for updating their CMDBs from multiple data sources. The CMDB itself was hosted on Salesforce and the data sources were Discovery, Active Directory, Spreadsheets and SCCM.

A particular challenge was being restricted to operating remotely using TeamViewer – which does not perform well over data instensive remote connections – and requires an active user on the end to keep the connection going. This did cause development work to go much slower than planned, but thankfully did not cause overall delay to project.

The integrations were developed, tested and deployed successfully to the satisfaction of the client.

The post Online Betting Platforms – CMDB Data appeared first on fitzmoskal.

]]>
1372
Major Fashion Retailer – Digital Transformation https://fitzmoskal.me/major-fashion-retailer-digital-transformation/ Tue, 20 Feb 2024 18:26:35 +0000 https://fitzmoskal.me/?p=1369 Over the years I built a relationship with this global fashion brand in facilitating their ongoing digital transformation. The first

The post Major Fashion Retailer – Digital Transformation appeared first on fitzmoskal.

]]>

Over the years I built a relationship with this global fashion brand in facilitating their ongoing digital transformation. The first engagement was in architecting and implementing a number of critical integrations with their CMDB from various data sources, allowing them to gain full central visiblity of their enterprise infrastructure. Later, I was again called upon to re-architect and extend the earlier integrations in preparation for a move of CMDB cloud platforms.

As a Principle Solutions Architect I also oversaw the technical delivery of their managed services – consulting on roadmaps and strategy with the client. I built a trusting working relationship and also proposed and worked on implementing a new feature of exception reporting, which helped them to identify, understand and narrow down the discrepencies between source data and CMDB data. The result was clear visibility and improved data quality of their CMDB.

The post Major Fashion Retailer – Digital Transformation appeared first on fitzmoskal.

]]>
1369
Online Betting Platform – ITAM & Cloud Migration https://fitzmoskal.me/online-betting-platform-itam/ Tue, 20 Feb 2024 17:33:49 +0000 https://fitzmoskal.me/?p=1357 Over the years I have worked with a leading online betting brand to facilitate various technical and digital transformation requirements

The post Online Betting Platform – ITAM & Cloud Migration appeared first on fitzmoskal.

]]>

Over the years I have worked with a leading online betting brand to facilitate various technical and digital transformation requirements including straightforward software configuration to cloud platform migration.

My first engagement was to assist in improving their scanning performance and software license reporting which was accomplished within a three day deadline.

Later, I architected a move of their ITAM discovery solution from on-premise to a new cloud platform. I assisted in a number of areas which needed security clarification, particularly with the storage of credentials and data transfer. This did cause some delay in the migration but the client was grateful for our due diligence and went ahead with the move.

I also delivered corporate training to this customer, and worked on some customisations of their scanning solution with population of the CMDB. As a Principal Consultant at the time I oversaw the technical aspect of managed services solution that was provided to them.

The post Online Betting Platform – ITAM & Cloud Migration appeared first on fitzmoskal.

]]>
1357
Investment Bank – ITAM & DB Migration https://fitzmoskal.me/investment-bank-itam/ Tue, 20 Feb 2024 15:57:32 +0000 https://fitzmoskal.me/?p=1321 I maintained a long-term relationship with a European Investment Bank based in Canary Wharf. As part of a team of

The post Investment Bank – ITAM & DB Migration appeared first on fitzmoskal.

]]>

I maintained a long-term relationship with a European Investment Bank based in Canary Wharf. As part of a team of consultants we had repeat work to maintain and upgrade their ITAM solution. I personally delivered on a 3-week critical project to facilitate a database migration. This involved liaising with lead DBAs to extract metadata from MS SQL, PostgreSQL and Sybase databases for the solution architecture.

This was a sensitive project given the job threat to the existing DBA SMEs. I faced open hostility from one SME in particular who’s tactic was to delay providing information and not respond to emails.

I resolved this with face to face contact, without letting any request go unanswered. I also used low-level escalation to my project sponsor. This did result in a tactical complaint from the SME to the project board, however my sponsor was satisfied with my approach, hitting their 3-week deadline and no fault could be found. I resolved not to make matters personal, which is how I managed to stay focused on delivery.

The post Investment Bank – ITAM & DB Migration appeared first on fitzmoskal.

]]>
1321
Middle Eastern Oil Company – CMDB ITAM Deployment https://fitzmoskal.me/me-oil-company-cmdb-itam-deployment/ Tue, 20 Feb 2024 15:04:19 +0000 https://fitzmoskal.me/?p=1307 I was assigned as a consultant to a large Middle Eastern Oil Company for six weeks onsite. BMC Discovery was

The post Middle Eastern Oil Company – CMDB ITAM Deployment appeared first on fitzmoskal.

]]>

I was assigned as a consultant to a large Middle Eastern Oil Company for six weeks onsite. BMC Discovery was their chosen IT Asset Management solution and their objectives were to deploy, scan their networks and model various services comprised of Oracle RAC Clusters, IBM LPARs and HMCs, Databases and add 2-way CMDB synchronisataion with custom asset locations.

The deployment and solution was delivered within the six week timeframe to satisfaction of the client. In addition they were supplied with various reports as value add. In one case they were able to identify a number of database servers which had been marked as decommissioned but not physically turned off. This improved their visibility and avoided ongoing costs and risk of exploited vulnerabilities in unmonitored infrastructure.

The post Middle Eastern Oil Company – CMDB ITAM Deployment appeared first on fitzmoskal.

]]>
1307
Mapping Food & Drinks Venues with Foursquare API https://fitzmoskal.me/mapping-food-drinks-venues-with-foursquare-api/ Tue, 20 Feb 2024 14:33:39 +0000 https://fitzmoskal.me/?p=1293 I was given a brief to extract data from Foursquare to identify the nearest food and drinks venues within 1000

The post Mapping Food & Drinks Venues with Foursquare API appeared first on fitzmoskal.

]]>
I was given a brief to extract data from Foursquare to identify the nearest food and drinks venues within 1000 meters latitude and longitude provided in an Excel sheet for a historical tour company.

I developed this with a mixture of Bash and Python, exporting the results to JSON and then converting to CSV. All code was handed to the client.

The post Mapping Food & Drinks Venues with Foursquare API appeared first on fitzmoskal.

]]>
1293