Starting Point#

  • Two separate AD forests syncing to a single O365 tenant
  • Three different firewall vendors
  • Six geo dispersed offices
  • Only limited S2S VPNs supporting only the O365 sync agent and a branch to brach backup

In short a very messy and disorganized network from lack of planning, under investment in IT, and each major corpo division doing their own IT. Executives traveling between branches would be unable to connect to the wifi as it was backed by a different forest, unable to access file shares at different locations, and have to access the corpo sharepoint with different accounts. New leadership was promoted and the divisional barriers started to drop. This led to IT becoming the first fully cross-division integrated department in the company. The IT team, led by me, then immediately started to find the pain points and develop an action plan.

Goals#

  • Short Term:
    • Ease executive travel
    • Access internal services from all locations
    • Cross site backups
  • Long Term:
    • Vendor standardization
    • IP readdressing
    • AD domain consolidation
    • FIPS validated VPN

Phase 1 - Basic Communication#

Initial work went to getting the two AD forests communicating. Basic statically routed IPSEC tunnels were stood up to enable the DCs to talk, then a trust was formed. This enabled cross forest file share and sharepoint access while at the site hosting the service. One major pain point eased!

Next was getting full inter-site network connectivity working. With the lack of standard network equipment the only common denominators were IPSEC with static routes. Static routing was quickly ruled out as unmanageable long term, but only 2 sites had firewalls that didn’t support OSPF. The sites with firewalls that didn’t support OSPF were quickly replaced with basic mini-pcs running pfSense. Then each site was configured to have IPSEC tunnels to HQ and the large branch site(PB1) hosting the second forest. The end result was a partial mesh with each site connecting to at least HQ and PB1.

Overall, this worked quite well as an initial fix. Users could access file shares from any site and IT personnel could access both forests without having to VPN in to another site. However, we knew this was not a long term sustainable solution. Multiple sites were running mini-pcs as firewalls and HQ and PB1’s firewalls were nearing end of life. Additionally, the quirks of supporting a multi-vendor IPSEC mesh were becoming tiresome.

Phase 2 - Standardization#

A proposal to replace all firewalls company wide was drafted and routed up to leadership. The “sell” was easy as looming compliance requirements and near EOL equipment would have forced the replacement quickly anyways. Fortinet’s offering won out on price to performance after evaluating options from multiple vendors. Our IP addressing and VLAN numbering schemes were overhauled to a single standard, and phpIPAM was setup for the system of record replacing spreadsheets.

With the planning completed, we moved to a site by site implementation phase. For each site the new FW was stood up in parallel with the existing. FW rules were built on the new FW and systems individually swapped to the new IP addressing and VLAN schemes. As during the project we were still multi-vendor no Fortinet propriety extensions or solutions were able to be utilized. This led to us only changing out the FWs and IP/VLAN scheme while the architecture of the IPSEC mesh with OSPF was unchanged.

The standardization led to immediate improvements in ease of configuration and reduced troubleshooting. With standard configs, IP addressing, and VLANs rule complexity dropped significantly and time to configure was massively reduced. However, we didn’t have centralized management of the firewalls and still had users split between two AD forests.

Phase 3 - Integration#

The consolidation of the AD forests was the next major challenge, all PB1 based user needed to be migrated to the HQ forest. The spit in AD was a continuous problem; many users had accounts on both forests and confusion for users and helpdesk personnel led to frustrations and poor ticket resolution. After extensive experimentation and testing a reliable migration procedure was developed that was minimally user impacting. Basically, create a new user account with the same UPN on the HQ forest, null the immutable ID on the cloud account, switch the cloud sync to see the new account, reset the password, and the swap the user’s laptop for a HQ based one. Over the course of a few months users were migrated to the HQ forest with the loss of zero user data. This significantly reduced helpdesk escalations and service times.

With the winding down of the PB1 based services during the AD consolidation, it was time to address the inter-site networking setup. The manually configured partial mesh worked satisfactory as an interim solution but complicated expansion and required fully custom configuration of each branch. Re-architecting to a hub-spoke setup would ease future expansion and allow for templated branch office configurations. Fortinet’s standard hub dial-up IPSEC with BGP on loopback architecture was selected for ease of expansion and troubleshooting. The rollout of the new overlay network was implemented without taking down the existing overlays. This allowed for gradual implementation during business hours and little time pressure. The implementation went exceeding well with zero user impact and additionally improved the resiliency of the the overlay network. SDWAN rules allowed for quick swap over during an ISP outage and ADVPN reduced spoke to spoke traffic via the hub.

The final part of centralized firewall management via FortiManager(FM) is still ongoing. Looking back starting with FM would have been the optimal solution; but budget and lack of expertise with Fortinet’s offerings led to us starting without it. The stand up of new FWs with FM has been a breeze but the importation and merging of existing FWs has been slow. There’s nothing “wrong” with the importation process but ensuring the standardized FM configurations will result in the same outcome as the current configurations is a slow, manual process. With many of the sites not having IT personnel on-site the ability to push the standard config and then tweak is risky and user impacting.

Conclusion#

Overall the project was extremely successful and low budget. The largest cost, the FW hardware, was already necessary due to EOL timelines and compliance requirements. Implementation was smooth and all service outages were short and outside business hours. IT manhours troubleshooting network and cross-forest issues have dropped massively and users have easy, consistent access to services no matter which office they’re at. Additionally, passing CMMC level 2 with the previous network would have been near impossible with the lack of FIPS encryption, poor ruleset consistency, and spit identities. This project was done in parallel with our CMMC prep work and it was a quite enjoyable if hectic three years.