Proper file transfer methods and processes can be tough to adhere to under regulatory laws and time constraints. Dan Sullivan, the author of The Shortcut Guide to Eliminating Insecure and Unreliable File Transfer Methods, discusses how traditional file transfer methods must adapt to new business operations and evolving wide area network (WAN) demands. Sullivan has more than 20 years of IT experience that includes engagements in application design, systems architecture and enterprise security across many different industries. In this interview, he explains how WAN managers can make file transfers secure and successful.
How are file transfers conducted traditionally?
Dan Sullivan: File transfers are sometimes one step or a subset of all the tasks in a complex business workflow involving multiple systems. For example, order fulfillment or credit risk assessment might require some file transfers, but much of the work in those processes involve other steps. Often times we will use a basic script, like a shell script or a Perl script, to transfer files usually using FTP or occasionally HTTP as the protocol. We often do it quickly with very narrow business requirements in mind, and we write the script in a way that satisfies immediate business requirements.… When we do this, we end up rewriting the same script over a few times and duplicating functionality. If we are rushing, we tend to limit error handling. Sometimes the code is difficult to maintain as different developers use different design patterns in their programs. So we get a lot of these file transfer silos, and they tend to be very application specific.… [Then they may not] necessarily scale well. Plus, there may be security issues. That is sort of the broad view of custom [file transfer methods].
How should file transfer methods change from the traditional method you just explained?
Sullivan: A file transfer itself is a significant business process; it's not just a small part of a single application. It really needs to be managed the way we manage other kind of common processes—whether its code developments or backup. Part of it begins with standard policies and procedures for defining file transfers and [determining which] requirements [are] based on certain types of file transfers.
[In example,] if there is highly sensitive information being transferred, there is going to be some security policy … associated with that; if there are performance issues, there are going to be guidelines associated with that. So part of … [what needs to change] is just the governance.
From an implementation perspective … [your file transfer method should be] more centralized; not necessarily so that all the file transfers go through one server, but that they have a single code base or a centralized application that is used for most, if not all, of your file transfer methods. That helps avoid maintainability problems. Also, if you have a proficiently- or robustly-designed application transfer system, then there is less concern about scalability and maintainability, because you’re treating file transfer as a first-class business problem that needs to be addressed. Ideally, you have perhaps a third-party solution that is designed for file transfers and is able to support multiple protocols. If you are dealing with business partners or customers who are going to use different protocols, it's very hard to do it right as a custom program because there are so many features that you want to capture. If you centralize it, manage it well and have a really robust application that you use, those are really the three key [priorities] to getting file transfers right.
What part of the file transfer method is a WAN manager of a large enterprise typically most concerned with? What should be their biggest priority?
Sullivan: It always varies by context, but if I had to prioritize, I would say security comes first. Depending on your industry, there are different compliance and regulation issues that you face. But, in the file transfer process, if you can't reasonably guarantee the integrity and confidentiality of the file transfer, then all of the other things really don’t matter or seriously jeopardize it. So I would say security is the No. 1 priority.
For No. 2, I would combine performance and reliability. Performance is just being able to finish the file transfers in whatever window of time you have and being able to scale as the business volume grows. Reliability is just being able to count on your application, [so] that you’re not constantly worried that there’s some combination of circumstances in which the file transfer method is going to break down, which sometimes happens when you have custom scripts.
A lot of us on the application side would see it as a favor if a WAN manager provided a centralized file transfer mechanism. I think a lot of us would run to them in droves.
-- Dan Sullivan, Author of The Shortcut Guide to Eliminating Insecure and Unreliable File Transfer Methods
No. 3, I would say is management and reporting [that provides] short-term, tactical alerts and reports if there are problems. [In example,] if there is a space or storage problem with a file transfer and the file transfer method fails, you would want to know right away so you could correct that. Also, long-term reporting [helps you] understand trends.
From a WAN manager's perspective, what would be your first step in making sure that the file transfer doesn't retrograde to the faulty and traditional file transfer method you described earlier?
Sullivan: This is really a governance issue. This gets into the heart of how you enforce your IT policies. Often times the best ways are not the technical ways; they tend to be more organizational issues. For example, if you have a ticketing system … [you could make a] new file transfer as one of the options on your ticketing system to make a request to IT. So, one part is defining the policies and procedures. Another part is making it real easy for someone to tap into the centralized mechanism. The third is writing simple scripts to search directories, where you, for example, will look for Perl scripts, pipeline scripts or ruby scripts that use modules or protocols that are particularly indicative of file transfer methods. But I am not as much of an advocate for monitoring to enforce them. If you set the polices, and then make them reasonably easy, I think most people are going to comply, especially developers who have been around for awhile, like myself. Writing file transfer programs is something that many of us have done a lot and is something we love to avoid. A lot of us on the application side would see it as a favor if a WAN manager provided a centralized file transfer mechanism. I think a lot of us would run to them in droves.
Continue reading this Q&A to learn more about secure file transfers and compliance.