Advanced Security Defenses

Security Engineering: Splunk Deployment Server V4

Note: This post was written for Splunk 4.2 on Linux.  Some concepts may be similar for other versions but you should consult the Splunk docs.

I've been hands-on working with a Splunk implementation again and found myself looking for good documentation on their Deployment Server functionality to help train others.  The docs on Splunk.com are quite good but require serious experience with Splunk in order to understand the settings and configurations available for centralizing the automated deployment functionality.  This post contains information for those starting out with the Deployment Server. 

1. Planning Deployments

Splunk Deployment Server functionality is designed to automate the deployment of policies, configurations, settings, and apps to remote Splunk components within your environment utilizing a basic client-server model.  But depending on your needs, this basic model can be extended in a variety of ways to meet your needs, some of which are discussed here.

Basic Deployment Pattern

The basic model for implementing a Deployment Server within most environments requires a simple set of classes for each type of Splunk component: search heads, indexers, and forwards (heavy, light, or universal).  This diagram depicts the basic functionality for how configurations and apps can be pushed out from the Deployment Server to each class of system within the environment.

Subclass Deployment Pattern

For those that require an additional layer of flexibility within their design of configurations and applications to accommodate separation of logging facilities or different retention policies for specific data, this design pattern utilizes subclasses to divide out the functionality of the indexers into idx-central and idx-critical.

Subclasses provide a basic construct for dividing out configurations and/or apps based on your needs and requirements.  Here you can determine what is different about each class and develop different deployment packages for each subclass.  In this case, retention and index storage may be different for each subclass which requires the configuration files to be defined specific to each.

Distributed Deployment Pattern

For some organizations, they may wish to have separate deployment servers in each data center or remote office to reduce failure domains and to enhance the security of each environment by alleviating the need to distribute configuration files among domains.  In this pattern, a master deployment server can be created to serve as both a deployment server for specific components and as a master for other slave deployment servers. 

This is a complex pattern and requires a significant amount of testing to ensure that you get the configurations correct but can be very powerful for multi-data center environments.

2. Identify Infrastructure

Depending on the pattern you chose during the previous step you will need a number of servers to support your Splunk implementation.  For the Deployment Server, I'd pick a server that is not part of your Splunk infrastructure to install and configure.  Why? 

1. Systems that run other Splunk components will become bogged down by deployment functionality within larger environments (more than 100 Splunk servers).   Splunk deployment servers should be distributed when the system begins to support approximately 300 nodes.
2. When you add the Splunk Deployment Server functionality to an existing Splunk server, this system will be configured different from other servers with similar functionality enabled.  The Deployment Server utilizes separate folders and configurations from other component infrastructure to establish the deployment functionality, ie. deployment-apps, apps, and serverclass.conf.  This means that you will either have to include the same server in the Deployment Server configurations for the component functionality already present or configure it separately as an exception.  Both can be tricky.
3. When troubleshooting the Deployment Server functionality and component functionality, it makes it easier to separate issues when the Deployment Server exists independent of other functionality.
4. In large environments, distributing deployment functionality ensures stability, facilitates automation and policy integrations, and strengthens the security model.

If you don't have a separate system for the Deployment Server initially, don't fret.  You just have to think about the system as though it provides separate Deployment Server functionality when configuring and troubleshooting.  But plan to replace it as your Splunk infrastructure grows.

3. Install Splunk

You will want to install the most recent version of Splunk on each system.  The Splunk Deployment Server should always be kept up to date since it is backwards compatible with agents but agents are not backwards compatible with old deployment server functionality.  To do this, it is also a good practice to create an install script which can be leveraged to automatically update the servers where configuration files are still considered compatible.

4. Configure the Deployment Server

It's simple, follow these steps:

1. Choose a name for the deployment package you are going to create.
2. You will add your deployment packages to /opt/splunk/etc/deployment-apps/ on the Deployment Server. 
3. Within the deployment package, you will want to create the configuration files or place apps that you will be distributing to other splunk servers.
4. Use the local directory within your package to deploy configuration settings. (/opt/splunk/etc/deployment-apps/package1/local)
5. Configure serverclass.conf with rules to push out apps and configurations by deployment class on the Deployment Server.
6. Configure deploymentclient.conf on Client Servers to check in with the correct Deployment Server.  This will also set the phoneHome feature for your deployment client.
7. Reload the Deployment Server.

Create Deployment Packages

Organizational skills really help when developing deployment packages.  We determined that it is necessary to define your architecture and create classes during planning in order to know how they will be different.  You can create classes and sub-classes by identifying the different use cases for each splunk component type.  For example, if you have central indexers and an indexer used for critical security events, you might want to set up the following classes: idx-central and idx-critical. 

Each deployment package should be developed with the functionality of the Splunk component type in mind.  You should name each deployment package for the class or subclass of each bundle so that rules and and packages coincide for easier troubleshooting.  And each package should contain only the conf files and apps necessary for each class of system to operate effectively.  For example, if you are configuring an indexer, then you should include inputs.conf with the splunktcp attribute defined.

Basics required for each Splunk component type:

1. Deployment Server: serverclass.conf
2. Search Head: web.conf, authentication.conf, authorize.conf, server.conf
3. Indexers: authentication.conf, server.conf, inputs.conf, web.conf
4. Heavy Forwarders: outputs.conf, web.conf
5. Light Forwarders: outputs.conf, web.conf
6. Universal Forwarders: outputs.conf, web.conf

Likewise, it is important to note that you may want to set up different information and settings for deployment clients in the field and that should be done by a separate deployment-clients package which will contain all of the settings required for deployment clients such as phoneHome settings in a single bundle.  This bundled package can then be added to all Splunk components and centrally managed.

Tip:  Consider using a spreadsheet to plan out your deployment classes identifying necessary conf files and apps that will be required for each class.

Demystifying serverclass.conf

Once I realized that this file was much like a firewall rule set it became easy for me to understand what was needed and how to set up complex deployment scenarios.  Afterall, some of the same firewall concepts apply: ordinality and cardinality are important, wildcards can be applied, and classes can be defined either top-level or granularly.

Some tricks:

1. Place a global whitelist rule with wildcard at the top of the serverclass.conf file. 
2. Use IPs (10.1.1.1) instead of hostnames (abc-def01.domain). 
3. Use VLANs (10.1.1.X) instead of naming prefixes (abc-defXX.domain.com).
4. Try to narrow the scope of wildcards within deployment classes. (10.1.1.* instead of 10.1.*)
5. If you whitelist or blacklist, check and double check that your numbering is ordered properly. (0,1,2,3)
6. The complexity of the Deployment Server is commonly caused by filtering and the order of rules within serverclass.conf.  Filtering and order of processing are paired.  If you specify whitelisting as the filter type, then whitelists are applied before blacklists.  If you choose the opposite, then blacklists are applied before the whitelists.

Rolling out deploymentclient.conf

The deployment functionality for Splunk requires setting up a deployment client so that each remote system will phoneHome.  This is an important part of the configuration for the Deployment Server since it helps to ensure that the remote systems have the latest configurations and apps as well as providing an administrator with a single place to check all deployed clients.

Reload instead of Restart

Splunk developed the reload function as part of its server which reloads the Deployment Server and saves you from having to restart the splunk service.

5. Troubleshooting and Debugging

There are a few common pitfalls when configuring deployment server functionality that can be avoided.  Here are some of the ones that we encountered and what was done to correct them:

1. Deployed configuration being trumped by local configuration - If someone logs into a system and changes a configuration located in system/local then restarts the system you may encounter issues with the Splunk functionality for that component.  Those unfamiliar with the deployment server functionality may try to make direct changes to configuration files which conflict with existing deployed settings.  This is actually more common than you would think and requires configuration monitoring to prevent. See Splunk FIM.
2. Conf files that require parameters -If you have copied and pasted files then things like hostname= are invariably going to be incorrect on each system and will likely cause problems.  To correct this, consider using environment variables within your configuration files to create parameter based configurations specific to each host.
3. Hostnames not resolving - I know I mentioned this in the serverclass.conf section above but I will explain why here.  I found that hostnames did not resolve properly due to DNS issues and this can cause settings on some systems to not get updated properly.  When we switched to IP addresses in the serverclass.conf it helped to resolve many of the issues.
4. To forward or not to forward -  When you have a serverclass.conf rule that does not work properly followed by rules with wildcards, you may find that settings and apps get pushed to the wrong classes.  Be sure to number your rules properly and review your serverclass.conf before reload.
5. SSL password hashes not correct - If you are running SSL within your Splunk infrastructure, and you should be, then you are likely to encounter this issue if you end up copying configurations from existing systems as you develop your deployment packages.  In order for SSL to operate properly you have to ensure that the password (sslKeysfilePassword) is the same in server.conf as it is in other configuration files for the same system.  If it is different, SSL will not operate properly and you will encounter a blocking effect which prevents messages from being delivered or functionality from operating properly.  You will also see error messages which read: SSL error or Can't read key file.  Determine which password is incorrect and fix or delete them and start over.
6. Indexers not running as expected - When indexers stop running there can be a variety of reasons for this to occur.  First and foremost, roll out the web.conf file and enable the web interface for your indexers so that you can validate issues more rapidly.  Using this method, I found that forwarders had been deployed to indexers which had caused them to stop running.  Inputs.conf had incorrect attribute definitions which prevented logs from being indexed.  And at one point, the indexers had been moved and indexes corrupted which required splunk fsck to be run before they would restart.

 

Resources:

• Splunk Docs - About Deployment Server 
• Splunk Docs - About serverclass.conf

Control: File Integrity Monitoring

It is quite common for novice and hobbyist attackers to break into an environment through an unpatched system and upload modded files to effectively over-write the functionality that may already exist within your environment in order to hide their activities.  On Linux, this may consist of uploading a rootkit that mods ls, ps, w, who, netstat, login, and top.  On Windows, this may consist of over-writing files within the system32 directory or hooking API calls.  Either way, many organizations add security controls to aid in preventing and detecting this type of anomalous behavior.  One such way to mitigate this activity is to use File Integrity Monitoring (FIM) to provide some basic visibility around unauthorized file modifications.  

FIM can be deployed as an agentless or agent based solution within an environment and can be run in a central or distributed architecture. It generally, runs on a scheduled basis and signatures are written to detect adds, changes, deletes, and modifications for each grouping of binaries and files on the file system.  Out of the box, most FIM solutions provide signatures for monitoring critical operating system files and to detect known rootkits.  And FIM software can alert when unauthorized changes occur depending on the rules you provide.  As part of your overall strategy, FIM also aids in providing value for SIEM solutions where FIM alerts and logs can be shipped via XML or syslog to a SIEM monitoring solution for triage and correlation to other events.

It is also important to note that much of the FIM market these days is driven by standards requirements such as PCI DSS which require that FIM be installed on systems within the Card Holder Data (CHD) environment.  In this case, an auditor will look for FIM to be installed and sending alerts, but its overall value to the organization is only as good as the signatures and monitoring alerts which rarely comes up during a PCI DSS audit.   The smaller market demands come from security professionals who use FIM knowing its strengths and weaknesses to protect and detect the server OS and other critical files as part of a larger strategy.

FIM Products

Because of PCI DSS there are many more tools, software, and services in this space than in previous years.  And each of these products has their own features and benefits which now include additional reporting and correlation elements.  Here is a partial list:

• Tripwire
• OSSEC
• NNT Change Tracker
• File Integrity Monitor (ncircle)
• File Integrity Monitoring (LogRhythm)
• NetIQ Change Guardian
• CIMTrak
• AIDE (Advanced Intrusion Detection Environment) 
• Verisys
• McAfee Integrity Control
• Osiris
• Samhain
• ...

Planning Questionnaire 

FIM planning is key to the success of this detection control.  Because of this, it is important to create a planning questionnaire which can help you to determine which product/service to choose and the necessary elements required for integration into your environment.  Here are some basic questions to consider adding to your planning questionnaire:

• What are the goals for FIM? __________
• What is the amount budgeted for FIM?  __________
• Will FIM be used to meet compliance requirements?  __________
• Will FIM be used as a key control within your environment? __________
• How many systems will need to be monitored with FIM? __________
• Which Operating Systems will require monitoring? __________
• Where will FIM be installed?  __________
• Which type of architecture will need to be deployed? __________
• Will you use agents on the servers/assets being monitored? __________
• How will agents be deployed?
• Does your environment leverage central configuration management? __________
• How will FIM alerts be evaluated? __________
• Will FIM logs be sent to a centralized log repository? __________
• Who will monitor and approve changes for FIM?  __________
• What reports should be developed and who should receive these?  __________
• Is FIM required to meet regulatory or standards requirements? __________
• How will FIM be tuned and who will be responsible for identifying new FIM signatures? __________
• Will a SIEM solution be utilized within the environment? __________
• ...

Use Cases

Use cases are necessary when implementing detailed monitoring solutions within your environment.  For FIM, security use cases provide a great deal of insight into what is important and how it relates to the specifics of the business.  A use case for FIM is usually structured towards what a FIM can provide which is information about modifications, new files, permissions, and file stamps.  Here are some use case examples to get you thinking:

• Detect critical binary changes.  
  • Monitor: Modifications, Permissions
  • Binaries: ps, ls, w, who, login, netstat
  • Rule: Dynamic
  • Why: Suspected Rootkit, Unauthorized Change
• Detect critical file changes.
  • Monitor: Modifications, Permissions
  • Files: passwd, group, httpd.conf, hosts, exports, crontab
  • Rule: Dynamic
  • Why: Unauthorized Change
• Discover new files in specific directories.
  • Monitor: Add
  • Directories: etc, var, init.d, xinetd.d
  • Rule: Dynamic
  • Why: Malware, Unauthorized Change

Implementation Guidance

To implement FIM, it is strongly recommended to utilize agile or scrum to phase the installation and configuration steps in order to ensure that the product is working properly and detecting events before spending time tuning.   Similarly, a solution design is essential in reaping rewards from a FIM implementation because it is necessary to achieve alignment for this solution from the start.  Consider the following as part of your FIM solution architecture: (1) placement, (2) hardware and OS, (3) access, (4) initial configuration, (5) logging, and (6) integrations.

If you go for an out of the box install during the implementation phase, you will quickly realize things about your FIM solution architecture that you had not considered.  For example, if you have a configuration manager within the environment, every time it runs a schedule job your FIM solution will detect a modification where timestamps are being monitored.

Tuning Guidance

Tuning is one of the most detailed tasks that a security engineer takes on.  Essentially, it requires a great deal of planning and knowledge about the environment in order to derive benefit from a FIM.  Here are some tasks to take into consideration for your tuning project:

• Organize your assets into groups, ie: by OS, by type, by function, by VLAN, by service.  
• Next you will want to determine the criticality of your assets since some products allow you to classify your assets for notification and alerting purposes.  This is easily accomplished by setting up a notation for criticality and ranking each server within the organization.  Ranking may not be very valuable for FIM by itself but plays into a much larger security strategy whereby it becomes essential to support correlation and triage.
• Identify the strategy for the rules being written and how a new rule will add value to other security use cases.
• If you are using a SIEM, determine what information needs to be added to the rule for it to be useful during correlation.
• Identify other software in the environment which may alter files or timestamps and which requires correlation to exclude false positives.

Maintenance

Most highly tuned security tools require extensive maintenance to ensure that their value is maintained over time.  For FIM products it is essential and vital to ensure that maintenance efforts continue to include review and upkeep for signatures, rules, actions, and reporting.  Without this continuous effort, a FIM will begin to break down as changes deviate from the initial patterns that were set up.  As a result, most FIM products require daily reviews and upkeep.  Depending on your server farm and FIM product, this effort could range from minutes per day to hours and in many cases require a dedicated staff member to trace changes back to their origin.

Limitations and Countermeasures

But it is not good to think that FIM by itself provides silver bullet protection for your servers without additional controls and processes.   Likewise, it is not a good strategy to think that FIM can protect your systems from all rootkit and activity hiding that can be attempted on a system.  Why do I say this?  Simple - FIM watches what you want it to watch and alerts you when something you did not expect to happen happens in a precise manner.  In this case, if you implement a FIM and write a signature to let you know when /sbin/ls has been modified, FIM will send an alert anytime the file gets modified. 

But FIM breaks down when file hooking is leveraged by files that have been hidden in directories, tmp space, memory or slack space.  In this way, a FIM rule that watches /sbin/ls may not be aware of a program on the system that hooks the functionality for /sbin/ls whenever it is opened.  Signatures for catching this type of activity are far more advanced and require a great deal of skill and knowledge about specific types of attacks and changes at a low level of each operating system type.  Because of this, FIM only works as well as the signatures being developed for the environment and finding skilled resources can be a real challenge.

Consider overcoming these limitations by utilizing FIM as part of a control group, where it provides value by adding to other control pairings.

 

 

 

 

 

Strategy: Security by Color and Number

Making security more tangible has been a long-standing and recently revitalized goal of mine.   This year it even seems more relevant because it is simply not enough to have a few talented security individuals who defend the organization from attack but instead security requires whole company participation with increased awareness by all.  And this notion of whole company security participation requires commitment, balance, and creativity to reduce the complexity of security so everyone can participate in the same strategy, alignment, and goals.  Because of this, I've worked on a concept to apply colors and numbers to assets and resources to assist the concept of whole company security participation by supplying users with more tangible risk information.

More simply stated - sometimes pulling out the crayons makes a world of difference...

Imagine that you have been given a list which contains assets that have been categorized by color and number supplied to you by the Security department and which correlates assets, vulnerabilities, and risks.  In this case, the asset list becomes more useful to you when trying to defend critical resources because you can quickly determine how to prioritize your defenses and triage efforts.  Similarly, this information becomes available instantly for tool integrations and can be supplied as metadata to a CMDB.   But best of all, you hardly need to have 20 years of malware expertise and security operations to know what to do to help protect the organization.

Moreover, as a security practitioner, the benefit from this type of concept is that these colors and numbers can be leveraged for grouping risks, to support high level policies and to reduce the risk overall within your organization.  And with this information tied into a CMDB, it is dynamic allowing for more flexible control over your security program and tactics simply because you can make incremental changes more rapidly than with traditional concepts.  And you can accomplish this rapidly throughout the organization without missing a beat.

And best of all, getting started with this type of strategy can be quite simple where you have a network map and basic understanding of your incident response process.  First, you will need to identify the colors and numbers appropriate for your environment.  At this point you can identify a basic model and make changes later as you begin to better understand the boundaries of your categories for risk.

Consider the following example of a network based model:

Color	Risk Rating	Zones	Notes
Red	1	Databases, Application Servers	This category represents applications, systems, and data stores which may store, transmit, or process sensitive information and which are regulated/governed by strict requirements.
Orange	2	Application Servers, Proxies, Utilities, Bastions, Monitoring Solutions	This category represents applications, systems, and data stores which may connect to Red assets for the purpose of management, application support, and which are not directly governed by external requirements.
Yellow	3	DMZ Servers, Public services (NTP, DNS, Mail)	This category represents systems that require public access and which are not allowed to directly interact with Red assets.
Blue	4	External Servers	This category represents systems which connect into the environment but which are not under the direct control of the organization.

Using this type of model you can then develop use cases, rules, and policies for how these zones inter-operate and what your expectations are for monitoring and event response.  Likewise, you can create adapted risk ratings by color to extend your color group for more granular risk evaluations.  And by posting this information within your intranet, wiki, or other common information share the rest of the organization can use this information as a reference when developing their solutions.  But as you can imagine, this type of matrix is also dependent on your environment and how these categories should be devised within your organization will likely be dependent on your business requirements.

Taking the above example one step further, you can identify some basic principles to support the development of firewall rules and zoning within your network and apply these colors to network maps and VLAN planners, like this:

Color	VLAN	IP Range	Notes
Red	500	10.0.50.0/23	Database Servers
Orange	400	10.0.40.0/23	Operations Servers
Yellow	300	10.0.30.0/23	Bastions
Blue	200	10.0.20.0/24	Partner #1

Similarly, you can leverage this same type of modeling concept for setting incident response and triage service level agreements (SLA).

In this case, color category can help to determine the SLA and response time required for each event type.  The following example event heat matrix really works when you identify relevant response times above and use them based on risk severity:

	Unauthorized Access  Success	Malicious Logic/ Malware*	Policy Violation	Attempted Unauthorized  Access/ Intrusion	Probes and Recon	Denial of Service attack	False Positives
Red Assets	10	9	8	7	7	7	3

In this way, those with less experience can leverage the knowledge of those who have seen what works in the field as part of their daily procedures.  And color provides the context required to highlight when something seems out of place.

In this way, color can greatly increase the inherent strength of your security design to empower others within your organization as they make important risk decisions.

 

 

Strategy: Security Use Cases

Ever engage in Mad Libs when you were young? I remember them vividly because of many long summer car rides up the coastline. And six months ago the concept came in quite handy when developing security use cases for our SIEM implementation.   A creative moment in a long and painful meeting sparked some ingenuity that would soon prove to elicit information from our systems engineers and network administrators with very little effort and hardly any education on security malfeasance. There on the white board, we began to identify security use cases using the Mad Libs concept.

So what is a security use case, you ask?   A security use case is a model which helps determine the relationship between threats, vulnerabilities, risks, and controls; such that, a useful pattern can be developed for achieving a security goal, ie. protecting an organization from unauthorized access or abuse.

Security Mad Libs applies so naturally to security and use case modeling because it helps to establish a known pattern with identified variables and placeholders for value identification.  In building a number of these quickly on paper, I realized a pattern that made it easier to work with others to develop these use cases and which was derived from my childhood obsession over Mad Libs.  

At a high level, they look something like this:

The _____1______ can hide their tracks by ______2______ on the  ______3______ and which _____4______.

1. system admin; internal threat                                                  

2. deleting logs; using someone else's account

3. log server; network

4. prevents analysis and correlation; makes it difficult to identify the attacker

When  ____1_____ logs into a ______2_____ and modifies _____3______ send ______4_____ to the ______5______ as a _______6________.

1. a user

2. network device

3. the running config

4. an email alert

5. SOC

6. high alert

 

At a more granular level, they look something like this:

When _______1_________  logs a _______2________ to ______3______ and which alters  ______4_____ send ______5_____ to the ______6______ as a _______7________.

1. Tripwire

2. change

3. /etc/pam.conf

4. login auth required pam_ldap.so.1 use_first_pass

5. an email alert

6. SOC

7. high alert

And these can become as granular as you like - even down to the actual config policy that is applied.

But more importantly, this concept was something that could be easily shared by all within the organization to increase visibility, support, protection, and response with minimal training towards development of these patterns used to defend the organization from attack.  And they became immediately valuable for those that develop and implement complex and advanced security monitoring solutions because they helped provide insight to plan out new signatures and focus security solutions based on security events and attack patterns.  

It is also worth considering that these can be added in real time or derived from an organization's enterprise threat model as part of a more targeted implementation effort and to adapt the security program using organization identified knowledge and resources. 

 

 

Strategy: Security Awareness

It's amazing to me that these days you hardly have to leave the confines of your home or office to test the effectiveness of social engineering at your organization.  In bunny slippers and armed with a cup of coffee, a skilled engineer can run several different use cases in a matter of minutes and wind up with a mountain of knowledge about your organization and its key people.  And with good use cases, it is highly likely that your key people can be infiltrated just as easily with a simple email.   This is due in large part to the rise in popularity of social media but also because of the nature of networks and partnering that has given rise to the Internet in general.  Likewise, it is much easier for attackers to organize their thoughts and ideas through software which has been designed to support social reconnaissance efforts, ie. Maltego.  (For a good discussion on Maltego and personal reconnaissance, go to this link.)

Recently, I had an opportunity to talk with more people about this topic and it quickly became apparent that the challenge for most security professionals is understanding how this plays into their overall security strategy and meeting the new demands for educational awareness.   Likewise, most security professionals find it difficult to keep current with popular attack vectors simply because of the number of new social media concepts popping up these days.   And the pattern of social media usage also contributes to an effective disadvantage for security staff, since the people with the greatest privileges are often those that are more active on social media services while those that govern and enforce authorized access are seemingly less active.  And when you are trying to develop training packs for a targeted audience this apparent disparity in usage can make for less credible awareness information. 

But while its understandable to be overwhelmed by the volume of things you might need to know as someone who governs, its more important to come up with a training concept which actually employs the benefits of social media.  Here are some thoughts:

Go Viral

I once watched the best awareness videos I think I've ever seen by a gentleman I worked with, and in simply watching the first 3 minute video I had several people at my desk wondering what all the laughing was all about.  Soon, we watched the next clip and the crowd began to grow.  After watching several more of the videos, I turned around and noticed that most of the building was huddled around my desk and wanted to know where they could find the videos because they wanted to send them out to their friends. 

Advertise

Don't just buy security posters for the office, personalize your security awareness campaign.  People are far more likely to notice security awareness materials when they are involved or their friends are.  And like the concept prior, they are far more likely to talk about and socialize about security when it is a less nebulous topic.   After all, who doesn't want to see Judy, the CEO's admin assistant, fighting cyber criminals?  The point here is that everyone loves a good story or concept that they can talk about. 

Challenges and Mystery

Most office professionals think that Information Security is very cloak and dagger among other things.  So why not take advantage of this and put together concepts which hone the skills of your user base.  In prior years, I worked with a group of people who set up stuffed animals (Garfield) in an office as part of our awareness campaign to see how long it would take for someone to send a notice to the Information Security department to identify the change.  It took 3 days and we posted the winner of our challenge on the Intranet site with a variety of other statistics and information to heighten awareness.  From that day forward, the leadership at that organization noticed a stronger commitment by its users towards security and ensuring that they did their part.

Build a Relationship

Establish a relationship with your users to make security more tangible to them.  I've used the concept of Lunch and Learn or newsletters to reach out to a broad audiences in order to educate and prior information.  People are smart and curious which makes it easy to discuss topics to establish a security dialogue with your user base.  And more important it creates a more direct relationship between the Security department and end users who protect the organization.

Key People

While it is important to extend awareness to the whole organization, it is also important to identify key people and access which require additional diligence.  Missing an opportunity to more directly target and train these individuals could be a serious mistake.  In the past, I've found one on one sessions or small roundtables provide the best key people awareness simply because the dialogue can be more targeted to their specific use cases within the organization.  But it is also worth noting that most key people may seek you out because they want the information that will help them protect their access and the organization.

Guest Speakers

The occasional notable guest speaker can also be a great addition to a security awareness program.  Outside experts bring with them knew information, stories, and concepts from external to the organization which can increase the overall knowledge of the user base but also of the Security department.  So it is worthwhile to think about extending invitations for guest speakers to larger audiences within your organization. 

 

 

 

 

Security Strategy: Security Project Management

Planning big security projects or changes?  Security project management is not for the weak of heart.  It takes nerves of steel for most project managers to jump in and tackle the security frontier simply because of the nuances and complexity.  And likewise, there are very few technical security professionals that can and would want to project manage security deliverables without authority.

So the problem that exists for both organization and employee is where to find a good security project manager that can help get things done.  Believe me when I tell you that they are not easy to find, but it is possible to pair a good project manager with a knowledgeable security professional and sometimes the combination can be quite powerful if the dynamics are right.

Having worked on several large scale compliance projects, I've had the fortune to work with several amazing project managers who soon became well-versed in security project management.  And to be honest, I wish there could be more like them.  The equation, in my case, was simple and effective.  Find a project manager who could coordinate well and who had a disposition that few could resist.  Then implement the following plan: 

To Security and Beyond...

Good security people think just a little different than most everyone else and often idealize a target state for security as opposed to setting current and realistic goals. Because of their nature and different way of thinking, it is important to provide your project manager with some security basics and training in order for them to understand how to work with security resources. 

Then work with the team to establish security vision and strategy with near-term and long-range goals.  This will help the team and the project manager to understand the work required and the path forward.  Setting the stage this way also prevents what-if scenarios and analysis paralysis from stifling your team's success.

Creating a Plan

After setting the stage and identifying a strategy for the team, you will also need to work with your project manager to develop a detailed plan.  During this process, I would highly recommend that you avoid over-simplifying identified security projects and simply develop a staged approach for how security work can be completed.    For the most part, its not enough to put a basic list together when managing large security projects.  And there is also no right answer on methodology for how to manage security projects.  Waterfall, Agile, and Scrum all work equally well to complete security implementations and updates.  You will have to determine what works best for your size of organization, resources, and goals.

In any case, whether your project manager is using Excel, MS Project, Agile Software, or Scrum boards, you will need to assist in identifying the details, constraints, dependencies, milestones, and resources necessary to complete each project.   And it is unlikely that as a security manager you will avoid hearing the word requirement with its many definitions being overused and misinterpreted.

Example: Spreadsheet Backlog

#	Description	Category	Due Date	Assigned	Constraints	Notes
P-000001	Implement Intrusion Detection in CN Office	Perimeter	Jan 10	Elliott	Network	-Requires 10GB Interfaces -Discuss Operations with NetEng

Example: Scrum Backlog

It is also important to have an understanding of how big each project is and the details behind the work tasks in order to properly determine target dates and to help with phasing and prioritization.

Phases

Preparing your project for multiple phases can help you to realize short and long term benefits in a matter of weeks as opposed to months or years.  Depending on your project methodology, you can achieve this by setting up a consistent implementation and delivery model.  These models are often referred to as life cycles and enable project participants to know how to achieve their work tasks as well as how to contribute to the next part of a project.

For security teams, phases are essential because most resources are not dedicated to an implementation and often the resources that are required to operate a solution are also needed to build out and tune the same solution so that it works as expected once it is released.  As the majority of an implementation is completed, the work becomes mostly operational with iterative work required to tune and maintain the solution during its lifetime.  Consider the following work breakdown where build out work burns down as operational work increases:

Phases should be consistent with established deliverables in order to realize the value from each phase.  For delivering and maintaining security solutions, this often includes some variation of the following phases: Research, Design, Implementation, Testing, Tuning, and Maintenance.

Prioritizing

The most difficult process in security project management, as with most project management, is setting priorities and targets for your deliverables.  This is due in part to the difficulty and complexity of aligning business and security/risk strategies.  Likewise, compliance targets can also impact delivery schedules and require a great deal of task shifting to ensure that resources and commitments are properly balanced. 

If you think that this step is where the magic happens, it most certainly is.

Good security project managers are able to effectively couple and de-dup tasks required for all initiatives to accommodate successful completion of targets and goals.  They tend to know what tasks are urgent and critical to the completion of larger milestones and help to ensure that constraining tasks are worked on in succession to alleviate blocking events before they occur.  Likewise, the completion of security projects often require collaboration and focus which a project manager can provide to keep tasks moving when other issues and events occur.

Depending on the project management techniques used by your organization, consider working with your project manager by using milestone planning or release management techniques to set priorities within your project portfolio.  In this way, your tasks can be prioritized within the overall plan to meet targets and address specific resourcing issues before they arise in the implementation of a security project and so that the team understands the demands of their project work.

Implementation

Implementation is usually a simple part of most projects once a plan is set forth but does have its own complexities especially during security projects.  As you work through most tasks, the cross-functional nature of security implementations can lead to new requirements and potential course corrections in order to overcome blocking events.

Take the following example:

The security team plans to implement SIEM by meeting with all project contributors, to include: network engineering, systems engineering, and application support.  Everyone identifies their *requirements* and a project plan is developed. 

A few weeks later, the security team begins to roll-out the product solution only to find out that they need a collector server within each zone to send back events to the centralized appliance in order to accommodate load required by their use cases.  The team prepares a request and submits it for change management.  The change is received by the systems engineering team who have their own priorities and slate the install for later in the month.  Without the servers, the security team is now stalled on their SIEM project. So they move onto another project while they wait.

Later that month, the systems engineering team sets up the servers and determines that the load at each collection point could be an issue which requires a change to the SIEM architecture plan.  A review of the architecture is requested and the requirements are reviewed.  The teams quickly determine that they did not understand the solution and question the value of sending some of the data to the SIEM.  Despite the written security use cases, the other teams begin to determine that their network and system architecture will not sustain the load and demands of security.  The project team decides that more analysis is required and re-enter the architecture phase of the project.

The SIEM project and targets for delivery are now in danger. 

As you can see, the demands of the solution became more apparent during the install and the project team's assumptions had established what seemed to be an effective plan.  But the issue occurred long before this project in a related project where these requirements were not understood before the enviroment was built because the systems engineering team had little if any experience identifying security issues.  Because of this the systems engineering team did not approach the project scope with correct assumptions and underestimated the security requirements in a project that provided the infrastructure for SIEM.

So how do you overcome this type of impacting event during implementation and hope to proceed with your detailed plans?

As you can imagine, some experience and general understanding in implementing these types of complex solutions is required to ensure that project teams do not encounter blocking events of the magnitude displayed above.  But generally speaking from experience, if you cannot get resources to align you need to identify constraints as early as you can within your project planning in order to avoid these types of issues during later more costly phases within your project.

With the above case study, the team may not have encountered this blocking event during implementation if they had sought architecture assistance in both the SIEM project and infrastructure project where a lack of experience was present.  Within this organization, it became apparent that constraints had to be identified early and the architecture phase required triple the amount of time for each project.

Communications

Security project managers are a huge benefit during security projects because they help to ensure that cross-functional project teams are working together and collaborating effectively.  This value is quickly realized during the project process because project managers tend to help bring team members together for discussions when technology and ticketing systems fail to provide a sense of common goals or alleviate blocking events.

The best security project manager, in my experience, is organized and helps drive completion of projects through both technology and face to face discussions when necessary.  While this style can seem odd for project managers with waterfall experience, it is highly desired in companies that leverage Agile and Scrum to move work along on a daily basis.

It is also worth noting that some project managers can overwhelm a project team with too much communication and too many meetings.  Successful project managers look to balance this with technology and drive meetings that have goals which is a highly recommended strategy for alleviating the ill effects of too many meetings.  Believe me when I tell you that you'll know when your team has been saturated by too many meetings or when your project manager is ineffective...it looks like a filled calendar every week.

Reporting

One of the most critical functions for a security project manager is reporting because detailed status is often needed to keep all stakeholders informed and often gets lost or becomes confusing when assigned to line management.  But, as with other steps within security project management, reporting for security projects is not without its caveats and pitfalls.

From my experience, a considerable amount of confusion arises for most project managers when they first get assigned to security projects due in no small part to requirements.   Project managers often confuse compliance reporting with security project reporting.  This can happen for a variety of reasons and requires some oversight by security line management to ensure that projects are being effectively tracked without overlapping into compliance reporting.  This is very important because the overlap of these two can result in less security within your environment and organization where compliance and security targets become confused.

Security project managers learn early within the development of a security project portfolio that compliance is a minimum set of targets and often overcome reporting inaccuracies by working with the security team and line management to identify phased goals for their security project reporting.  In this way, executive and senior management are presented with consistent project reports that address resourcing and target completion instead of confusing compliance and process reports which are based on risk and where the goals may or may not be finite.  

Reporting can also benefit from the earlier development of phases and a project lifecycle since these phases can provide valuable metrics for senior management.  In this way, project and phase metrics can provide visibility on the success of security project completion as well as identify other problems within the organization that lead to long term and more risky security gaps.  Because security is a maturity dependent business process, security project reporting, like compliance reporting, can identify when infrastructure teams lack process or cannot successfully meet SLAs and can often indicate where operational inadequacies exist.

Likewise, a good reporting strategy is required between security project managers and line managers to ensure that project reports and metrics are helpful and create agility to deal with and resolve issues quickly.  Without this visibility, both parties cannot be effective in achieving their goals for the organization.  Likewise, the reports that a security project manager completes for line management should be simplified and not require a great deal of plan manipulation to determine if the project is on plan or off.

Compliance

Although there are a few new reasons for why security professionals do security projects,  for the most part, many organizations embark on these projects as a result of compliance requirements.  I think it goes without saying that compliance and security have very different goals within an organization and that security is not meant to be managed the same as compliance.  Likewise, both disciplines require very different styles of project management in order to complete projects and meet organizational goals.  Compliance projects require very prescriptive tracking while security projects are more fluid and occur in response to changing business requirements. 

 

 

#	Description	Category	Due Date	Assigned	Constraints	Notes
P-000001	Implement Intrusion Detection in CN Office	Perimeter	Jan 10	Elliott	Network	Requires 10GB Interfaces Discuss Operations with NetEng

Security Strategy: How to Avoid Running Default Configurations

You can't really blame folks for installing solutions in default mode OR can you?  I've worked with a lot of companies and technology professionals during my career and it seems like going default is more common than you would expect.  Many organizations buy solutions and don't realize that they have a default implementation until it is pointed out.  And sadly, this covers all types of solutions where there can be alot of debate about what default really means.  I guess that's why I chose to blog about this topic.  

So - what does going default mean?  Well - let's discuss the implementation of an IDS/IPS.  Top-end software and appliances usually come with thousands of rules and a standard out-of-the-box configuration so that they can be installed and show some immediate value for a majority of customers.   The IDS/IPS is brought in and set up on a span port, users are added, and a rule set is crafted from the existing signatures available in the signature repository.  The rule set is applied and the system is configured to send syslog to a central log management system.  The engineer in charge reads a few pages in the user guide and starts monitoring.  And the implementation is considered done... OR is it? 

Things like this happen all the time in companies - small and large. Why? Well - it's pretty simple, the market for security professionals is currently targeting Security Architects and Application Security experts who are familiar with concepts and tools instead of Security Engineers with low level knowledge of systems and protocols.... this is evident in some basic but interesting search stats from the public profiles on LinkedIn:

1. packet analysis: 1800
2. custom signatures and IDS: 3750
3. regular expressions: 4180
4. reverse engineering: 37,800
5. application security: 59,200
6. security architect: 84,300

Note: these statistics were from a basic search on profiles and do not eliminate duplicates or false positives.

And because of trends like these, it is likely that your organization would require the assistance from a knowledgable trusted advisor or vendor willing to point out and correct the default nature of your solutions.  But most of the hot talent with this level of experience is in higher demand this year after the large scale hacks, so it is also likely that your solution will continue in default mode or until something happens where it may come to light that the products used to defend your organization are not tuned for your environment. 

But since I believe that pointing out a problem requires highlighting possible solutions, the question is how can your company avoid running default solutions without the need for external help?  Well there are a variety of things to think about when installing solutions that require constant tuning and maintenance.  Here are some steps designed to aid your efforts in tuning and enhancing your security solutions:

Step 1: Identify the Solutions

The first step is to identify solutions that require tuning and enhancement. Here's a list of the top 10 security solutions that require long-term tuning and maintenance:

1. Event Correlation Systems
2. Firewalls
3. Intrusion Detection/Prevention Systems
4. Web Application Firewalls
5. Code Review Tools
6. Identity and Access Management
7. Policy and Risk Analysis Tools
8. Vulnerability Scanners
9. Hardening and Verification Systems
10. Centralized Logging and Monitoring

Each of these solutions can be implemented with a variety of open source and commercial products which come with default settings and rules to get a company started. 

If you already have some of these solutions implemented, it is likely that a few are still running an out-of-the-box configuration and rules/signatures that the manufacturer identified to meet the needs of its customers.  In this case, you may want to consider performing a configuration review against the default parameters to see how much your solution has been tuned.  Likewise, if the solution was installed and considered operational within days, it is likely that the solution has not been completed tuned for your environment.

Similarly, many of these solutions are required to meet the demands of compliance requirements and certification efforts.  If the solution has been implemented to meet the needs of an audit or certification it may have been installed to fulfill a checkbox requirement in which case you may be well-served to perform a gap assessment on the settings and rules. 

Step 2: The Goal

Whether you are beginning an install or deciding to tune an existing solution, make sure you outline the goal of the solution being implemented.  While it may be obvious to most that this is necessary, many organizations rush through this step and find themselves in a circular situation where meetings continue to prevent the solution from being properly tuned.

In the case of a new install for a solution that requires tuning, goals are relatively basic and will likely change after the first report is run.  The goal for most new installs where staff have not yet utilized the supporting product is to implement the solution and perform basic stability tests.  If these tests are successful, identify a next phase goal.  For example, if your team has recently procured a new web application firewall with the goal of preventing XSS attacks, successfully installed and tested it, then the next step is to assess the rules and determine:

1. Whether new rules are needed to stop additional XSS patterns.
2. If existing default rules are still running and needed.
3. And identify if the solution can be used to prevent additional attacks.

In the case of an existing solution, it is important to perform an assessment to determine the state of the solution while considering how the solution would best prevent or detect relevant threats and attacks.  For example, an existing intrusion detection system which was installed when the organization only had a Windows environment and now has a heterogenous operating environment is likely in need of some tuning.  Here the goal may result from use case definition and a re-work of the solution to meet the changing requirements of the organization.

Step 3: Definition of Done

The next step is to define when the solution is considered installed and operational.  Sometimes organizations do this by establishing a lifecycle process because some of their solutions are never really considered done.  And sometimes, an organization will require each project to be defined until done is effectively achieved.  The real trick here is to understand how your organization engages in projects and the current state of your solution, either new or existing.

If your organization engages in incremental work instead of long term projects, then it is likely that your definitions of done will be based on increments.

If your organization has an established project management office which manage large scale long-term projects often with the waterfall planning and management method, then you will be required to define your solution in terms of a set of goals and phases such that a return on investment (ROI) can be realized.

Step 4: Managing Milestones

Outline each phase and a milestone goal for the solution.  Generally, I begin my planning and resourcing efforts for a solution by identifying a basic threat model, requirements, and use cases.  This can help to drive milestone and phase plans for new installs as well as establish re-work plans for existing solutions.  During this step, it makes sense to build a basic powerpoint which provides agreed status information and enough detail for all to understand how the implementation is progressing and when it has reached completion either in increment or as a result of a long-term plan.

Step 5: Training

The best way to avoid having a solution that does not meet the needs of the organization is to train both the support staff and those using the system so that they understand the advanced features and configuration capabilities available.  

Many organizations fail to complete this step of the implementation and then wonder why an engineer believes that the solution they support daily is operating effectively when it is not.  As an example, consider an intrusion detection system that continues to run without being tuned for specific VLANs and which is running Windows rules in a 100% Linux environment.  In this case, the security engineer did not know these rules were running and it took weeks to tune the system using reports and a group effort to reduce false positives.

For complex solutions and products, I'd also recommend sending your security engineering staff to training or plan for offsite time to complete webinars.  Training is only effective when staff get a chance to focus and become immersed in training exercises and materials.

One Final Thought

Sometimes organizational management may hear that a solution is running and get a false sense that security is  operating effectively.  Security has become quite complex and executive management may also need some education about how security solutions are completed.  If this sounds like your situation, then you will need a gap assessment to be performed by an external consultant or by an audit team with a meeting to review the results.